Batching techniques for handling unbalanced training data for chatbots

Abstract

The present disclosure relates to chatbot systems, and more particularly to batching techniques for handling imbalanced training data when training a trained machine learning model so that bias is removed from the model when performing inference. In one embodiment, a plurality of raw utterances is obtained. A bias-removing distribution is determined, and a subset of the plurality of raw utterances is batched according to a bias-reducing distribution. The resulting unbiased training data may be input to a predictive model to train the predictive model. The trained predictive model may be obtained and utilized to predict unbiased results from new inputs received by the trained predictive model.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit and priority of U.S. Provisional Patent Application No. 62 / 002,151, filed on Mar. 30, 2020, entitled “BATCHING TECHNIQUES FOR HANDLING UNBALANCED TRAINING DATA FOR A CHATBOT”. The entire content of the above application is hereby incorporated by reference in its entirety for all purposes.

[0002] Copyright A portion of the disclosure of this patent application contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent application or patent disclosure as it appears in the Patent and Trademark Office's file or records, but reserves all copyrights whatsoever in all other cases.

[0003] Technical Field This disclosure relates to chatbot systems, and more particularly, to batching techniques for handling unbalanced training data when training a model so that bias is reduced or removed from a trained machine learning model when making inference decisions.

Background Art

[0004] Background Service providers worldwide utilize instant messaging chat platforms to communicate with their customers. Providers often use these platforms to engage with customers, resolve issues, or fulfill requests. For example, a customer of a service might describe a problem they are experiencing, and a responder to that service might describe steps to improve the problem during a live conversation. In another example, a customer might make a request to a responder as part of a live conversation to utilize some aspect of the service. Using human responders to facilitate live communication with customers or end-users can be very costly for providers. Training human employees to handle the various features offered by the provider is time-consuming, and their ability to converse live with customers can be limited over certain periods, especially if the service runs 24 / 7.

[0005] Chatbot systems offer a variety of technologies for managing interactions between entities, such as customers, and services. Service providers may use chatbots to communicate with customer entities, eliminating the costly and time-consuming manual interactions typically performed between two people. Chatbot systems may utilize text recognition technology to identify and respond to specific cues and contexts unique to live conversations with customer entities. Typically, highly refined intelligent chatbots, developed through training, can communicate more naturally with customers during live conversations. This provides customers with a rational and comfortable conversational experience while efficiently utilizing service resources. Intelligent chatbot systems may be continuously improved by regularly training the chatbot with comprehensive training inputs. Continuous training of the chatbot constantly improves the bot's ability to interact with customer entities and allows the bot to remain flexible through training with fresh and relevant use cases.

[0006] A chatbot system is greatly influenced by the quality of the data it is trained on. For example, training a chatbot system with a larger amount of training data typically allows the chatbot to learn more contextually complex behaviors and respond to a wider range of conversational situations. Grouping or "batching" high-quality training data allows the chatbot to efficiently learn functions and provide higher-quality interactions with customer entities. However, a chatbot may also learn poor behavior if it is provided with less optimal training data. For example, providing a chatbot system with a batch of unbalanced feedback, such as training data with a large number of training examples that are significantly biased towards a particular conversational context, will cause the chatbot system to learn biased behavior over time. Biased behavior can lead the chatbot system to engage in inefficient or inaccurate interactions with entities. These actions include interacting with entities in the wrong context, recommending solutions that are not helpful to customer entities, and / or wasting important resources by incorrectly engaging with irrelevant elements of the service. These and other biased chatbot interactions are detrimental to both the service provider and the entities it interacts with. Therefore, when training a chatbot system's dialogue model, it is important to provide the chatbot system with unbiased and efficient training data batches. [Overview of the Initiative] [Means for solving the problem]

[0007] overview The technologies disclosed herein relate, in general, to chatbots. More specifically, but not limited to, the technologies disclosed herein relate to techniques for batching training data to train a chatbot system. A chatbot can classify user utterances into different classes, such as predefined intents of the user / customer it interacts with. The classification unit of a chatbot may include a trained machine learning (ML) model that generates outputs (e.g., intents) based on inputs (e.g., user utterances). If the training data used to train the ML model is inadequate, the chatbot may more frequently identify incorrect intents. Specifically, biased training data, when used to train a chatbot system, can cause the chatbot to exhibit biased and invalid behavior. The technologies disclosed herein enable the batching of effective and unbiased training data so that chatbots may be trained using batches. Generating unbiased training batches significantly reduces or eliminates biases that would otherwise become deeply ingrained in the learned behavior of the chatbot. Eliminating bias in chatbot models allows chatbot systems to be trained in the most effective and resource-efficient manner. For example, by eliminating bias at the training level, services utilizing chatbots can prevent poor conversational behavior before it is learned, saving valuable time and resources that would otherwise be spent retraining the chatbot. Eliminating biased behavior at the training level will also prevent customer disputes that might otherwise result from operating with poorly trained chatbots.

[0008] This section describes various embodiments, including methods, systems, and non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors. A chatbot system uses a training set of utterances to train an intent classifier to identify one or more intents for one or more utterances. Utterances may be expected interactions with entities, such as a string of words in a conversation. Intents may be contextual categories of a conversation that are specific to an utterance. The intent classifier may be a tool for identifying correspondences between utterances and categorical intents. The intent classifier may be a model designed to be trained iteratively to more efficiently identify corresponding intents for particular utterances. The training set of utterances is typically provided as batches of utterances selected from a set of raw utterance data of a certain size. In a system that uses random sampling to generate such training batches, output categories (e.g., specific intents) of an intent classifier trained with more training utterances will be selected more frequently by the intent classifier than output categories with fewer training utterances, even if both categories match the input utterances equally well.

[0009] Therefore, such systems may suffer from bias when training the intent classifier during future training and when interacting with entities. These biases often occur when the training data contains an unbalanced amount of training utterances for a particular output category compared to other output categories. This disclosure addresses this bias by using a technique to balance the training batches used to train the intent classifier. More specifically, the system utilizes an intent distribution to generate distributed, unbiased training batches. This ensures that an intent classifier trained on distributed, unbiased training batches makes more equal use of each output category and reduces the inefficient bias inherent in previous random sampling methods.

[0010] One embodiment relates to a method, which includes acquiring training data comprising multiple utterances, each utterance corresponding to one intent among multiple intents, each intent among multiple intents corresponding to one skill among one or more skills, the method further includes determining an intent distribution across the multiple utterances, the intent distribution comprising multiple intent proportions, each intent proportion corresponding to one intent among multiple intents, the method further includes generating a batch containing a batch number of utterances selected from the multiple utterances based on the multiple intent proportions, and applying the set of training data from the batch to a predictive model for one or more skills The method further includes, including reciprocal input, constructing a predictive model as an intent classifier that includes a plurality of model parameters learned by the use of an objective function, training the predictive model on a set of training data by minimizing or maximizing the objective function, which measures the difference between predicted intents and ground truth intents, and in response to the training, learning relationships in the set of training data used by the predictive model to generate predicted intents, and obtaining a trained predictive model that includes a plurality of model parameters associated with the learned relationships. Another embodiment relates to a system comprising one or more data processors and a non-temporary computer-readable storage medium to perform operations including the method described above. Yet another embodiment relates to a non-temporary computer-readable storage medium storing a plurality of instructions that, when executed by one or more processors, trigger the execution of the steps of the method described above.

[0011] These exemplary embodiments are mentioned not to limit or define the present disclosure, but to provide examples to aid its understanding. The foregoing, along with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief explanation of the drawing]

[0012] [Figure 1] This is a simplified diagram of a chatbot system according to a specific embodiment. [Figure 2] This is a block diagram of an exemplary batch processing system according to one embodiment. [Figure 3] This figure shows an exemplary process for generating a training batch and obtaining a predictive model according to one embodiment. [Figure 4A] This figure shows an exemplary embodiment for generating training batches from a training data pool according to one embodiment. [Figure 4B] This figure shows an exemplary embodiment for generating training batches from a training data pool according to one embodiment. [Figure 5] This is a simplified diagram of a distributed system for realizing one embodiment. [Figure 6] This is a simplified block diagram of a cloud-based system environment providing cloud services, according to one embodiment. [Figure 7] This figure shows an exemplary computer system that may be used to implement several embodiments. [Modes for carrying out the invention]

[0013] Detailed explanation In the following description, certain details are included for illustrative purposes to facilitate a full understanding of the particular embodiment. However, it will be apparent that various embodiments may be carried out without these specific details. Figures and descriptions are not intended to be limiting. The term “exemplary” is used here to mean “provided as an example, case, or illustration.” Any embodiment or design described herein as “exemplary” should not necessarily be construed as being preferable or advantageous to other embodiments or designs. The term “based on” as used herein should not necessarily be construed as meaning an exclusive requirement for an action, but any such action may be construed as being nonexclusive and at least partially based on the requirement that follows this term.

[0014] 1.0 Overview of an Exemplary Chatbot System Figure 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to a specific embodiment. Environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistant or chatbot systems. DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, as shown in Figure 1, a user 104 representing a specific company can use DABP 102 to create and deploy a digital assistant 106 for users of that company. For example, a bank can use DABP 102 to create one or more digital assistants for use by its customers. Multiple companies can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables restaurant customers to order food (e.g., order pizza).

[0015] For the purposes of this disclosure, “digital assistant” is an entity that helps the user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant may be implemented using only software (for example, a digital assistant is a digital entity implemented using a program, code, or instructions that can be executed by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, etc. A digital assistant is also sometimes referred to as a chatbot system. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0016] A digital assistant such as a digital assistant 106 built using DABP 102 can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user may provide one or more user inputs 110 to the digital assistant 106 and receive responses 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and responses 112. Through these conversations, the user may request that one or more tasks be performed by the digital assistant 106, and in response, the digital assistant 106 is configured to perform the user-requested tasks and respond to the user with appropriate responses.

[0017] User input 110 is generally in the form of natural language and is referred to as an utterance. The user utterance 110 can be in text form, such as when the user types a sentence, a question, a text snippet, or even a single word and provides it as input to the digital assistant 106. In some embodiments, the user utterance 110 can be in voice input or spoken form, such as when the user says or speaks something that is provided as input to the digital assistant 106. The utterance is typically the language spoken by the user 108. For example, the utterance can be in English or some other language. If the utterance is in voice form, the voice input is converted to text form of the utterance in that particular language, and then the text utterance is processed by the digital assistant 106. Various voice-to-text processing techniques can be used to convert the voice or auditory input to text utterance, and the text utterance is then processed by the digital assistant 106. In some embodiments, the conversion from voice to text can be performed by the digital assistant 106 itself.

[0018] The utterance, which can be a text utterance or a voice utterance, can be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. Once the meaning of the utterance is understood, the digital assistant 106 can execute one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that the utterance is either a text utterance directly provided by the user 108 of the digital assistant 106 or the result of the conversion of an input voice utterance to text form. However, this is not intended to be limiting or restrictive in any way.

[0019] For example, the input of user 108 may request that a pizza be ordered by providing an utterance such as "I want to order a pizza". Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate actions. Appropriate actions may include, for example, asking questions that request user input regarding the type of pizza the user wants to order, the size of the pizza, any toppings for the pizza, etc., and responding to the user. The responses provided by digital assistant 106 may also be in natural language form and typically may be in the same language as the input utterance. As part of generating these responses, digital assistant 106 may perform natural language generation (NLG). To order a pizza, through the conversation between the user and digital assistant 106, the digital assistant may guide the user to provide all the necessary information for ordering a pizza, and then, at the end of the conversation, may cause the pizza to be ordered. Digital assistant 106 may end the conversation by outputting to the user information indicating that the pizza has been ordered.

[0020] At a conceptual level, digital assistant 106 performs various processes in response to an utterance received from the user. In some embodiments, this process involves, for example, understanding the meaning of the input utterance (which may also be referred to as natural language understanding (NLU)), determining the actions to be performed in response to the utterance, causing the actions to be performed if appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc., and involves a series of processing steps or a pipeline of processing steps. The NLU process can include parsing the received input utterance to understand the structure and meaning of the utterance, and refining and restructuring the utterance to develop a more understandable form (e.g., logical form) or structure for the utterance. Generating a response may include using NLG techniques.

[0021] NLU processing performed by a digital assistant such as digital assistant 106 may include various NLP-related processes such as sentence analysis (e.g., tokenization, sorting, part-of-speech tagging for sentences, named entity recognition in sentences, generation of dependency trees to represent sentence structure, sentence splitting into clauses, analysis of individual clauses, resolution of anaphora, and chunking). In some embodiments, the NLU processing or a portion of it is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may perform a portion of the NLU processing using other resources. For example, the syntax and structure of an input utterance may be identified by processing the sentence using parsing, part-of-speech tagging, and / or named entity recognition. In one implementation, for English, parsing, part-of-speech tagging, and named entity recognition, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0022] The various examples provided in this disclosure demonstrate English utterances, but these are meant to be examples only. In certain embodiments, the digital assistant 106 may also process utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes the NLU processing flexible and extensible for each language, including allowing processing in different orders. Language packs may be provided for individual languages, and language packs may register a list of subsystems that can be serviced from the NLU core server.

[0023] Digital assistants such as the digital assistant 106 shown in Figure 1 can be made available or accessible to their user 108 through various different channels, including, but not limited to, a certain application, a social media platform, various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for it, so that it can run on different services simultaneously and be accessed by different services simultaneously.

[0024] A digital assistant or chatbot system typically includes or is associated with one or more skills. In some embodiments, these skills are individual chatbots (referred to as skillbots) configured to interact with the user and fulfill specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, verifying bank accounts, making reservations, and purchasing widgets. For example, in the embodiment shown in Figure 1, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the term “skill” is used synonymously with the term “skillbot.”

[0025] Each skill associated with a digital assistant helps the user complete tasks through conversation with the user, and the conversation may include a combination of text or auditory input provided by the user and responses provided by the skill bot. These responses may take the form of text or auditory messages to the user, and / or use simple user interface elements (e.g., a selection list) that present the user with a choice to make.

[0026] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In one example, a skillbot may be developed by a company and then added to a digital assistant using DABP102. In another example, a skillbot may be developed and created using DABP102 and then added to a digital assistant created using DABP102. In yet another example, DABP102 provides an online digital store (called a "skill store") offering multiple skills aimed at a wide range of tasks. Skills offered through the skill store may also be published to various cloud services. To add skills to a digital assistant generated using DABP102, a DABP102 user can access the skill store via DABP102, select the desired skill, and indicate that the selected skill will be added to the digital assistant created using DABP102. Skills from the Skill Store can be added to a digital assistant either as is or in a modified form (for example, a DABP102 user may select and clone a specific skill bot provided by the Skill Store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP102).

[0027] Various different architectures may be used to implement a digital assistant or chatbot system. For example, in one embodiment, the digital assistant created and deployed using DABP102 may be implemented using a master-bot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in Figure 1, the digital assistant 106 includes a master bot 114 and skill bots 116-1, 116-2, etc., which are child bots of the master bot 114. In certain embodiments, the digital assistant 106 itself is considered to act as the master bot.

[0028] A digital assistant implemented according to a master-sub-bot architecture allows users of the digital assistant to interact with multiple skills through an integrated user interface, i.e., through the master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself. If not, the master bot selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows the user to converse with the digital assistant through a common, single interface while still providing the ability to use several skill bots configured to perform specific tasks. For example, in the case of a digital assistant developed for an enterprise, the digital assistant's master bot can interface with skill bots having specific functions, such as a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), and an HCM bot to perform functions related to human capital management (HCM). Thus, end-users or consumers of digital assistants only need to know how to access the digital assistant through a common master bot interface, with multiple skill bots provided behind the scenes to handle user requests.

[0029] In one embodiment, in a master-bot / child-bot infrastructure, the master bot is configured to recognize a list of available skill bots. The master bot may have access to various available skill bots and, for each skill bot, metadata that identifies the capabilities of each skill bot, including tasks that each skill bot can perform. Upon receiving a user request in the form of an utterance, the master bot is configured to identify or predict from among several available skill bots which particular skill bot is best suited to respond to or process the user request. The master bot then routes the utterance (or a portion of the utterance) to that particular skill bot for further processing. Thus, control flows from the master bot to the skill bots. The master bot may support multiple input and output channels.

[0030] The embodiment shown in Figure 1 illustrates a digital assistant 106 comprising a master bot 114 and skill bots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented using software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), hardware only, or a combination of software and hardware.

[0031] DABP102 provides infrastructure, services, and features that enable DABP102 users to create digital assistants that include one or more skill bots associated with the digital assistant. In some cases, skill bots can be created by cloning existing skill bots, for example, by cloning skill bots provided by the skill store. As mentioned above, DABP102 provides a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP102 users can clone skill bots from the skill store. If necessary, they may modify or customize the cloned skill bots. In some other cases, DABP102 users have created skill bots from scratch using the tools and services provided by DABP102. As mentioned above, the skill store or skill catalog provided by DABP102 may offer multiple skill bots for performing various tasks.

[0032] In a particular embodiment, creating or customizing a skill bot at a higher level includes the following steps: (1) Configure settings for the new skillbot (2) Set one or more intents for SkillBot (3) Set up one or more entities for one or more intents (4) Train SkillBot (5) Create a dialog flow for SkillBot (6) Add custom components to the skillbot as needed (7) Test and deploy the skillbot. The following is a brief explanation of each step.

[0033] (1) Configure settings for a new skillbot - various settings may be configured for the skillbot. For example, a skillbot designer can specify one or more invocation names for the skillbot being created. These invocation names can then be used by the digital assistant user to explicitly invoke the skillbot. For example, a user can explicitly invoke the corresponding skillbot by typing the invocation name into their utterance.

[0034] (2) Setting up one or more intents and associated exemplary utterances for a skillbot - The skillbot designer specifies one or more intents (also called bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes for which the skillbot is trained to infer about input utterances. Upon receiving an utterance, the trained skillbot infers the intent for the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes the appropriate action to respond to the utterance based on the intent inferred for that utterance. In some cases, the intents for a skillbot represent tasks that the skillbot can perform for the user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot trained for a bank, the intents specified for that skillbot may include "CheckBalance," "TransferMoney," and "DepositCheck."

[0035] For each intent defined for a skillbot, the skillbot designer may also provide one or more exemplary utterances that represent that intent. These exemplary utterances are intended to represent utterances that a user may input to the skillbot for that intent. For example, for a balance inquiry intent, exemplary utterances may include "What's my savings account balance?", "How much is in my checking account?", and "How much money do I have in my account?". Thus, various permutations of typical user utterances may be designated as example utterances for an intent.

[0036] Intents and their associated exemplary utterances are used as training data to train a skillbot. Various different training techniques may be used. As a result of this training, a predictive model is generated, which is configured to take utterances as input and output intents inferred about the utterances by the predictive model. In some cases, the input utterances are provided to an intent analysis engine, which is configured to predict or infer intents for the input utterances using the trained model. The skillbot may then take one or more actions based on the inferred intents.

[0037] (3) Set up one or more entities for one or more intents - In some examples, additional context may be required to enable the skillbot to respond appropriately to user utterances. For example, there may be situations where user input utterances resolve to the same intent in the skillbot. For example, in the example above, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same balance inquiry intent, but these utterances are different requests that want different answers. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skillbot, an entity called AccountType that defines values ​​called "checking" and "saving" may enable the skillbot to parse user requests and respond appropriately. In the example above, the utterances resolve to the same intent, but the values ​​associated with the AccountType entity are different for the two utterances. This allows SkillBot to perform potentially different actions for two utterances, even though they resolve to the same intent. One or more entities may be specified for a particular intent set for SkillBot. Thus, entities are used to add context to the intent itself. Entities help to better describe the intent, enabling SkillBot to complete the user request.

[0038] In one embodiment, there are two types of entities: (a) built-in entities provided by DABP102, and (2) custom entities that may be specified by the skillbot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, cycle period, currency, telephone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, the AccountType entity may be defined by the skillbot designer to enable various banking transactions by checking user input for keywords such as current, savings, and credit card.

[0039] (4) Training the SkillBot - The SkillBot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select intents related to the received user input. As described above, the SkillBot must be trained for this purpose. In one embodiment, the SkillBot is trained on intents set for it and exemplary utterances associated with those intents (collectively, the training data), so that the SkillBot can resolve user input utterances to one of the SkillBot's set intents. In a particular embodiment, the SkillBot uses a predictive model that is trained with the training data and enables the SkillBot to identify what the user is saying (or, in some cases, what the user is trying to say). DABP102 provides a variety of different training techniques that can be used by SkillBot designers to train SkillBots, including a variety of machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In one embodiment, a portion of the training data (e.g., 80%) is used to train the SkillBot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes called a trained skillbot) can then be used to process and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for the skillbot. This allows the skillbot to output a response to a user request without needing to update the dialogue definition. Q&A intents are generated in the same way as regular intents. The dialogue flow for Q&A intents may differ from the dialogue flow for regular intents.

[0040] (5) Create a dialogue flow for the skillbot - The dialogue flow specified for the skillbot describes how the skillbot responds when different intents to the skillbot are resolved in response to received user input. The dialogue flow defines the actions or behaviors that the skillbot takes, such as how the skillbot responds to user utterances, how the skillbot prompts the user for input, and how the skillbot returns data. The dialogue flow is like a flowchart that the skillbot follows. The skillbot designer specifies the dialogue flow using a language such as Markdown. In one embodiment, a version of YAML called OBotML can be used to specify the dialogue flow for the skillbot. The dialogue flow definition for the skillbot acts as a model of the conversation itself, allowing the skillbot designer to choreograph the interaction between the skillbot and the user it corresponds to.

[0041] In one embodiment, the SkillBot's dialog flow definition includes three sections: (a) Context section (b) Default transition section (c) State section.

[0042] Context Section - In the context section, skillbot designers can define variables used in the conversation flow. Other variables that may be designated in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that enable skillbot to recognize and persist user preferences.

[0043] Default Transition Section - Transitions for SkillBot can be defined in either a Dialogflow State section or a Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when there are no applicable transitions defined within a state or when the conditions necessary to trigger a state transition are not met. The Default Transition section can be used to define routing that allows SkillBot to smoothly handle unexpected user actions.

[0044] The State Section – Dialog Flow and its associated behavior are defined as a set of transient states that govern the logic within the dialog flow. Each state node in the dialog flow definition designates a component that provides the functionality required for that point in the dialog. In this way, states are built around the components. Each state contains component-specific characteristics and defines transitions to other states that are triggered after the component has been executed.

[0045] Special case scenarios can be handled using the state section. For example, you might want to give a user the option to temporarily leave the first skill they are working on and do something in a second skill within the digital assistant. For instance, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), they might want to jump to a banking skill (e.g., the user wants to make sure they have enough money for the purchase) and then return to the shopping skill to complete their order. To address this, an action in the first skill can be configured to initiate a conversation with a second, different skill within the same digital assistant, and then return to the original flow.

[0046] (6) Adding Custom Components to the SkillBot - As described above, the states specified in the dialog flow for the SkillBot designate the components that provide the necessary functionality corresponding to that state. The components enable the SkillBot to perform the functionality. In one embodiment, DABP102 provides a set of pre-configured components for performing a wide range of functions. The SkillBot designer can select one or more of these pre-configured components and associate them with states in the dialog flow for the SkillBot. The SkillBot designer can also create custom or new components using the tools provided by DABP102 and associate the custom components with one or more states in the dialog flow for the SkillBot.

[0047] (7) Test and deploy SkillBots - DABP102 provides several features that enable SkillBot designers to test SkillBots under development. The SkillBots can then be deployed and included in a digital assistant.

[0048] The above explanation describes how to create a skillbot, but similar techniques can also be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, you can set up built-in system intents for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without calling the skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit: when the user wants to signal to the digital assistant that they wish to end the current conversation or context; (2) Help: when the user is asking for help or direction; (3) Unresolved Intent: for user input that does not well match the Exit intent or Help intent. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information allows the masterbot to select a specific skillbot to process an utterance.

[0049] At the master bot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to process how to route the utterance and the associated conversation. The digital assistant makes this determination using a routing model that can be rule-based, AI-based, or a combination of both. Using the routing model, the digital assistant determines whether the conversation corresponding to the user-input utterance should be routed to a specific skill for processing, processed by the digital assistant or master bot itself according to an embedded system intent, or processed as a different state in the current conversation flow.

[0050] In certain embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skillbot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skillbot corresponding to the invocation name. In such scenarios, the digital assistant can route the user input to the explicitly invoked skillbot for further processing. If there is no specific or explicit invocation, in some embodiments, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intents and skillbots associated with the digital assistant. The scores calculated for a skillbot or system intent represent the likelihood that the user input represents a task or system intent configured for the skillbot to perform. System intents or skillbots whose associated calculated confidence scores exceed a threshold (e.g., Confidence Threshold routing parameter) are selected as candidates for further evaluation. The digital assistant then selects a specific system intent or skillbot from the identified candidates for further processing of the user input utterance. In certain embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill), and a confidence score is determined for each intent. Generally, intents with a confidence score exceeding a threshold (e.g., 70%) are treated as candidate intents. If a specific skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0051] 2.0 Example of a batching system Figure 2 shows a block diagram of an exemplary batch processing system according to one embodiment. Specifically, Figure 2 shows a batch processing system 200 comprising elements for batching training data for training a chatbot, according to various embodiments. The batch processing system may receive an incoming request 202. The incoming request 202 may be a request containing data for generating a batch of training data. The incoming request 202 may be a batch generation request containing raw utterance data, which includes a plurality of proposed training utterances. For example, the batch processing system 200 may be a system operated by a chatbot service. A customer of the chatbot service may send a plurality of training utterances to the batch processing system 200 to obtain training data for training a chatbot to process customer interactions.

[0052] An incoming request 202 may be received in a communication portal 204. The communication portal 204 may be an entity in the batch processing system 200 for processing incoming and outgoing data. The communication portal 204 may include a user communication instruction 206 that determines one or more ways in which the communication portal 204 may interact with a user sending and receiving data to and from the batch processing system 200. The communication portal 204 may further include a batch routing instruction 208 that determines where in the batch processing system 200 the incoming request and / or data associated with the incoming request will be routed. For example, the batch routing instruction 208 may be executed to determine one or more batch generation modules in the batch processing system 200 that will generate batched training data according to the incoming request 202.

[0053] In various embodiments, once the communication portal 204 determines a batch generation entity in the batch processing system 200 to batch training data, the communication portal may forward the raw utterance data 210 from the incoming request 202 to the batch generation unit 212. The batch generation unit 212 may be an entity within the batch processing system 200 for batching training data to train a chatbot. The batch generation unit 212 may include an utterance data pool 214. The utterance data pool 214 may be a storage location within the batch processing system 200 that stores raw utterance data and sorts the utterances based on the associated output category determined. For example, the batch generation unit 212 may receive the raw utterance data 210 in the utterance data pool 214 and sort the raw utterance data into an intent pool based on the intent corresponding to each utterance in the raw utterance data.

[0054] The batch generation unit 212 may further include commands for generating batches based on raw speech data, such as a model analysis command 216. The model analysis command 216 may be a command for analyzing a training model associated with the chatbot to determine one or more aspects of that model. For example, the model analysis command 216 may trigger an analysis of the training model to determine several skills and / or intents known to the model. The batch generation unit 212 may include a metric selection command 218. The metric selection command 218 may be a command for selecting a metric for generating batches of training data. For example, based on the analyzed information about the chatbot training model, the metric selection command 218 may trigger a selection of a metric or distribution for proportional batching of training utterances in a batch. The batch generation unit 212 may include an utterance selection command 220, which is a command for selecting a certain number of utterances from the utterance pool 214 for batching based on the selected distribution metric. The batch generation unit 212 may include a batch generation command 222, which may be a command on how to batch the selected number of utterances into one or more training batches. For example, the batch generation command 222 may generate batch data for each batch that includes a certain number of selected utterances.

[0055] Batch data may be stored in a repository within the batch generation unit 212, such as a batch pool 224. The generated batch data, such as batch data 226, may then be sent back to the communication portal 204 and exported as an outgoing batch 228 for training a chatbot model. The batch processing system 200 may further include a data store 230. The data store 230 may include any number of entities, repositories, or instructions to implement the processes described herein, and may be combined with any of the system entities described above. The data store 230 may include a training data repository 232, which may be a repository containing additional raw or synthetic training data used by the batch processing system 200 to generate training data batches. The data store 230 may include a batch metric repository 234, which may be a repository of metrics or models used to determine the intent distribution for batching a given number of utterances. The data store 230 may include a user data repository 236, which may be a repository of user or customer data that may be used as part of the batch generation process. For example, the user data repository 236 may contain a number of known user chatbot models for analysis according to the model analysis instruction 216.

[0056] Those skilled in the art will understand that the systems and entities shown in Figure 2 may be reconfigured in any manner or configuration that enables the batching of training data according to the embodiments described herein.

[0057] 3.0 Exemplary batching processes and techniques Batching techniques for handling unbalanced training data when training a model are described so that bias is removed from the trained machine learning model when performing inference. In one embodiment, a batching technique is described for resolving bias in predictions made by the model (e.g., intents inferred by a trained intent classifier for an input utterance) by using a technique to balance batches of training data. For example, a skillbot may be trained to infer a certain intent from a set of intents for a given utterance. The training data used to train the skillbot may include examples of utterances and, for each utterance, the intents associated with that utterance. The training data can be biased if the number of training utterances for one intent is much greater than the number of training utterances for another intent. When such training data is used to train a model, the resulting trained model will be biased to infer intents with more training utterance examples. For example, a skill with two intents may be trained with 1000 training utterances. 950 training utterances can correspond to Intent 1, and 50 training utterances can correspond to Intent 2. Therefore, the skill is trained more highly for Intent 1 than for Intent 2. Consequently, the intent classification unit of this skill may incorrectly attribute the context of an actual customer utterance to Intent 1, even when the actual context of the customer utterance corresponds to Intent 2. This is undesirable.

[0058] Imbalance can also arise when training a model to select a particular skill bot from among many skill bots, for example, when a master bot must select a specific skill bot to process a given utterance. The available training data for training the master bot may be imbalanced. This also creates imbalance. When such training data is used to train a master bot, the resulting trained master bot model will be biased to infer skill bots with a greater number of learned training utterance examples. For example, two skills, namely a first skill with two intents and a second skill with one intent, may be trained with 900 training utterances. Each intent may correspond to 300 training utterances, meaning the intent distribution is balanced. However, the first skill is more trainable than the second skill because the former has twice as many training utterances as the latter. Therefore, the master skill bot classification unit may incorrectly attribute the context of a customer utterance to an intent in the first skill, even when the actual context of the customer utterance corresponds to an intent in the second skill. This, too, is undesirable. The process described here reduces and eliminates these biases when batching utterances for training data.

[0059] Figure 3 shows an exemplary process for generating a training batch and obtaining a predictive model according to one embodiment. Specifically, exemplary process 300 shows a flowchart for generating a training batch of unbiased utterances in order to train a model to obtain a predictive model that can interact in a more efficient manner when implementing chatbot functionality. Process 300 begins in step 310 by acquiring multiple utterances. The multiple utterances may be multiple raw utterance data that, when trained on the model, would result in producing a biased trained model. For example, the acquired multiple utterances may be proposed multiple utterances that are acquired as part of an incoming request, such as incoming request 202, and should be unbiased before training can proceed.

[0060] In step 320, the distribution of intents among multiple utterances is determined. The distribution may be determined based on the multiple utterances obtained, metadata associated with the multiple utterances, and a model proposed to train the multiple utterances. For example, the distribution of intents corresponding to multiple utterances obtained may be determined based on the relative amount of each expressed intent in the multiple utterances. In another example, metadata obtained as part of an incoming request may specify one or more desired intent distributions and may be analyzed to influence the determination of the distribution. In yet another example, a known user chatbot model may be obtained from a model repository such as the user data repository 236, and an instruction such as the model analysis instruction 216 may analyze that model to determine the relative distribution of skills and / or intents used by that model.

[0061] In step 330, a batch containing a certain number of utterances is generated based on the distribution of intents. The batch may be generated based on instructions such as the metric instruction 218, the utterance selection instruction 220, and the batch generation instruction 222. The instructions may specify a known subset of the certain number of utterances to be included in the batch, or the relative probability of a subset of the certain number of utterances. For example, a static number of utterances corresponding to a particular intent may be automatically selected to be included in the batch. In another example, the relative probability of inclusion for multiple intents is determined, and utterances corresponding to multiple intents are randomly selected to be included in the batch based on the relative probability distribution of the corresponding multiple intents.

[0062] In step 340, a set of training data is input to the predictive model from a batch. The set of training data may be a subset of a certain number of utterances in a generated batch that are systematically input to the predictive model for training. In various embodiments, the step of inputting training data from a batch includes sending the generated batch to a model generation system that iteratively provides training data from the batch to the predictive model. In step 350, the predictive model is trained with the set of training data input to the model. The model may be trained according to any of the embodiments discussed herein. For example, the model may include an intent classifier that can “learn” the difference between a predicted intent based on analyzing a given utterance and the actual intent corresponding to that utterance. The intent classifier may use these differences to modify parameters or thresholds for intent determination to more closely resemble an ideal model that predicts intents with perfect accuracy.

[0063] In step 360, the predictive model learns relationships within a set of training data. The learned relationships may be between one or more aspects of multiple utterances and corresponding intents of multiple utterances. For example, the intent classifier of a predictive model may use the determined difference between a predicted intent and the actual corresponding intent to model the relationship between an utterance aspect and a possible intent corresponding to the utterance. In a further example, those aspects are strings and contextual probability values ​​that are parsed to determine the overall predicted context of the utterance. The overall predicted context corresponds to possible intents based on the relationships learned by the predictive model. In step 370, a trained predictive model is obtained, which includes multiple model parameters associated with the learned relationships. The trained predictive model may be the predictive model described above after a training period has been completed using batched training data. In various embodiments, the acquired, trained predictive model is used to parse untrained utterances and predict the corresponding intents as part of a chatbot service that interacts with the user.

[0064] 3.1 Uniform Batching Technology This disclosure describes a novel batching technique for eliminating the aforementioned bias, including at least at the intent level. The uniform batching technique is described at the level of a single masterbot, across the distribution of each known intent in the masterbot system. For example, the distribution of utterances may correspond uniformly to each intent in multiple intents based on the total number of intents in multiple intents. The novel batching technique is used to select batches for model training such that training batches are generated by selecting an appropriate number of training examples from the training data within each output category. This ensures that an intent classifier trained on the batch has the potential to score each output category equally.

[0065] As an example, consider a scenario where training data is provided for nine different intents (a-i). In this example, the training data for multiple intents a-i may be unbalanced, such that significantly more training examples are available for intent a than for other intents b-i. For example, consider the following training examples available for each of intents a-i: Intent a = 900 total training utterances Intents b, c, and d each consist of 20 training utterances. Intents e, f, g, h, and i each consist of 8 training utterances. In this example, we assume that a batch of 100 training examples is generated for each training run. Those skilled in the art will recognize that simple random sampling of the training data described above will result in a batch containing significantly more training examples for intent a than for each of the other intents, because training examples from intent a are nine times more likely to be selected, which will lead to a bias towards intent a (in the intent classifier trained using that batch). To correct this, the system may perform uniform equilibration at the intent level. This involves allocating equal portions of the batch size to each intent so that each intent has an equal chance of being selected. For example, the constraints on the training data described above may be generated as follows: Intent ratio: 1 / (total number of intents) = 1 / 9 Therefore, each intent corresponds to a 1 / 9 chance that its corresponding utterance will be selected for batching. As will be recognized by those skilled in the art, setting the above constraints on the selection of training data will eliminate any bias towards any particular intent, since each intent will achieve an equal probability of inclusion in the final batch. In various embodiments, if the number of available batch slots is greater than the number of intents, at least one intent may be selected via its corresponding utterance for inclusion in the batch. In various further embodiments, if the number of batch slots in a batch is not equally divisible among the number of intents, a certain number of batch slots may be reserved in subsequent batches for inclusion of utterances corresponding to any intents that were not fairly distributed in the first batch. As will be recognized by those skilled in the art, setting the above constraints on the selection of training data will eliminate any bias towards any particular intent, and the intents will be represented equally in the batches of training data.

[0066] 3.2 Hierarchical Batching Techniques This disclosure describes a novel batching technique to eliminate the aforementioned bias at both the skillbot level and the masterbot level. The hierarchical equilibration technique is described at both the masterbot level and the skillbot level. For example, the training infrastructure performs multi-hierarchical equilibration, utilizing a first hierarchical equilibration at the masterbot level between different skillbots and a second hierarchical equilibration at the skillbot level between various intents set for the skillbots. The novel batching technique is used to select batches for model training so that training batches are generated by selecting an appropriate number of training examples from the training data within each output category. This ensures that the intent classifier trained on the batch has the potential to score each output category equally.

[0067] As an example, consider a scenario where training data is provided to two separate skill bots (Skill A and Skill B). In this example, Skill A is associated with a set of five different intents (a-e), and Skill B is associated with a set of four different intents (f-i). In this example, the training data for the two skills may be unbalanced, with more training examples available for Skill A than for Skill B. For example, consider the following training examples available for each of the intents for Skill A and Skill B (key: Ni, where "i" identifies the intent, and "N" identifies the number of training utterances available in the training set for intent "i"): Skill A: 5(a) + 100(b) + 100(c) + 295(d) + 500(e) = 1000 (Total training utterances for Skill A) Skill B: 5(f) + 10(g) + 25(h) + 60(i) = 100 (Total training utterances for Skill B) In this example, we assume that a batch of 100 training examples is generated for each training run. Those skilled in the art will recognize that simple random sampling of the above training data will result in a batch containing significantly more training examples for skill A than for skill B, because training examples from skill A are 10 times more likely to be selected, which will lead to a bias towards skill A (in the intent classifier trained using that batch). To correct this, the system may perform a first hierarchical equilibration at the skill level. This involves allocating equal portions of the batch size to each skill so that skill A and skill B have an equal chance of being selected. For example, the constraints on the above training data may be generated as follows: Skill A: 5 / 2000(a) + 100 / 2000(b) + 100 / 2000(c) + 295 / 2000(d) + 500 / 2000(e) = 1 / 2 Skill B: 5 / 200(f) + 10 / 200(g) + 25 / 200(h) + 60 / 200(i) = 1 / 2 As those skilled in the art will recognize, setting the constraints on the selection of training data as described above will eliminate any bias toward any particular skill, since each skill will be represented equally. However, even within a skill, each intent may be represented unbalanced. To address this, the system may perform a second hierarchical equilibration at the intent level. This involves assigning equal proportions to each intent within a particular set of intents for a certain skill, so that all those intents have an equal chance of being selected. For example, the constraints on the training data described above may be adjusted as follows: Skill A: 1 / 10(a) + 1 / 10(b) + 1 / 10(c) + 1 / 10(d) + 1 / 10(e) = 1 / 2 Skill B: 1 / 8(f) + 1 / 8(g) + 1 / 8(h) + 1 / 8(i) = 1 / 2 As will be recognized by those skilled in the art, setting the constraints described above on the selection of training data will eliminate any bias towards any particular intent within a skill, since each intent will be represented equally within the skill. Using the constraints described above, an appropriate number of training examples can be calculated for each intent, which is done by multiplying the batch size by the portion of the batch allocated to each skill and then by the ratio for that intent. For example, when calculating the appropriate number of training examples to be included in a batch of 100 training examples for intent e of skill A, we multiply the batch size (e.g., 100) by the portion of the batch allocated to each skill (e.g., 1 / 2) and then by the ratio allocated to that intent (1 / 10). In this example, 100 * 1 / 2 * 1 / 10 = 5. Thus, a batch of 100 training examples is generated so that it contains 5 training examples along with the output category for intent e of skill A. Note that when generating the batch, the 5 training examples may be randomly selected from the available training examples for intent e of skill A.

[0068] 3.3 Scaled Batching Techniques This disclosure describes a novel batching technique for eliminating the bias described above according to a universally scaled factor. The scaled balancing technique enables the advantages of the uniform and hierarchical balancing described above and includes additional bias control according to a scaled factor or function. For example, the training infrastructure may perform proportional balancing as described above, or it may utilize scaled rates and probabilities according to a defined function provided by the model user or model administrator. The scaling factor allows for the preservation of specific characteristics of multiple raw training utterances while simultaneously reducing harmful biases.

[0069] As an example, consider a scenario where training data is provided for five separate intents, intents a-e. In this example, intent "a" is the intent corresponding to the model's primary use case and is expected to correlate with approximately one-third of all utterances analyzed by the model. In this example, the training data for the five intents may be unbalanced, with far more training examples available for intent "a" than for intents b-e. For example, consider the following training examples available for each of the intents: Intent a = 100 total training utterances Intent b = 30 total training utterances Intent c = 20 total training utterances Intent d = 10 total training utterances Intent e = 5 total training utterances In this example, we assume that a batch of 100 training examples is generated for each training run. Those skilled in the art will recognize that simple random sampling of the above training data will result in a batch containing significantly more training examples for intent a than for each of the other intents, because training examples from intent a are at least three times more likely to be selected than any other individual intent, which will lead to a bias towards intent a (in the intent classifier trained using that batch). However, since intent a is the primary use case and should be expected to be selected for approximately one-third of the time spent analyzing utterances, a uniform batching technique can completely eliminate this feature when training intent a equally with each other intent. To correct existing biases while maintaining some similarity to the original distribution, the system may perform scaled batching. This involves allocating a portion of the batch size to each intent according to a functional transformation of the data, so that each intent has a contextually correct chance of being selected for the batch. For example, a certain log-scaling technique may transform the constraints on the above training data as follows: Biased raw speech distribution (x) = [100, 30, 20, 10, 5] Logarithmic distribution = log(x) / sum(log(x)) = [0.31, 0.23, 0.20, 0.15, 0.11] As those skilled in the art will recognize, setting the constraints described above on the selection of training data will reduce bias towards any particular intent that is overrepresented in the provided raw utterance pool. However, each intent may still be represented proportionally to a model being trained on the data, according to some relative importance. For example, knowing that intent "a" is expected to be ideally selected as corresponding to a certain utterance for about one-third of the time the model is running, we can compare the following distribution of proportional selection for intent "a": Speech training (a) = 100 / (100 + 30 + 20 + 10 + 5) = ~3 / 5 Uniform batching (a) = 1 / 5 Logarithmic distribution (a) = ~1 / 3 Therefore, the log-scaled batching technique described above reduces the bias inherent in the raw training data while maintaining the relative training importance given to intent "a," which may be beneficial to the model. As will be recognized by those skilled in the art, the log-scaled function technique described above is merely one example of a scaling technique for batching training data, and any transformation, function, or scaling technique may be used in addition to or instead of this particular technique.

[0070] 3.4 Weighted Batching Technology This disclosure describes a novel batching technique for removing the bias described above according to a weighted distribution. The weighted distribution technique enables the advantages of the uniform and hierarchical balancing described above and includes additional bias control by a specified weighted distribution. For example, the training infrastructure may perform proportional balancing as described above, or it may utilize a separate weight distribution provided by the model user or model administrator. The weighted distribution allows for the preservation of specific characteristics of multiple raw training utterances while simultaneously reducing harmful biases.

[0071] As an example, consider the scenario presented in the above-mentioned scaled batching technique where training data is provided for five separate intents, a-e: Biased raw speech distribution x=[100,30,20,10,5] In this example, we assume that a batch of 100 training examples is generated for each training run. Those skilled in the art will recognize that simple random sampling of the above training data will result in a batch containing significantly more training examples for intent a than for each of the other intents, because training examples from intent "a" are at least three times more likely to be selected than any other individual intent, which will result in a bias towards intent a (in the intent classifier trained using that batch). However, a customer or service may provide a weight distribution corresponding to the relative importance of training specific intents. In one example, a customer may provide a weighted distribution according to the relative importance of intent recognition for a given model. In another example, a service maintaining a model on behalf of a customer may analyze a model that receives batch training data as input to determine the expected relative importance of a particular intent to other intents. For example, a customer may send a weighted distribution to the batch system describing the relative importance of weights ranging from 0 to 1 according to the customer, as follows: Weighted importance distribution y = [0.4, 0.3, 0.3, 0.1, 0.1] The batching technique may apply a weighted distribution to another batching technique described herein to form a composite weighted batching distribution. For example, the weighted distribution may be used to transform a uniform distribution into a weighted uniform distribution: Weighted uniform distribution = (x*y) / sum(x*y)=[(0.2*0.4),(0.2*0.3),(0.2*0.3),(0.2*0.1),(0.2*0.1)] / 0.24=[0.33,0.25,0.25,0.08,0.08] As those skilled in the art will recognize, setting the constraints described above on the selection of training data will reduce any bias towards any particular intent that is overrepresented in the provided raw utterance pool. However, each intent may still be represented proportionally to a model trained on the data according to some relative importance, as specified by a given weight distribution. For example, knowing that intent "a" is given a relative weight of 0.4 in a weight distribution, and assuming that the sum of the given weight distributions totals 1.2, the following possible representations of intent "a" in a batch are given according to the following technique: Speech training (a) = 100 / (100 + 30 + 20 + 10 + 5) = ~3 / 5 Uniform batching (a) = 1 / 5 Weighted distribution (a) = ~1 / 3 Therefore, the weighted distribution batching technique described above reduces the bias inherent in the raw training data while maintaining the relative training importance given to intent "a," which may be beneficial to the model. As those skilled in the art will recognize, the addition of a weighted distribution function to the existing uniform batching technique described above is merely one example of a composite weighting technique for batching training data, and any transformation, function, or scaling technique may be used in addition to or instead of this particular technique.

[0072] 3.5 Tailed Batting Technology This disclosure describes a tail batching technique for eliminating the aforementioned bias while enabling training for at least each intent included in a master bot model. The tailed balancing technique enables a combined approach for proportional and static representation of intents in batches, allowing at least a certain number of utterances to be included in a batch, with the remaining batch slots filled proportionally. For example, a novel tail batching technique is used to select batches for model training such that any intent with a selection probability below a certain threshold is included in the generated batch in the smallest possible volume. This ensures that each intent is represented in a batch, but that the batch is still determined largely proportionally.

[0073] As an example, consider a scenario in which training data is provided for six intents a-f. The represented intents may be distributed across multiple raw utterances in the following respects: Intent distribution = [100, 50, 45, 3, 1, 1] In this example, we assume that a batch of 25 training examples is generated for each training run. Those skilled in the art will recognize that simple random sampling of the above training data will result in a batch containing significantly more training examples of intent "a" than of intents "d", "e", and "f". To correct this, the system may perform proportional balancing techniques to reduce bias at the intent level. However, batching using techniques such as uniform proportional balancing would assign the same selection probability to intent "f" as to intent "a", even though intent "a" is represented more than 100 times more often in the raw utterance data. To eliminate bias towards overrepresented utterances without reducing efficiency by proportionally selecting a relatively small utterance pool, tail batching techniques allow the inclusion of minimal training utterances in a batch when the proportional representation of the batch is below a threshold amount. For example, certain utterances that fall below a specific threshold of selection probability are automatically included in the batch in minimal amounts. All remaining utterances above the threshold may then be distributed between batches according to proportional selection techniques. For example, the raw probabilities from the raw speech data above correspond to the following distribution: Raw speech training distribution = [0.5, 0.25, 0.23, 0.01, 0.05, 0.05] Given a tailed threshold of 0.01, each of intents a-d will satisfy or exceed the tailed threshold, while intents e and f will each fall below it. Various techniques may be used to specify that any intent below the threshold should have at least one corresponding utterance in any batch of training data. In this case, one slot in a batch of 25 utterances will contain one utterance corresponding to intents e and f. The remaining 23 utterances will be filled proportionally among the remaining four intents. An exemplary distribution for batches generated using tail batching techniques may be as follows: Training batch intent representation = [11,6,5,1,1,1] As those skilled in the art will recognize, setting the constraints described above on the selection of training data will eliminate any bias toward any particular intent. However, intents that would ordinarily almost never be selected in proportional techniques may be represented in the batch without requiring the overrepresentation of these less represented intents in the batch.

[0074] 3.6 Automated Mixing Batching Technology In various embodiments, two or more of the techniques described herein may be used to batch one or more training utterances for training a model. In various embodiments, two or more techniques may be combined to batch multiple training batches. In various embodiments, a first single technique may be used to generate a first batch, a second single technique may be used to generate a second batch, and the first and second batches may be combined into a composite batch. In various embodiments, the techniques described herein may be provided to a hypertuner entity as parameters to determine the best batching technique for a particular raw utterance dataset. For example, the hypertuner may perform multiple interactions of batching techniques on the same raw utterance dataset and compare the resulting batches to determine the most efficient batching style. The hypertuner itself may be a model that can be trained using input training data to iteratively determine and learn the best batching practice according to several metrics and techniques available to the hypertuner as well as a set of raw utterance inputs.

[0075] 4.0 Additional Exemplary Batching Embodiments Figure 4A shows an exemplary embodiment for generating training batches from a training data pool according to one embodiment. Specifically, Figure 4A shows the environment before generating batches of training data using the hierarchical batching technique described herein. As shown in Figure 4A, a data pool such as utterance data pool 214 contains a number of raw utterances. Raw utterances such as utterance 400 may contain metadata corresponding to skills and / or intents known to correspond to the utterance. For example, utterance 400 contains skill metadata 410 indicating that utterance 400 corresponds to skill "A". Utterance 400 further contains intent metadata 420 indicating that utterance 400 corresponds to intent "1" within skill "A". In various embodiments not shown in Figure 4A, utterances such as utterance 400 may further contain metadata corresponding to weights for the corresponding utterances, skills, or intents, which should be used as part of a weighted batching technique such as the one described herein.

[0076] The intent distribution 430 is shown in Figure 4A as a chart displaying possible skill-intent correspondences between ratios. Although the intent distribution 430 is shown as a simplified table in Figure 4A, those skilled in the art will recognize that the intent distribution may take any form necessary to perform or assist in the performance of the process described herein. A batched data pool, such as batch pool 224, may comprise batch slots 440. The batch slots 440 may be a subset of the storage within batch pool 224 that accepts utterances as part of the batching process.

[0077] Figure 4B shows an exemplary embodiment for generating training batches from a training data pool according to one embodiment. Specifically, Figure 4B shows the environment after batches of training data have been generated using the hierarchical batching technique described herein. As shown in Figure 4B, a number of utterances are selected from the utterance data pool 214 and copied into batch slots in the batch pool 224. The intent distribution 430 is updated to include ratio information for the number of skills and the number of intents for each skill. For example, the intent distribution for each skill "A" and "B" includes a corresponding skill ratio of "0.5" for each skill, representing an even distribution of utterance probabilities across the number of skills. The intent distribution is further decomposed into intent ratios for each intent of each skill. For example, each of the intents "1", "2", and "3" divides the "0.5" ratio for skill "A" evenly among the intent ratios for equal intent ratios of "0.16". A similar number for skill "B" also includes three intents "4", "5", and "6".

[0078] Figure 4B shows an exemplary embodiment for generating training batches from a training data pool according to one embodiment. Specifically, Figure 4B shows the environment after batches of training data have been generated using the hierarchical batching technique described herein. As shown in Figure 4B, a number of utterances are selected from the utterance data pool 214 and copied into batch slots in the batch pool 224. The intent distribution 430 is updated to include ratio information for the number of skills and the number of intents for each skill. For example, the intent distribution for each skill "A" and "B" includes a corresponding skill ratio of "0.5" for each skill, representing an even distribution of utterance probabilities across the number of skills. The intent distribution is further decomposed into intent ratios for each intent of each skill. For example, each of the intents "1", "2", and "3" divides the "0.5" ratio for skill "A" evenly among the intent ratios for equal intent ratios of "0.16". A similar number for skill "B" also includes three intents "4", "5", and "6".

[0079] As shown in Figure 4B, the number of batch slots is greater than the number of intents recorded in the intent distribution. In this case, batching may include including at least one utterance in the batch pool corresponding to each intent. For example, utterances containing intent metadata corresponding to intents "1", "2", "3", "4", "5", and "6" are randomly selected from the utterance data pool 214 to fill the batch slots. In this case, the remaining batch slots are less than the number of intents. Therefore, the remaining two batch slots may be filled by a random selection of two additional utterances based on the selection probabilities defined in the intent distribution 430. As shown in Figure 4B, the two additional utterances corresponding to intents "1" and "5" are randomly selected to fill these remaining batch slots in batch pool 224. In alternative embodiments not shown in Figure 4B, each batch slot in batch pool 224 may be filled only according to the selection probabilities defined in the intent distribution 430, without including at least one corresponding intent in batch pool 224.

[0080] 5.0 Infrastructure Implementation Examples Figure 5 shows a simplified diagram of a distributed system 500 for realizing one embodiment. In the illustrated embodiment, the distributed system 500 includes one or more client computing devices 502, 504, 506, and 508 connected to a server 512 via one or more communication networks 510. The client computing devices 502, 504, 506, and 508 may be configured to run one or more applications.

[0081] In various embodiments, the server 512 may be adapted to run one or more services or software applications that enable the processing described in this disclosure.

[0082] In some embodiments, the server 512 may also provide other services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided to users of client computing devices 502, 504, 506, and / or 508 as web-based or cloud services, such as under a Software-as-a-Service (SaaS) model. Users operating client computing devices 502, 504, 506, and / or 508 may then use one or more client applications to interact with the server 512 and use the services provided by these components.

[0083] In the configuration shown in Figure 5, server 512 may include one or more components 518, 520, and 522 that implement the functions performed by server 512. These components may include one or more processors, hardware components, or software components that can be executed by a combination thereof. It should be recognized that a wide variety of system configurations are possible, which may differ from the distributed system 500. Therefore, the embodiment shown in Figure 5 is an embodiment of a distributed system for implementing the example system and is not intended to be limiting.

[0084] A user may interact with the server 512 in accordance with the teachings of this disclosure using client computing devices 502, 504, 506, and / or 508. A client device may provide an interface that enables a user of that client device to interact with that client device. A client device may also output information to the user through this interface. Although Figure 5 shows only four client computing devices, any number of client computing devices may be supported.

[0085] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, game systems, thin clients, various messaging devices, sensors or other sensing devices. These computing devices may include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, and various mobile operating systems, such as Google Chrome® OS, including Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, and Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), and personal digital assistants (PDAs). Wearable devices may include Google Glass® head-mounted displays and other devices. The game system may include various handheld game devices and internet-connected game devices (for example, Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, and various game systems provided by Nintendo®). Client devices may be capable of running a wide variety of applications, such as various internet-related applications and communication applications (for example, email applications and short message service (SMS) applications), and may use various communication protocols.

[0086] Network 510 may be any type of network known to those skilled in the art that can support data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. Just as an example, Network 510 may be a local area network (LAN), an Ethernet®-based network, Token Ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (for example, a wireless network operating under any of the IEEE 802.11 protocol suites, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0087] Server 512 may consist of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, rack-mount servers, etc.), server farms, server clusters, or other appropriate configurations and / or combinations. Server 512 may include one or more virtual machines running a virtual operating system, or other computing architectures with virtualization. This could be, for example, one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the server. In various embodiments, Server 512 may be adapted to run one or more services or software applications that provide the functions described above.

[0088] The computing system within server 512 may run one or more operating systems, including any of the above-mentioned operating systems, and commercially available server operating systems. Server 512 may also run any of a variety of other server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Java® servers, and database servers. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and others.

[0089] In some implementations, server 512 may include one or more applications for analyzing and organizing data feeds and / or event updates received from users of client computing devices 502, 504, 506, and 508. For example, data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams, which may include real-time events related to sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc. Server 512 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client computing devices 502, 504, 506, and 508.

[0090] The distributed system 500 may also include one or more data repositories 514, 516. In certain embodiments, these data repositories can be used to store data and other information. For example, one or more of the data repositories 514, 516 may be used to store data / information generated by the processes described herein, and / or data or information used for the processes described herein. The data repositories 514, 516 may reside in various locations. For example, a data repository used by server 512 may be located locally with server 512, or it may be located remotely from server 512 and communicates with server 512 via a network-based connection or a dedicated connection. The data repositories 514, 516 may be of different types. In certain embodiments, a data repository used by server 512 may be a database, for example, a relational database such as a database provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to commands in SQL format.

[0091] In certain embodiments, one or more of the data repositories 514, 516 may be used by the application to store application data. The data repositories used by the application may be of various types, such as a key-value store repository, an object store repository, or a general-purpose storage repository supported by the file system.

[0092] In certain embodiments, the functions described herein may be provided as services through a cloud environment. Figure 6 is a simplified block diagram of a cloud-based system environment that may provide the functions described herein as cloud services according to a particular embodiment. In the embodiment shown in Figure 6, the cloud infrastructure system 602 may provide one or more cloud services that a user may request using one or more client computing devices 604, 606, and 608. The cloud infrastructure system 602 may include one or more computers and / or servers, which may include those described above with respect to server 512. The computers within the cloud infrastructure system 602 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0093] Network 610 can facilitate data communication and exchange between clients 604, 606, and 608 and the cloud infrastructure system 602. Network 610 may include one or more networks. The networks may be of the same type or different types. Network 610 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0094] The embodiment shown in Figure 6 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in several other embodiments, the cloud infrastructure system 602 may have more or fewer components than those shown in Figure 6, may be a combination of two or more components, or may have components in different configurations or arrangements. For example, while Figure 6 shows three client computing devices, any number of client computing devices may be supported in alternative embodiments.

[0095] The term "cloud service" is generally used to refer to services made available to users on demand via communication networks such as the Internet, through a service provider's system (e.g., cloud infrastructure system 602). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use the cloud services provided by the cloud service provider without purchasing separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can order and use applications on demand via the Internet without purchasing infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, some cloud services, such as middleware services, database services, and Java® cloud services, are offered by Oracle Corporation® in Redwood Shores, California.

[0096] In certain embodiments, the cloud infrastructure system 602 may provide one or more cloud services using various models, including a hybrid service model, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, and an Infrastructure as a Service (IaaS) model. The cloud infrastructure system 602 may include a suite of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0097] The SaaS model enables the delivery of applications or software as a service to customers over a communication network such as the internet, without requiring customers to purchase the underlying hardware or software for the application. For example, the SaaS model can be used to allow customers to access on-demand applications hosted on a cloud infrastructure system 602. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0098] The IaaS model is generally used to provide flexible computing and storage capabilities by offering infrastructure resources (such as servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation®.

[0099] The PaaS model is generally used to provide a platform and environmental resources as a service, enabling customers to develop, run, and manage applications and services without having to procure, build, or manage those resources themselves. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0100] Cloud services are generally delivered in an on-demand, self-service, subscription-based, flexibly scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by the cloud infrastructure system 602 through a subscription order. The cloud infrastructure system 602 then provides the services requested in the customer's subscription order by performing processing. For example, in one embodiment, the chatbot-related functionality described herein may be provided as a cloud service that is subscribed to regularly by users / subscribers. The cloud infrastructure system 602 may be configured to provide one cloud service or more cloud services.

[0101] The cloud infrastructure system 602 can provide cloud services through various deployment models. In a public cloud model, the cloud infrastructure system 602 may be owned by a third-party cloud service provider, and the cloud services are provided to a general public customer. This customer may be an individual or a company. In another embodiment, under a private cloud model, the cloud infrastructure system 602 may function within an organization (for example, within a corporate organization), and the services are provided to customers within this organization. For example, this customer may be various departments within the company, such as the human resources department or the payroll department, or an individual within the company. In yet another embodiment, under a community cloud model, the cloud infrastructure system 602 and the services provided may be shared among various organizations within a relevant community. Other models, such as hybrid models of the above models, may also be used.

[0102] The client computing devices 604, 606, and 608 may be of different types (for example, devices 502, 504, 506, and 508 shown in Figure 5) and may be capable of operating one or more client applications. Users can interact with the cloud infrastructure system 602, such as requesting services provided by the cloud infrastructure system 602, by using the client devices. For example, a user can use a client device to request the chatbot-related services described in this disclosure.

[0103] In some embodiments, the processing performed by the cloud infrastructure system 602 may include big data analysis. This analysis may include using, analyzing, and processing large datasets to detect and visualize various trends, behaviors, relationships, etc., within the data. This analysis may be performed by one or more processors, which may, in some cases, process the data in parallel and run simulations using the data. The data used for this analysis may include structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0104] As shown in the embodiment of Figure 6, the cloud infrastructure system 602 may include infrastructure resources 630 used to facilitate the provision of various cloud services provided by the cloud infrastructure system 602. The infrastructure resources 630 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0105] In certain embodiments, resources may be grouped into resource sets or resource modules (also referred to as "pods") to facilitate the efficient provisioning of these resources to support various cloud services provided by the cloud infrastructure system 602 to different customers. Each resource module or pod may contain a pre-integrated and optimized combination of one or more types of resources. In certain embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for a database service, and a second set of pods, which may contain different resource combinations than the pods in the first set, may be provisioned for a Java service, etc. For several services, resources allocated to provisioning these services may be shared among the services.

[0106] The cloud infrastructure system 602 itself may internally use services 632 that are shared by different components of the cloud infrastructure system 602 and facilitate the provisioning of services by the cloud infrastructure system 602. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services that enable cloud support, email services, notification services, and file transfer services.

[0107] The cloud infrastructure system 602 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in Figure 6, the subsystem may include a user interface subsystem 612 that enables users or customers of the cloud infrastructure system 602 to interact with the cloud infrastructure system 602. The user interface subsystem 612 may include various different interfaces, such as a web interface 614, an online store interface 616 where cloud services offered by the cloud infrastructure system 602 are advertised and available for purchase by consumers, and other interfaces 618. For example, a customer may use a client device to request (service request 634) one or more services that the cloud infrastructure system 602 offers using one or more of the interfaces 614, 616, and 618. For example, a customer may access an online store, browse the cloud services offered by the cloud infrastructure system 602, and place a subscription order for one or more services offered by the cloud infrastructure system 602 that the customer wishes to subscribe to. This service request may include information identifying the customer and the one or more services that the customer wishes to subscribe to.

[0108] In certain embodiments, such as the embodiment shown in Figure 6, the cloud infrastructure system 602 may include an order management subsystem (OMS) 620 configured to process new orders. As part of this process, the OMS 620 may be configured to prepare the order for provisioning by creating a customer account if one does not already exist, receiving billing and / or account information from the customer to be used to charge the customer in order to provide the requested services to the customer, verifying the customer information, reserving the order for the customer after verification, and coordinating various workflows.

[0109] If properly validated, OMS620 may invoke an Order Provisioning Subsystem (OPS)624 configured to provision resources for this order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring those resources to facilitate the services requested by the customer order. The way resources are provisioned for an order and the types of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a certain workflow, OPS624 may be configured to determine the specific cloud service being requested and identify the number of pods that would have been pre-configured for that particular cloud service. The number of pods allocated for an order may depend on the size / volume / level / scope of the requested service. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to suit the specific customer making the request in order to provide the requested service.

[0110] The cloud infrastructure system 602 may send a response or notification 644 to the requesting customer to indicate when the requested service will be available. In some examples, the customer may be sent information (e.g., a link) that will enable the customer to begin using and utilizing the benefits of the requested service.

[0111] The cloud infrastructure system 602 may provide services to multiple customers. For each customer, the cloud infrastructure system 602 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and provides the requested services to the customer. The cloud infrastructure system 602 may also collect usage statistics about the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information may be used to charge customers. Billing may be done, for example, on a monthly basis.

[0112] The cloud infrastructure system 602 may provide services to multiple customers in parallel. The cloud infrastructure system 602 may store information about these customers, which may include copyright information. In certain embodiments, the cloud infrastructure system 602 includes an identity management subsystem (IMS) 628 configured to manage customer information and separate the managed information so that information about one customer cannot be accessed from information about another customer. The IMS 628 may be configured to provide various security-related services, such as identity services including information access management, authentication and authorization services, and services for managing customer identities and roles and related capabilities.

[0113] Figure 7 shows an exemplary computer system 700 that may be used to implement a particular embodiment. For example, in some embodiments, the computer system 700 may be used to implement either the system or subsystems of a chatbot system, as well as the various servers and computer systems described above. As shown in Figure 7, the computer system 700 includes various subsystems, including a processing subsystem 704 that communicates with several other subsystems via a bus subsystem 702. These other subsystems may include a processing acceleration unit 706, an I / O subsystem 708, a storage subsystem 718, and a communication subsystem 724. The storage subsystem 718 may include a non-temporary computer-readable storage medium including a storage medium 722 and system memory 710.

[0114] The bus subsystem 702 provides a mechanism for various components and subsystems of the computer system 700 to communicate with each other as intended. Although the bus subsystem 702 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 702 may be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, local bus, etc., using one of various bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard.

[0115] The processing subsystem 704 controls the operation of the computer system 700 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 700 can be organized into one or more processing units 732, 734, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 704 may include one or more dedicated coprocessors, such as graphics processors or digital signal processors (DSPs). In some embodiments, some or all of the processing units in the processing subsystem 704 may use customized circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0116] In some embodiments, processing units within the processing subsystem 704 may execute instructions stored in system memory 710 or a computer-readable storage medium 722. In various embodiments, processing units may execute various program or code instructions and maintain multiple programs or processes running concurrently. At any given time, some or all of the program code to be executed may reside in system memory 710 and / or a computer-readable storage medium 722, potentially containing one or more storage devices. Through appropriate programming, the processing subsystem 704 may provide the various functions described above. In an example where the computer system 700 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0117] In certain embodiments, a processing acceleration unit 706 may be optionally provided to perform customized processing to accelerate the overall processing performed by the computer system 700, or to offload a portion of the processing performed by the processing subsystem 704.

[0118] The I / O subsystem 708 may include devices and mechanisms for inputting information into and / or outputting information from or through the computer system 700. Generally, the use of the term “input device” is intended to include all conceivable types of devices and mechanisms for inputting information into the computer system 700. User interface input devices may include, for example, pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors, Microsoft Xbox® 360 game controllers, and devices that provide interfaces for receiving input using gestures and voice commands, enabling users to control and interact with input devices. The user interface input device may also include an eye gesture recognition device, such as the Google Glass® blink detector, which detects eye movements from the user (e.g., blinking while taking a picture and / or making a menu selection) and translates the eye gestures into input to an input device (e.g., Google Glass®). The user interface input device may also include a voice recognition sensing device that enables the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0119] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as auditory / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0120] Generally, the use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from the computer system 700 to a user or another computer. User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, planning devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0121] The storage subsystem 718 provides a repository or datastore for storing information and data used by the computer system 700. The storage subsystem 718 provides a tangible, non-temporary, computer-readable storage medium for storing basic programming and data configurations that provide functionality in some embodiments. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 704, provides the functionality described above may be stored in the storage subsystem 718. The software may be executed by one or more processing units of the processing subsystem 704. The storage subsystem 718 may also provide a repository for storing data used in accordance with the teachings of this disclosure.

[0122] The storage subsystem 718 may include one or more non-temporary memory devices, including volatile and non-volatile memory devices. As shown in Figure 7, the storage subsystem 718 includes system memory 710 and computer-readable storage medium 722. The system memory 710 may include several memories, including volatile primary random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS), which includes basic routines to assist in the transfer of information between elements within the computer system 700, such as during startup, may typically be stored in ROM. Typically, the RAM contains data and / or program modules currently being operated and executed by the processing subsystem 704. In some implementations, the system memory 710 may include several different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0123] As an example, without limitation, as shown in Figure 7, the system memory 710 may load a running application program 712, program data 714, and an operating system 716, which may include various applications such as a web browser, a middle-tier application, and a relational database management system (RDBMS). As an example, the operating system 716 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems and Google Chrome® OS), and / or various versions of mobile operating systems such as iOS®, Windows Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0124] The computer-readable storage medium 722 can store programming and data structures that provide functionality in several embodiments. The computer-readable storage medium 722 can provide storage for computer-readable instructions, data structures, program modules, and other data for the computer system 700. Software (programs, code modules, instructions) that provides the above functionality when executed by the processing subsystem 704 may be stored in the storage subsystem 718. As an example, the computer-readable storage medium 722 may include non-volatile memory such as a hard disk drive, magnetic disk drive, optical disk drive such as a CD-ROM, DVD, or Blu-ray® disc, or other optical media. The computer-readable storage medium 722 may also include, but is not limited to, a Zip® drive, flash memory card, Universal Serial Bus (USB) flash drive, Secure Digital (SD) card, DVD disc, digital videotape, and the like. Computer-readable storage media 722 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, and solid-state ROMs; SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM; DRAM-based SSDs; magnetoresistive RAM (MRAM) SSDs; and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0125] In certain embodiments, the storage subsystem 718 may also include a computer-readable storage medium reader 720 that can be further connected to the computer-readable storage medium 722. The reader 720 may be configured to receive and read data from a memory device such as a disk or flash drive.

[0126] In certain embodiments, the computer system 700 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, the computer system 700 may provide support for running one or more virtual machines. In certain embodiments, the computer system 700 may run programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing resources (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating system run by other virtual machines run by the computer system 700. Thus, potentially multiple operating systems can run simultaneously by the computer system 700.

[0127] The communication subsystem 724 provides interfaces to other computer systems and networks. The communication subsystem 724 functions as an interface for sending and receiving data between other systems and the computer system 700. For example, the communication subsystem 724 may enable the computer system 700 to establish communication channels to one or more client devices via the internet in order to send and receive information with one or more client devices.

[0128] The communication subsystem 724 may support both wired and / or wireless communication protocols. For example, in one embodiment, the communication subsystem 724 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, 3G, 4G, or EDGE (High Speed ​​Data Rate for Global Evolution)), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communication subsystem 724 may provide wired network connectivity (e.g., Ethernet®) in addition to or instead of a wireless interface.

[0129] The communication subsystem 724 can receive and transmit data in various formats. For example, in some embodiments, the communication subsystem 724 may receive input communications in formats such as structured data feeds and / or unstructured data feeds 726, event streams 728, and event updates 730, in addition to other formats. For example, the communication subsystem 724 may be configured to receive (or transmit) data feeds 726 in real time from users of other communication services such as social media networks and / or web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0130] In certain embodiments, the communication subsystem 724 may be configured to receive data in the form of a continuous data stream, which may include an event stream 728 and / or event update 730 of real-time events that are inherently continuous or infinite and do not have a clear end. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0131] The communication subsystem 724 may be configured to transmit data from the computer system 700 to other computer systems or networks. This data may be transmitted in various different formats, such as structured and / or unstructured data feeds 726, event streams 728, event updates 730, etc., to one or more databases that can communicate with one or more streaming data source computers connected to the computer system 700.

[0132] The computer system 700 may be one of many types, including handheld portable devices (e.g., iPhone® cellular phone, iPad® computing tablet, PDA), wearable devices (e.g., Google Glass® head-mounted display), personal computers, workstations, mainframes, kiosks, server racks, or other data processing systems. Because the nature of computers and networks is constantly changing, the description of the computer system 700 shown in Figure 7 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in Figure 7. Based on the disclosures and teachings herein, those skilled in the art will understand other aspects and / or methods for realizing various embodiments.

[0133] While specific embodiments have been described, various modifications, changes, alternative configurations, and equivalents are possible. The embodiments are not limited to operation within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, while specific embodiments have been described using a specific set of transactions and steps, it should be apparent to those skilled in the art that this is not intended as limitation. Some flowcharts describe operations as sequential processes, but many of these operations may be performed in parallel or concurrently. In addition, the order of operations may be re-specified. Processes may have additional steps not shown in the diagrams. The various features and aspects of the embodiments described above may be used individually or together.

[0134] Furthermore, while specific embodiments have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are also possible. A particular embodiment may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0135] Where it is stated that a device, system, component, or module is configured to perform a particular operation or function, such configuration can be achieved, for example, by designing electronic circuits to perform the operation; by programming programmable electronic circuits (such as a microprocessor) to perform the operation; by executing computer instructions or code, or any combination thereof, stored in a non-temporary memory medium, or by executing a processor or core. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0136] This disclosure provides specific details to ensure that embodiments are fully understood. However, embodiments can be implemented without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary details to avoid ambiguity in the embodiments. This specification provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the above description of embodiments provides explanations that enable various embodiments to be implemented by those skilled in the art. Various modifications are possible within the scope of the function and configuration of the elements.

[0137] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be clear that additions, reductions, deletions, and other modifications and changes can be made to them without departing from the broader spirit and scope set forth in the claims. Thus, specific embodiments have been described, but these are not intended to be limiting. Various variations and equivalents are within the scope of the appended claims.

Claims

1. A method performed by one or more data processors, The method includes acquiring multiple utterances, each utterance including metadata indicating the skill and intent corresponding to that utterance, each utterance corresponding to one of the multiple intents, each intent of the multiple intents corresponding to one of one or more skills, and the method further includes The method includes determining the intent distribution across the plurality of utterances based on the intent indicated by the metadata contained in each utterance, wherein the intent distribution includes a plurality of intent proportions, each of which corresponds to one of the plurality of intents, and the method further includes: A batch is generated containing a number of utterances selected from the plurality of utterances based on the plurality of intent ratios, The method further includes iteratively inputting a set of training data from the batch into a predictive model for one or more skills, wherein the predictive model is constructed as an intent classifier including a plurality of model parameters learned by the use of an objective function, and the method further includes The method includes training the predictive model on the training data set by minimizing or maximizing the objective function, training by measuring the difference between the predicted intent and the ground truth intent, and further, In response to the training, learn the relationships within the set of training data used by the predictive model to generate the predicted intent, Obtaining a trained predictive model that includes the multiple model parameters associated with the learned relationship, The process further includes generating multiple intent pools, each intent pool corresponding to a specific intent among the multiple intents, and including each utterance among the multiple intents corresponding to the specific intent, and each intent percentage of the multiple intents corresponding to a specific intent pool, and generating the batch includes selecting the number of utterances from the multiple intent pools based on the multiple intent percentages, The intent distribution is a uniform distribution, and the proportion of each intent is a uniform ratio calculated from the number of utterances in the batch and the number of intents in the plurality of intents. The method further comprises selecting each particular utterance of the number of utterances to be included in the batch based on a specific intent corresponding to the particular utterance and the uniform ratio.

2. A method performed by one or more data processors, The method includes acquiring multiple utterances, each utterance including metadata indicating the skill and intent corresponding to that utterance, each utterance corresponding to one of the multiple intents, each intent of the multiple intents corresponding to one of one or more skills, and the method further includes The method includes determining the intent distribution across the plurality of utterances based on the intent indicated by the metadata contained in each utterance, wherein the intent distribution includes a plurality of intent proportions, each of which corresponds to one of the plurality of intents, and the method further includes: A batch is generated containing a number of utterances selected from the plurality of utterances based on the plurality of intent ratios, The method further includes iteratively inputting a set of training data from the batch into a predictive model for one or more skills, wherein the predictive model is constructed as an intent classifier including a plurality of model parameters learned by the use of an objective function, and the method further includes The method includes training the predictive model on the training data set by minimizing or maximizing the objective function, training by measuring the difference between the predicted intent and the ground truth intent, and further, In response to the training, learn the relationships within the set of training data used by the predictive model to generate the predicted intent, Obtaining a trained predictive model that includes the multiple model parameters associated with the learned relationship, The process further includes generating multiple intent pools, each intent pool corresponding to a specific intent among the multiple intents, and including each utterance among the multiple intents corresponding to the specific intent, and each intent percentage of the multiple intents corresponding to a specific intent pool, and generating the batch includes selecting the number of utterances from the multiple intent pools based on the multiple intent percentages, The intent distribution is a stratified distribution, where each intent ratio corresponds to a certain stratified ratio among a plurality of stratified ratios, and each stratified ratio includes ratio information about the number of skills in one or more skills and the number of intents corresponding to each skill. The ratio information includes, for each skill, a skill ratio showing an equal ratio between skills, and for each intent, an intent ratio obtained by equally dividing the skill ratio of that skill among the intents of the skills corresponding to that intent. The method further includes selecting each specific utterance of the number of utterances to be included in the batch based on a specific intent corresponding to the specific utterance, a specific skill corresponding to the specific intent, and a specific intent ratio corresponding to the specific intent.

3. A method performed by one or more data processors, The method includes acquiring multiple utterances, each utterance including metadata indicating the skill and intent corresponding to that utterance, each utterance corresponding to one of the multiple intents, each intent of the multiple intents corresponding to one of one or more skills, and the method further includes The method includes determining the intent distribution across the plurality of utterances based on the intent indicated by the metadata contained in each utterance, wherein the intent distribution includes a plurality of intent proportions, each of which corresponds to one of the plurality of intents, and the method further includes: A batch is generated containing a number of utterances selected from the plurality of utterances based on the plurality of intent ratios, The method further includes iteratively inputting a set of training data from the batch into a predictive model for one or more skills, wherein the predictive model is constructed as an intent classifier including a plurality of model parameters learned by the use of an objective function, and the method further includes The method includes training the predictive model on the training data set by minimizing or maximizing the objective function, training by measuring the difference between the predicted intent and the ground truth intent, and further, In response to the training, learn the relationships within the set of training data used by the predictive model to generate the predicted intent, Obtaining a trained predictive model that includes the multiple model parameters associated with the learned relationship, The process further includes generating multiple intent pools, each intent pool corresponding to a specific intent among the multiple intents, and including each utterance among the multiple intents corresponding to the specific intent, and each intent percentage of the multiple intents corresponding to a specific intent pool, and generating the batch includes selecting the number of utterances from the multiple intent pools based on the multiple intent percentages, The intent distribution is a tail distribution, where each intent ratio corresponds to a tail ratio among a plurality of tail ratios, each tail ratio is calculated from the proportion of intent utterances corresponding to that tail ratio in the plurality of utterances, and the method further, Identifying one or more intents corresponding to an intent percentage below the intent threshold, To include in the aforementioned batch, select at least one utterance corresponding to each of one or more intents corresponding to an intent percentage below the intent threshold, A method comprising selecting, from the aforementioned number of utterances, the remaining number of utterances excluding the at least one selected utterance, for inclusion in the batch, utterances of other intents excluding the one or more intents, based on the intent proportion corresponding to those other intents.

4. A method performed by one or more data processors, The method includes acquiring multiple utterances, each utterance including metadata indicating the skill and intent corresponding to that utterance, each utterance corresponding to one of the multiple intents, each intent of the multiple intents corresponding to one of one or more skills, and the method further includes The method includes determining the intent distribution across the plurality of utterances based on the intent indicated by the metadata contained in each utterance, wherein the intent distribution includes a plurality of intent proportions, each of which corresponds to one of the plurality of intents, and the method further includes: A batch is generated containing a number of utterances selected from the plurality of utterances based on the plurality of intent ratios, The method further includes iteratively inputting a set of training data from the batch into a predictive model for one or more skills, wherein the predictive model is constructed as an intent classifier including a plurality of model parameters learned by the use of an objective function, and the method further includes The method includes training the predictive model on the training data set by minimizing or maximizing the objective function, training by measuring the difference between the predicted intent and the ground truth intent, and further, In response to the training, learn the relationships within the set of training data used by the predictive model to generate the predicted intent, Obtaining a trained predictive model that includes the multiple model parameters associated with the learned relationship, The process further includes generating multiple intent pools, each intent pool corresponding to a specific intent among the multiple intents, and including each utterance among the multiple intents corresponding to the specific intent, and each intent percentage of the multiple intents corresponding to a specific intent pool, and generating the batch includes selecting the number of utterances from the multiple intent pools based on the multiple intent percentages, The intent distribution is a weighted distribution, and each intent proportion is a weighted ratio, each weighted ratio is calculated from the number of utterances corresponding to a particular intent in the plurality of utterances and the total number of utterances in the plurality of utterances, and multiplied by a specific weight value from a plurality of weight values. The method further comprises selecting each particular utterance to be included in the batch based on a specific intent corresponding to the particular utterance and a specific intent ratio corresponding to the particular intent.

5. The method according to claim 4, wherein the plurality of weight values ​​are partially based on the frequency distribution of the intents corresponding to the plurality of utterances.

6. The method according to claim 4, wherein each of the plurality of weight values ​​is partially based on the average string length of all utterances among the plurality of utterances that correspond to a specific intent.

7. A method performed by one or more data processors, The method includes acquiring multiple utterances, each utterance including metadata indicating the skill and intent corresponding to that utterance, each utterance corresponding to one of the multiple intents, each intent of the multiple intents corresponding to one of one or more skills, and the method further includes The method includes determining the intent distribution across the plurality of utterances based on the intent indicated by the metadata contained in each utterance, wherein the intent distribution includes a plurality of intent proportions, each of which corresponds to one of the plurality of intents, and the method further includes: A batch is generated containing a number of utterances selected from the plurality of utterances based on the plurality of intent ratios, The method further includes iteratively inputting a set of training data from the batch into a predictive model for one or more skills, wherein the predictive model is constructed as an intent classifier including a plurality of model parameters learned by the use of an objective function, and the method further includes The method includes training the predictive model on the training data set by minimizing or maximizing the objective function, training by measuring the difference between the predicted intent and the ground truth intent, and further, In response to the training, learn the relationships within the set of training data used by the predictive model to generate the predicted intent, Obtaining a trained predictive model that includes the multiple model parameters associated with the learned relationship, The process further includes generating multiple intent pools, each intent pool corresponding to a specific intent among the multiple intents, and including each utterance among the multiple intents corresponding to the specific intent, and each intent percentage of the multiple intents corresponding to a specific intent pool, and generating the batch includes selecting the number of utterances from the multiple intent pools based on the multiple intent percentages, Each intent ratio is a batch ratio, and each batch ratio is calculated from the number of utterances corresponding to a particular intent in the plurality of utterances and the number of utterances that should be included in the batch, and the method further, It is determined that at least one of the aforementioned multiple intent percentages does not meet or exceed the threshold, A method comprising increasing the number of utterances to be included in the batch in response to determining that at least one of the plurality of intent percentages does not meet or exceed a threshold.

8. The method according to any one of claims 1 to 7, wherein each utterance selected from the plurality of intent pools is randomly selected from the corresponding intent pool.

9. It is a system, One or more data processors, The system comprises a non-temporary computer-readable storage medium containing instructions, wherein, when the instructions are executed on one or more data processors, the one or more data processors cause the data processors to perform an operation, and the operation is This includes acquiring multiple utterances, each utterance including metadata indicating the skill and intent corresponding to that utterance, each utterance corresponding to one of the multiple intents, each intent of the multiple intents corresponding to one of the skills, and the operation further includes, The operation includes determining the intent distribution across the plurality of utterances based on the intent indicated by the metadata contained in each utterance, wherein the intent distribution includes a plurality of intent proportions, each of which corresponds to one of the plurality of intents, and the operation further includes: A batch is generated containing a number of utterances selected from the plurality of utterances based on the plurality of intent ratios, The operation further includes iteratively inputting a set of training data from the batch into a predictive model for one or more skills, wherein the predictive model is constructed as an intent classifier including a plurality of model parameters learned by the use of an objective function, and the operation further includes The operation includes training the predictive model on the set of training data by minimizing or maximizing the objective function, which measures the difference between the predicted intent and the ground truth intent, and further, In response to the training, learn the relationships within the set of training data used by the predictive model to generate the predicted intent, Obtaining a trained predictive model that includes the multiple model parameters associated with the learned relationship, The process includes generating multiple intent pools, each intent pool corresponding to a specific intent among the multiple intents, and including each utterance among the multiple intents corresponding to the specific intent, and each intent percentage of the multiple intents corresponding to a specific intent pool, and generating the batch includes selecting the number of utterances from the multiple intent pools based on the multiple intent percentages, Furthermore, when executed on one or more data processors, the instruction includes instructions that cause one or more data processors to perform an operation, and in the operation, Each intent ratio is a batch ratio, and each batch ratio is calculated from the number of utterances corresponding to a particular intent in the plurality of utterances and the number of utterances that should be included in the batch, and the operation further, It is determined that at least one of the aforementioned multiple intent percentages does not meet or exceed the threshold, A system that includes increasing the number of utterances to be included in the batch in response to determining that at least one of the plurality of intent percentages does not meet or exceed a threshold.

10. The system according to claim 9, wherein each utterance selected from the plurality of intent pools is randomly selected from the corresponding intent pool.

11. A program for causing one or more processors to perform the method according to any one of claims 1 to 8.

12. It is a system, One or more processors, A system comprising a computer-readable storage medium containing instructions, wherein, when the instructions are executed on one or more processors, the system causes the one or more processors to perform the method according to any one of claims 1 to 8.

Images