Out-of-Domain Data Augmentation for Natural Language Processing

Abstract

This provides a method for extending the domain with out-of-domain data to train a chatbot system. [Solution] The method includes the steps of receiving a training set of utterances 505 and expanding the training set of utterances with OOD examples 510. The expansion step includes generating a dataset of OOD examples, filtering out OOD examples from the dataset of OOD examples, determining a difficulty value for each OOD example remaining in the filtered dataset of OOD examples, and generating a batch of expanded utterances that include utterances from the training set of utterances and utterances from the filtered dataset of OOD examples based on the difficulty value for each OOD. The method also includes the step of training a machine learning model with the batch of expanded utterances according to a curriculum training protocol 515.

Description

Technical Field

[0001] Cross-reference of related applications This application claims the benefit and priority of U.S. Provisional Application No. 63 / 119,526, filed on November 30, 2020, and the benefit and priority of U.S. Non-Provisional Application No. 17 / 452,743, filed on October 28, 2021, the entire content of which is incorporated herein by reference for all purposes.

[0002] Field The present disclosure generally relates to chatbot systems, and more specifically, to techniques for out-of-domain (O OD) data augmentation for training chatbot systems in natural language processing.

Background Art

[0003] Background Many users around the world are present on instant messaging or chat platforms to obtain instant responses. Organizations often use these instant messaging or chat platforms to participate in live conversations with customers (or end users). However, hiring service personnel to participate in live communication with customers or end users can be very costly for organizations. Chatbots or bots have begun to be developed to simulate conversations with end users, especially on the Internet. End users can communicate with bots via messaging apps that they have already installed and used. Generally, intelligent bots driven by artificial intelligence (AI) are In live conversations, this allows for more intelligent and contextual communication, potentially enabling more natural conversations between bots and end-users for an improved conversational experience. Instead of end-users learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots may be able to understand the end-user's intent based on their natural language utterances and respond accordingly.

[0004] However, building chatbots is difficult because these automated solutions require specific knowledge in a particular field and the application of specific technologies that can only exist within the capabilities of expert developers. As part of building such a chatbot, developers may first understand the needs of the company and end users. Then, developers may analyze and perform decisions related to, for example, selecting the dataset to be used for analysis, preparing the input dataset for analysis (e.g., data cleansing, pre-analysis data extraction, formatting and / or transformation, performing data feature engineering, etc.), identifying appropriate machine learning (ML) techniques or models for performing the analysis, and improving the techniques or models to improve results / outcomes based on feedback. The task of identifying appropriate models may include developing multiple models, possibly in parallel, and identifying a particular model for use after iteratively testing and experimenting with these models. Furthermore, supervised learning-based solutions typically include a training phase, a subsequent application (i.e., inference) phase, and an iterative loop between the training and application phases. Developers may be responsible for carefully implementing and monitoring these phases to achieve an optimal solution. For example, to train an ML technique or model, the ML technique or model predicts a desired outcome (e.g., inferring intent from utterances). Accurate training data is required to enable algorithms to understand and learn specific patterns or features that will be used (for example, in the case of a chatbot, intent extraction and careful parsing, rather than simply raw language processing). To ensure that ML techniques or models properly learn these patterns and features, developers may be responsible for selecting, expanding, and optimizing the set of training data for the ML techniques or models. [Overview of the Initiative]

[0005] overview The technology disclosed herein relates, in general, to chatbots. More specifically, and not limited to, the technology disclosed herein relates to a technology for OOD data augmentation of training data sets for training chatbot systems in natural language processing. A chatbot can classify user utterances into various classes, such as predefined user intents. The chatbot's classifier may include a trained ML model that produces outputs (e.g., intents) based on inputs (e.g., user utterances). User utterances may take the form of speech. In this case, the trained ML model can be understood as improving speech recognition, where speech recognition enables more accurate identification of user intents. If the training data used to train the trained ML model is inadequate, the chatbot may more frequently determine incorrect intents. The technology disclosed herein can provide OOD-augmented datasets for training ML models so that the ML model is more robust to irrelevant contexts and learns more accurately patterns or boundaries of domain or OOD intents.

[0006] In various embodiments, a computer-implemented method is provided which includes: a data processing system receiving a training set of utterances for training a machine learning model to identify one or more intents for one or more utterances; and the data processing system extending the training set of utterances with out-of-domain (OOD) examples. The extension step includes the steps of generating a dataset of OOD examples, filtering out OOD examples from the dataset of OOD examples that have a context substantially similar to the context of the utterances in the training set of utterances, determining a difficulty value for each OOD example remaining in the filtered dataset of OOD examples, generating a batch of extended utterances that includes utterances from the training set of utterances and utterances from the filtered dataset of OOD examples based on the difficulty value for each OOD, and training a machine learning model using the batch of extended utterances by a data processing system, the training step including supplying the batch of extended utterances to the machine learning model based on the difficulty value of the OOD example in each batch of the batch of extended utterances.

[0007] In some embodiments, the substantial similarity between the context of an OOD example and the context of an utterance in a training set of utterances is determined based on a distance scale to avoid collisions between classes.

[0008] In some embodiments, the substantial similarity between the context of the OOD example and the context of the utterance in the training set of the utterance is determined by the Multilingual Universal Sentence Encoder (MUSE) Determined using embedding, if min(d_i) < predetermined threshold (d_i = Euclidean distance (v_i,u)), then the context of the OOD example and the tone of the utterance The context of the utterance in question within the training set is determined to be substantially similar.

[0009] In some embodiments, the difficulty value is derived from the d_i value such that easy OOD examples have a high d_i value, while difficult OOD examples have a lower d_i value that is closer to the predetermined threshold.

[0010] In some embodiments, the batch of augmented utterances is generated subject to the following constraints: (i) a predetermined batch size; (ii) inclusion of a predetermined number of in-domain examples of utterances from the training set of the utterances in each batch; (iii) inclusion of a predetermined number of OOD examples of utterances from the filtered OOD dataset in each batch; (iv) the predetermined number of in-domain examples and the predetermined number of OOD examples of utterances are selected to maintain the predetermined batch size; (v) the OOD examples of utterances included in each batch have substantially similar difficulty values; and optionally, (vi) each ODD example from the filtered OOD dataset is included in only one batch of the augmented utterances.

[0011] In some embodiments, the method further includes the step of deploying a trained machine learning model in a chatbot system.

[0012] In some embodiments, batches of the extended utterances are generated using a batch balancing scheme.

[0013] In various embodiments, a computer-implemented method is provided. The computer-implemented method includes the steps of: a chatbot system receiving utterances generated by a user interacting with the chatbot system; and classifying the utterances into intent categories corresponding to intents using an intent classifier deployed within the chatbot system, the intent classifier including a number of model parameters identified using training data, the training data including a training set of augmented utterances for training the intent classifier to identify one or more intents for one or more utterances, the training set of augmented utterances being artificially generated to include augmented utterances from the training set of utterances, the augmented utterances being trained A good example of a speech utterance has a substantially different context from that of an utterance in the training set. The substantial difference between the context of the OOD example and the context of the utterance in the training set is determined using a Multilingual Universal Sentence Encoder (MUSE) single embedding, and if min(d_i) => a predetermined threshold (d_i = Euclidean distance (v_i, u)), then the context of the OOD example and the context of the utterance in the training set are determined to be substantially different. Multiple model parameters are identified using the training data based on minimizing the loss function. The computer-implemented method further includes the step of using an intent classifier to output intents based on the classification.

[0014] In various embodiments, a system is provided that includes one or more data processors and a non-temporary computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods disclosed herein.

[0015] In various embodiments, the computer is tangibly embodied in a non-temporary machine-readable storage medium. A computer program product is provided. This computer program product includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0016] The techniques described above and below can be implemented in several ways and in several situations. Several exemplary implementations and situations are provided with reference to the attached diagrams, which will be described in more detail below. However, the following implementations and situations are only a few examples of many. [Brief explanation of the drawing]

[0017] [Figure 1] This is a simplified block diagram showing a distributed environment incorporating an exemplary embodiment. [Figure 2] This is a simplified block diagram showing a computing system that implements a master bot according to a specific embodiment. [Figure 3] This is a simplified block diagram showing a computing system that implements a skillbot according to a specific embodiment. [Figure 4] This is a simplified block diagram illustrating chatbot training and deployment systems according to various embodiments. [Figure 5] This figure shows the process flow for expanding a training dataset with OOD examples according to various embodiments. [Figure 6] This is a simplified diagram illustrating a distributed system for realizing various embodiments. [Figure 7] This is a simplified block diagram showing one or more components of a system environment in which a service provided by one or more components of a system according to various embodiments may be provided as a cloud service. [Figure 8] This figure shows an exemplary computer system that can be used to realize various embodiments. [Modes for carrying out the invention]

[0018] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. It will be apparent, however, that the various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or designs.

[0019] introduction A digital assistant is an artificial intelligence-driven interface that helps users accomplish various tasks in natural language conversations. For each digital assistant, a customer can assemble one or more skills. Skills (also described herein as chatbots, bots, or skillbots) are individual bots that focus on specific types of tasks such as inventory tracking, time card submission, and expense report creation. When an end user engages with a digital assistant, the digital assistant evaluates the end user input and routes the conversation to the appropriate chatbot. The digital assistant can be made available to the end user via various channels such as FACEBOOK (registered trademark) Messenger, SKYPE MOBILE (registered trademark) Messenger, or Short Message Service (SMS). The channel enables chat to flow back and forth between the end user and the digital assistant and its various chatbots on various messaging platforms. The channel may also support progressive expansion of the user agent, event-triggered conversations, and testing.

[0020] An intent allows a chatbot to understand what the user wants it to do. An intent can be thought of as a command given by a user to a computer (e.g., in voice form) with a degree of understanding that corresponds to the accuracy with which the user's utterance (e.g., a command or instruction, sometimes delivered verbally) is captured. This is directly related to whether the computer's response to the command or instruction is appropriate, because the better the command or instruction is understood, the better the response from the computer will be. In this context, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. An intent may be created by providing a name that indicates some user action (e.g., ordering a pizza) and compiling a set of actual user statements or utterances that are generally associated with triggering that action. Since the chatbot's cognition is derived from these intents, each intent can be created and modified from a dataset that is robust (1 to 20 or so utterances) so that the chatbot can interpret ambiguous user input. A rich set of utterances allows the chatbot to understand what the user wants when it receives messages that mean the same thing but are expressed differently, such as "Ignore this order!" or "Cancel the delivery!". Collectively, intents and the utterances belonging to them constitute a training corpus for the chatbot. By training a machine learning model with the corpus, customers can transform that machine learning model into a reference tool for essentially breaking down end-user input into single intents. Customers can improve the chatbot's cognitive agility through a cycle of intent testing and intent training.

[0021] However, building a chatbot that can determine an end-user's intent based on user utterances is a challenging task, partly due to the nuances and ambiguities of natural language, as well as the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (number of intents). Specific examples of this difficulty arise from the characteristics of natural language, such as employing euphemisms, synonyms, or ungrammatical language to represent intents. For example, an utterance might express the intent to order a pizza without explicitly mentioning pizza, order, or delivery. For instance, in a particular local language, "pizza" might be called "pie." These tendencies, such as inaccuracies or variability in natural language, lead to uncertainty, prompting the introduction of confidence as a parameter for predicting intents, as opposed to explicitly indicating intents by including keywords, for example. Therefore, it may be necessary to train, monitor, debug, and retrain chatbots to improve their performance and the user experience they provide. Conventional systems provide training systems for training and retraining machine learning models for digital assistants or chatbots in Spoken Language Understanding (SLU) and Natural Language Processing (NLP). Traditionally, models used in chatbot systems are trained in NLP with "manufactured" utterances for arbitrary intents. For example, the utterance "Do you do price changes?" may be used to train the chatbot system's classifier model to classify this type of utterance as the intent "Do you offer a price match?" Training the model with manufactured utterances initially helps train the chatbot system to provide the service, and then the chatbot system may be retrained once it is deployed and begins to receive actual utterances from users.

[0022] Traditional training of text classification models begins with a training dataset of utterances labeled with ground truth intent. However, utterances The training dataset typically lacks out-of-domain (OOD) test utterances for training the model to classify OOD utterances as unresolved intents. Specifically, machine learning models may have a tendency to overestimate or overestimate in their assessment of in-domain intents. The lack of OOD test utterances during training can lead to false positives during the testing or deployment execution of the trained machine learning model. For example, it has been observed that some out-of-domain (OOD) test utterances can be misclassified under in-domain intents because the OOD test utterances that would otherwise trigger a learning attraction to in-domain intents are missing during training.

[0023] Therefore, a different approach is needed to address these problems. In order for a machine learning model to better learn and classify utterances for precise OOD intents, such as unresolved intents, the approach described herein attempts to train the machine learning model by augmenting the training data with OOD examples (negative data augmentation). These OOD examples would be labeled for precise intents or ground truths such as unresolved intents. As a result, the machine learning model will better generalize and learn that OOD utterances should be classified as precise intents or ground truths such as unresolved intents. In various embodiments, a method is provided that includes the steps of: receiving a training set of utterances in a data processing system for training a machine learning model to identify one or more intents for one or more utterances; and the data processing system augmenting the training set of utterances with out-of-domain (OOD) examples. The extension steps include generating a dataset of OOD examples; filtering out OOD examples from the OOD example dataset that have contexts substantially similar to the contexts of the utterances in the training set of utterances; determining a difficulty value for each OOD example remaining in the filtered OOD example dataset; and generating an extended batch of utterances, based on the difficulty value for each OOD, which includes utterances from the training set of utterances and utterances from the filtered OOD example dataset, wherein the method further includes training an intent classifier using the extended batch of utterances by a data processing system, which training steps include feeding the extended batch of utterances to a machine learning model, based on the difficulty values ​​of the OOD examples in each batch of the extended batch of utterances.

[0024] Bot and analytics systems A bot (also known as a skill, chatbot, chatterbot, or talkbot) is a computer program that can engage in conversations with end users. Bots can generally respond to natural language messages (e.g., questions or comments) through messaging applications that use natural language messaging. Businesses may use one or more bot systems to communicate with end users through messaging applications. These messaging applications, sometimes called channels, can be the end user's preferred messaging application, one they have already installed and are familiar with. Therefore, end users do not need to download and install a new application to chat with the bot system. Examples of messaging applications include over-the-top (OTT) messaging channels (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), and virtual private assistants (e.g., Amazon Dot, Echo, or Show, Google® Home, Apple HomePod, etc.). ), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat functionality, or voice-based input (e.g., Siri, Cortana, Google Voice, or other voice input for dialogue) This may include devices or applications that have an interface to use.

[0025] In some examples, a bot system may be associated with a Uniform Resource Identifier (URI). A URI may identify a bot system using a string. A URI may be used as a webhook for one or more messaging application systems. A URI may include, for example, a Uniform Resource Location Specifier (URL) or a Uniform Resource Name (URN). A bot system may be designed to receive messages (e.g., Hypertext Transfer Protocol (HTTP) post-call messages) from messaging application systems. HTTP post-call messages may be directed from messaging application systems to URIs. In some embodiments, messages may differ from HTTP post-call messages. For example, a bot system may receive messages from the Short Message Service (SMS). While this description may refer to communications that a bot system receives as messages, it should be understood that messages can be HTTP post-call messages, SMS messages, or any other type of communication between two systems.

[0026] End users can interact with bot systems through conversational dialogue (sometimes called a conversational user interface (UI)), similar to how people interact with each other. In some cases, the dialogue might involve the end user saying "Hello" to the bot, and the bot responding... The bot may respond with "Hi" and ask the end user how it can assist them. In some cases, the conversation may also include a transactional conversation with a bank bot, such as transferring money from one account to another; an informational conversation with an HR bot, such as matching a vacation balance; or a conversation with a retail bot, such as explaining a return of a purchased item or requesting technical support.

[0027] In some embodiments, a bot system may intelligently handle end-user interactions without interaction with the bot system's administrator or developers. For example, an end-user may send one or more messages to the bot system to achieve a desired goal. The messages may include specific content such as text, emojis, voice, images, video, or other methods of conveying the message. In some embodiments, the bot system converts the content into a standardized format (e.g., a REST (representational state transfer) call to an enterprise service with appropriate parameters) and a natural language response. It may generate. The bot system may also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system may also initiate communication with the end user rather than passively responding to the end user utterance. Various techniques for identifying explicit calls to a bot system and determining input to the called bot system are described here. In certain embodiments, explicit call analysis is performed by the master bot based on the detection of a call name in the utterance. In response to the detection of a call name, the utterance may be refined for input to the skill bot associated with the call name.

[0028] A conversation with a bot may follow a specific conversational flow that includes multiple states. The flow may define what happens next based on the input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in or depending on the state. A conversation may take various paths based on end-user input, which can influence the decisions the bot makes about the flow. For example, in each state, based on end-user input or utterance, the bot may determine the end-user intent and decide on the appropriate next action. As used herein and in the context of utterances, the term “intent” refers to the intent of the user who gave the utterance. For example, a user might intend to engage the bot in a conversation to order a pizza, and for this reason, the user The intent of the user can be expressed by an utterance such as "Order a pizza." The user's intent can be directed towards a specific task that the user wants the chatbot to perform on their behalf. Therefore, utterances can be expressed as questions, commands, requests, etc., that reflect the user's intent. The intent may include goals that the end user would like to achieve.

[0029] In the context of chatbot configuration, the term "intent" is used here to refer to configuration information for mapping user utterances to specific tasks / actions or categories of tasks / actions that the chatbot can perform. To distinguish between utterance intents (i.e., user intents) and chatbot intents, the latter are sometimes referred to here as "bot intents." A bot intent may include a set of one or more utterances associated with that intent. For example, an intent about ordering a pizza might have various permutations of utterances expressing the desire to order a pizza. These associated utterances can be used to train the chatbot's intent classifier, which can then determine whether an input utterance from the user matches the pizza order intent. A bot intent may also be associated with one or more dialogue flows for initiating a conversation with the user in a particular state. For example, the first message for a pizza order intent might be the question, "What kind of pizza would you like?" In addition to associated utterances, a bot intent may also include named entities associated with that intent. For example, a pizza order intent might include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, and pizza quantity. The entity's values ​​are typically obtained through conversation with the user.

[0030] Figure 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to a specific embodiment. The environment 100 includes a digital assistant builder platform (DABP) 102, which enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, as shown in Figure 1, a user 104 representing a particular company can use the DABP 102 to create and deploy a digital assistant 106 for users of that particular company. For example, a bank can use the DABP 102 to create one or more digital assistants for use by its customers. Multiple companies can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) may use the DABP 102 to create and deploy a digital assistant that enables restaurant customers to order food (e.g., order pizza).

[0031] For the purposes of this disclosure, “digital assistant” is an entity that helps users of the digital assistant accomplish various tasks through natural language conversation. A digital assistant may be implemented using only software (for example, a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, and vehicles. A digital assistant is also sometimes referred to as a chatbot system. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0032] A digital assistant such as the digital assistant 106, built using DABP102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user may provide one or more user inputs 110 to the digital assistant 106 and receive responses 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and responses 112. Through these conversations, the user can request the digital assistant 106 to perform one or more tasks, and in response, the digital assistant 106 is configured to perform the user-requested tasks and respond to the user with appropriate responses.

[0033] User input 110 is generally in natural language form and is called utterance. User utterance 110 can be in text form, such as when the user types a sentence, a question, a piece of text, or even a single word and provides it to the digital assistant 106 as input. In some embodiments, user utterance 110 can be in voice input or utterance form, such as when the user says or speaks something that is provided to the digital assistant 106 as input. Utterance is typically a language spoken by the user 108. For example, utterance may be English or some other language. If utterance is in voice form, voice input is converted to a text-form utterance in that particular language, and the text utterance is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert voice or auditory input to text utterances, which are then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion may be performed by the digital assistant 106 itself.

[0034] Utterances, which may be text or voice utterances, may include fragments, sentences, multiple sentences, one or more words, one or more questions, or combinations of the aforementioned types. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to utterances in order to understand the meaning of user input. As part of the NLU processing of an utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. Once the meaning of the utterance is understood, the digital assistant 106 may perform one or more actions or behaviors in accordance with the understood meaning or intent. For the purposes of this disclosure, we assume that utterances are either text utterances directly provided by the user 108 of the digital assistant 106, or the result of converting input voice utterances into text format. However, this is not intended to be limiting or restrictive in any way.

[0035] For example, user 108's input may be a request to order a pizza by providing an utterance such as "I want to order a pizza." Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. Appropriate action may include responding to the user with a question requesting user input, such as the type of pizza the user wants to order, the size of the pizza, and any toppings. The responses provided by the digital assistant 106 may also be in natural language form and may typically be in the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). In order for the user to order a pizza, the digital assistant 106 may guide the user through a conversation to provide all the necessary information to order the pizza, and then, at the end of the conversation, may have the user order the pizza. The digital assistant 106 may terminate the conversation by outputting information to the user indicating that the pizza has been ordered.

[0036] At a conceptual level, the digital assistant 106 responds to utterances received from the user. Various processes are performed. In some embodiments, this process involves a series of processing steps or a pipeline of processing steps, which may include, for example, understanding the meaning of an input utterance (sometimes called natural language understanding (NLU)), determining what actions should be taken in response to the utterance, performing the actions as appropriate, generating a response to be output to the user in response to a user utterance, and outputting the response to the user. NLU processing may include parsing the received input utterance to understand its structure and meaning, and refining and reconstructing the utterance to develop a more understandable form (e.g., logical form) or structure for the utterance. Generating a response may include using NLG techniques.

[0037] NLU processing performed by a digital assistant such as digital assistant 106 may include various NLP-related processes such as sentence analysis (e.g., tokenization, headword creation, identification of part-of-speech tags for a sentence, identification of named entities in a sentence, generation of dependency trees to represent sentence structure, splitting of a sentence into clauses, analysis of individual clauses, resolution of anaphora, and chunking). In certain embodiments, the NLU processing or a portion of it is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may perform a portion of the NLU processing using other resources. For example, the syntax and structure of an input utterance may be identified by processing the sentence using a parser, a part-of-speech tagger, and / or a named entity recognition unit. In one implementation, for English, a parser, a part-of-speech tagger, and a named entity recognition unit, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0038] The various examples provided in this disclosure demonstrate utterances in English, but this is intended only as examples. In certain embodiments, the digital assistant 106 may also process utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for various languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes the NLU processing flexible and extensible for each language, including enabling processing in various sequences. Language packs may be provided for individual languages, and language packs may register a list of subsystems that can be serviced from the NLU core server.

[0039] Digital assistants such as the digital assistant 106 shown in Figure 1 can be made available or accessible to their user 108 through various channels, including, but not limited to, specific applications, social media platforms, various messaging services and applications, and other applications or channels. A single digital assistant may have several channels configured for it, so that it can run on various services simultaneously and be accessed simultaneously by various services.

[0040] A digital assistant or chatbot system generally includes or is associated with one or more skills. In certain embodiments, these skills are individual chatbots (referred to as skillbots) configured to interact with the user and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, verifying bank accounts, making reservations, and purchasing widgets. For example, in the embodiment shown in Figure 1, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the term “skill” is used synonymously with the term “skillbot.”

[0041] Each skill associated with a digital assistant helps the user complete tasks through conversation with the user, in which case the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may take the form of text or voice messages to the user, and / or use simple user interface elements (e.g., selection lists) presented to the user so that the user can make a choice.

[0042] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In some cases, skillbots may be developed by a company and then added to a digital assistant using DABP102. In other examples, skillbots may be developed and created using DABP102 and then added to a digital assistant created using DABP102. In yet another example, DABP102 provides an online digital store (called a "skill store") offering multiple skills aimed at a wide range of tasks. Skills offered through the skill store may also expose various cloud services. To add skills to a digital assistant generated using DABP102, a DABP102 user may access the skill store via DABP102, select the desired skill, and indicate that the selected skill will be added to the digital assistant created using DABP102. Skills from the Skill Store can be added to a digital assistant either as is or in a modified form (for example, a DABP102 user may select and clone a specific skill bot provided by the Skill Store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP102).

[0043] Various different architectures may be used to implement a digital assistant or chatbot system. For example, in certain embodiments, the digital assistant created and deployed using DABP102 may be implemented using a master-bot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in Figure 1, the digital assistant 106 includes a master bot 114 and skill bots 116-1, 116-2, etc., which are child bots of the master bot 114. In certain embodiments, the digital assistant 106 itself is considered to act as the master bot.

[0044] A digital assistant implemented according to a master-sub-bot architecture allows users of the digital assistant to interact with multiple skills through an integrated user interface, i.e., through the master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself; if not, the master bot selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows users to converse with the digital assistant through a common, single interface while still retaining the ability to use several skill bots configured to perform specific tasks. For example, in the case of a digital assistant developed for an enterprise, the digital assistant's master bot may interface with skill bots having specific functions, such as a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), and an HCM bot to perform functions related to human capital management (HCM). End users or consumers of digital assistants only need to know how to access the digital assistant through a common master bot interface, with multiple skill bots provided behind the scenes to handle user requests.

[0045] In certain embodiments, within a master-bot / child-bot infrastructure, the master bot is configured to recognize a list of available skill bots. The master bot may have access to metadata that identifies the various available skill bots and, for each skill bot, the capabilities of each skill bot, including the tasks that each skill bot can perform. Upon receiving a user request in the form of an utterance, the master bot is configured to identify or predict from among several available skill bots which particular skill bot is best suited to respond to or process the user request. The master bot then routes the utterance (or a portion of the utterance) to that particular skill bot for further processing. Thus, control flows from the master bot to the skill bots. The master bot may support multiple input and output channels. In certain embodiments, routing may be performed with the help of processing carried out by one or more available skill bots. For example, as described below, a skill bot may be trained to infer the intent of an utterance and determine whether the inferred intent matches an intent on which the skill bot is configured. Therefore, routing performed by the master bot may involve the skill hot communicating with the master bot whether the skill hot is composed of an intent suitable for processing the utterance.

[0046] The embodiment shown in Figure 1 illustrates a digital assistant 106 comprising a master bot 114 and skill bots 116-1, 116-2, and 116-3, but is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented in software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), in hardware only, or in implementations using a combination of software and hardware.

[0047] DABP102 provides infrastructure, services, and features that enable DABP102 users to create digital assistants that include one or more skill bots associated with the digital assistant. In some cases, skill bots can be created by cloning existing skill bots, for example, by cloning skill bots provided by the skill store. As mentioned above, DABP102 provides a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP102 users can clone skill bots from the skill store. If necessary, they may modify or customize the cloned skill bots. In some other cases, DABP102 users have created skill bots from scratch using the tools and services provided by DABP102. As mentioned above, the skill store or skill catalog provided by DABP102 may offer multiple skill bots for performing various tasks.

[0048] In certain embodiments, creating or customizing a skill bot at a higher level involves the following steps: (1) Steps to configure settings for a new skillbot (2) Steps to configure one or more intents with respect to SkillBot (3) Steps to create one or more entities with respect to one or more intents (4) Steps to train SkillBot (5) Steps to create a dialogue flow for SkillBot (6) Steps to add custom components to the skillbot as needed (7) Steps to test and deploy SkillBot The following is a brief explanation of each of the processes mentioned above.

[0049] (1) A step to configure settings for a new skillbot. Various settings may be configured for the skillbot. For example, a skillbot designer may specify one or more invocation names for the skillbot being created. These invocation names can then be used by users of the digital assistant to explicitly invoke the skillbot. For example, a user may explicitly invoke the corresponding skillbot by typing the invocation name into their utterance.

[0050] (2) The step of setting up one or more intents and associated exemplary utterances for a skillbot. The skillbot designer specifies one or more intents (also called bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes for which the skillbot is trained to infer about input utterances. Upon receiving an utterance, the trained skillbot infers an intent for the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes the appropriate action for the utterance based on the intent inferred for that utterance. In some cases, the intents for a skillbot represent tasks that the skillbot can perform for the user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot trained on banking, the intents specified for that skillbot would be "CheckBalance," "TransferMoney," and "DepositCheck." This may include, etc.

[0051] For each intent defined for a skillbot, the skillbot designer may also provide one or more exemplary utterances that represent and exemplify that intent. These exemplary utterances are intended to represent utterances that a user might input to the skillbot for that intent. For example, for the balance inquiry intent, exemplary utterances might be "What's my savings account balance?" and "How much is "How much money do I have in my checking account?", "How much money do I have in my account?", etc. This may include various permutations of typical user utterances. Therefore, various permutations of typical user utterances may be specified as utterance examples related to intents.

[0052] Intents and their associated exemplary utterances are used as training data to train a skillbot. Various different training techniques may be used. As a result of this training, a predictive model is generated, which is configured to take utterances as input and output intents inferred by the predictive model about the utterances. In some cases, the input utterances are provided to an intent analysis engine configured to predict or infer intents for the input utterances using the trained model. The skillbot may then take one or more actions based on the inferred intents.

[0053] (3) Steps to configure entities with respect to one or more intents of the skillbot. In some cases, to enable the skillbot to respond appropriately to user utterances. Additional context may be needed. For example, there may be situations where user input utterances translate to the same intent in a skill bot. For example, in the example above, the utterances "What's my savings account balance?" and "How much is in my checking account?" both Although both are intents for checking a balance, these utterances are separate requests that ask for different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skills bot, an entity called AccountType defines values ​​called "checking" and "saving." This can enable the skillbot to analyze user requests and respond appropriately. In the example above, the utterances are the same intent, but they are associated with the AccountType entity. The values ​​are different for the two utterances. This allows the skillbot to perform different actions for the two utterances, even though they share the same intent. One or more entities may be specified for a particular intent configured for the skillbot. Thus, entities are used to add context to the intent itself. Entities help to better describe the intent, enabling the skillbot to complete the user request.

[0054] In certain embodiments, there are two types of entities: (a) built-in entities provided by DABP102, and (2) custom entities that may be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, cycle period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, in the case of a banking skill, the AccountType entity... Titi may be defined by the skillbot designer to enable various banking transactions by checking user input for keywords such as current, savings, and credit card.

[0055] (4) Step of training the skillbot. The skillbot is configured to receive user input in the form of utterances, parse or process the received input, and identify or select intents related to the received user input. As described above, the skillbot must be trained for this purpose. In certain embodiments, the skillbot is trained on intents configured for the skillbot and exemplary utterances associated with those intents (collectively, training data), so that the skillbot can transform user input utterances into one of the skillbot's configured intents. In certain embodiments, the skillbot uses a predictive model that has been trained with the training data to enable the skillbot to identify what the user is saying (or, in some cases, trying to say). DABP102 provides a variety of different training techniques that can be used by skillbot designers to train skillbots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In certain embodiments, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes called a trained skillbot) can then be used to process and respond to user utterances. In certain cases, a user utterance may be a question that requires only a single answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for the skillbot. This allows the skillbot to output a response to a user request without having to update the dialogue definition. Q&A intents are generated in the same way as regular intents. The dialogue flow for intents may differ from the dialogue flow for regular intents.

[0056] (5) The step of creating a dialogue flow for the skillbot. The dialogue flow specified for the skillbot describes how the skillbot will respond as various intents about the skillbot are resolved in response to received user input. The dialogue flow defines the actions or behaviors that the skillbot may take, such as how the skillbot responds to user utterances, how the skillbot prompts the user for input, and how the skillbot returns data. The dialogue flow is like a flowchart that the skillbot will follow. The skillbot designer specifies the dialogue flow using a language such as Markdown. In certain embodiments, a version of YAML called OBotML may be used to specify the dialogue flow for the skillbot. The dialogue flow definition for the skillbot serves as a model for the conversation itself, allowing the skillbot designer to choreograph the interaction between the skillbot and the user it corresponds to.

[0057] In a particular embodiment, the dialogue flow definition for the skill bot includes three sections. (a) Context section (b) Default transition section (c) State section.

[0058] Context section. In the context section, SkillBot designers can define the variables used in the conversation flow. Other variables that may be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that enable SkillBot to recognize and persist user preferences.

[0059] Default Transition Section. Transitions for SkillBot can be defined in the Dialogflow State section or the Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when there are no applicable transitions defined within a state or when the conditions necessary to trigger a state transition cannot be met. The Default Transition section can be used to define routing that allows SkillBot to smoothly handle unexpected user actions.

[0060] State section. A dialog flow and its associated behavior are defined as a set of transient states that govern the logic within the dialog flow. Each state node in the dialog flow definition designates a component that provides the functionality required for that point in the dialog. In this way, states are built around the components. A state contains component-specific characteristics and defines transitions to other states that are triggered after the component has been executed.

[0061] Special case scenarios can be handled using the state section. For example, you might want to give a user the option to temporarily leave the first skill they are involved with and do something with a second skill within the digital assistant. For instance, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some selection for a purchase), the user might want to jump to a banking skill (e.g., the user might want to make sure they have enough money for that purchase) and then return to the shopping skill to complete their order. To address this, an action in the first skill may be configured to initiate an interaction with a second, different skill within the same digital assistant, and then return to the original flow.

[0062] (6) Step of adding custom components to the skillbot. As described above, the states specified in the dialog flow for the skillbot designate the components that provide the necessary functionality corresponding to that state. The components enable the skillbot to perform the functionality. In certain embodiments, DABP102 provides a set of pre-configured components for performing a wide range of functions. The skillbot designer can select one or more of these pre-configured components and associate them with states in the dialog flow for the skillbot. The skillbot designer can also create custom or new components using the tools provided by DABP102 and associate the custom components with one or more states in the dialog flow for the skillbot.

[0063] (7) Steps to test and deploy the skillbot. DABP102 provides several features that enable skillbot designers to test the skillbot under development. The skillbot can then be deployed and included in the digital assistant.

[0064] The above description explains how to create a skillbot, but similar techniques may be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, embedded system intents may be configured for the digital assistant. These embedded system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without calling the skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit: when the user wants to end the current conversation or context in the digital assistant; (2) Help: when the user is seeking help or direction; (3) Unresolved Intent: for user input that does not appropriately match the Exit intent or Help intent. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information allows the masterbot to select a specific skillbot to process an utterance.

[0065] At the master bot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to process how to route the utterance and the associated conversation. The digital assistant makes this determination using a routing model that may be rule-based, AI-based, or a combination thereof. Using the routing model, the digital assistant determines whether the conversation corresponding to the user-input utterance should be routed to a specific skill for processing, processed by the digital assistant or master bot itself according to an embedded system intent, or processed as a different state in the current conversation flow.

[0066] In certain embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skillbot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skillbot corresponding to that invocation name. In such scenarios, the digital assistant may route the user input to the explicitly invoked skillbot for further processing. In the absence of a specific or explicit call, in certain embodiments, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intents and skillbots associated with the digital assistant. The scores calculated for skillbots or system intents represent the likelihood that the user input represents a task or system intent configured to be performed by a skillbot. System intents or skillbots whose associated calculated confidence scores exceed a threshold (e.g., Confidence Threshold routing parameter) are selected as candidates for further evaluation. The digital assistant then selects a specific system intent or skillbot from the identified candidates for further processing of the user input utterance. In certain embodiments, after one or more skillbots have been identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Generally, intents with a confidence score exceeding a threshold (e.g., 70%) are treated as candidate intents. If a specific skillbot is selected, the user utterance is routed to that skillbot for further processing. If a system intent is selected, the master bot itself will perform one or more actions according to the selected system intent.

[0067] Figure 2 is a simplified block diagram of a master bot (MB) system 200 according to a particular embodiment. The MB system 200 can be implemented using software alone, hardware alone, or a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill bot call unit 240, and a data store 250. The MB system 200 shown in Figure 2 is merely an example of the configuration of components in a master bot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in Figure 2, may combine two or more subsystems, or may have subsystems with different configurations or arrangements.

[0068] The preprocessing subsystem 210 receives the utterance "A" 202 from the user and processes the utterance through the language detection unit 212 and the language parser 214. As described above, the utterance can be provided in various ways, including speech or text. The utterance 202 may be a fragment, a complete sentence, multiple sentences, etc. The utterance 202 may contain punctuation. For example, if the utterance 202 is provided as speech, the preprocessing subsystem 210 may convert the speech to text using a speech-to-text converter (not shown) that inserts punctuation, such as commas, semicolons, periods, etc., into the resulting text.

[0069] The language detection unit 212 detects the language of the utterance 202 based on the text of the utterance 202. Since each language has its own grammar and semantics, the way in which the utterance 202 is processed depends on that language. Language differences are taken into consideration when analyzing the syntax and structure of the utterance.

[0070] The language parser 214 syntactically analyzes the utterance 202 and extracts part-of-speech (POS) tags for each linguistic unit (e.g., word) within the utterance 202. POS tags include, for example, nouns (noun: NN), pronouns (pronoun: PN), verbs (verb: VB), etc. The language parser 214 may also tokenize the language units of the utterance 202 (for example, to convert each word into a separate token) and turn the words into headwords. A headword is the primary form of a set of words represented in a dictionary (for example, "run" is a headword for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform are This includes chunking of compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". The language parser 214 also analyzes the relationships between words in the utterance 202. It can be identified. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., a particular noun) are direct objects, which parts of the utterance are prepositions, and so on. The results of the processing performed by the language parser 214 form extracted information 205, which is provided to the MIS 220 as input along with the utterance 202 itself.

[0071] As described above, utterance 202 may contain multiple sentences. For the purpose of detecting multiple intents and explicit calls, utterance 202 may be treated as a single unit, even if it contains multiple sentences. However, in certain embodiments, preprocessing may be performed, for example by the preprocessing subsystem 210, to identify a single sentence within multiple sentences for composite intent analysis and explicit call analysis. Generally, the results produced by MIS220 and EIS230 are substantially the same whether utterance 202 is processed at the level of individual sentences or as a single unit containing multiple sentences.

[0072] MIS220 determines whether utterance 202 represents multiple intents. While MIS220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS220 does not involve determining whether the intents in utterance 202 match any intent configured for the bot. Instead, the processing to determine whether the intents in utterance 202 match a bot intent may be performed by the intent classifier 242 of the MB system 200 (for example, as shown in the embodiment of Figure 3) or by the skill bot's intent classifier. The processing performed by MIS220 assumes the existence of a bot (e.g., a specific skill bot or the master bot itself) that can process utterance 202. Therefore, the processing performed by MIS220 does not require knowledge of what bots are in the chatbot system (e.g., the identity of a skill bot registered with the master bot) or what intents are configured for a particular bot.

[0073] To determine that utterance 202 contains multiple intents, MIS220 applies one or more rules from the set of rules 252 in datastore 250. The rules applied to utterance 202 are language-dependent of utterance 202 and may include sentence patterns indicating the presence of multiple intents. For example, a sentence pattern may include a conjunction connecting two parts of a sentence (e.g., a coordinating term), where both parts correspond to separate intents. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents directed to different bots, or different intents within the same bot). Instead, the utterance may have separate instances of the same intent, e.g., "order a pizza using payment account X, then order a pizza using payment account Y."

[0074] As part of determining that utterance 202 represents multiple intents, MIS220 also determines which parts of utterance 202 are associated with each intent. For each intent represented by an utterance containing multiple intents, MIS220 constructs a new utterance for a different processing, such as utterance "B" 206 and utterance "C" 208, in place of the original utterance, as shown in Figure 2. Thus, the original utterance 202 can be split into two or more separate utterances, each processed one at a time. Using the extracted information 205 and / or from an analysis of utterance 202 itself, MIS220 determines which of the two or more utterances should be processed first. For example, MIS220 determines that utterance 202 is related to a particular intent. It may be determined that the tent should be handled first, and that it contains a marker word indicating that the tent should be handled first. The newly formed utterance corresponding to this particular intent (e.g., one of utterances 206 or 208) will be sent first for further processing by the EIS230. After the conversation triggered by the first utterance has ended (or been temporarily interrupted), the next highest priority utterance (e.g., the other of utterances 206 or 208) may then be sent to the EIS230 for processing.

[0075] The EIS230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) contains a skillbot's invocation name. In certain embodiments, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes it from other skillbots in the chatbot system. A list of invocation names can be maintained in the data store 250 as part of the skillbot information 254. When an utterance contains a word that matches an invocation name, the utterance is considered an explicit invocation. If a bot is not explicitly invoked, the utterance received by the EIS230 is considered an implicit invocation utterance 234 and is fed into the master bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to process the utterance. In some examples, intent classifier 242 would determine that the master bot should process implicit invocation utterances. In other examples, intent classifier 242 would determine which skillbot to route the utterance to for processing.

[0076] The explicit invocation feature provided by EIS230 offers several advantages. It can reduce the amount of processing that the master bot must perform. For example, with explicit invocation, the master bot may not have to perform any intent classification analysis (e.g., using intent classifier 242), or it may have to perform reduced intent classification analysis to select a skill bot. Thus, explicit invocation analysis can enable the selection of a specific skill bot without relying on intent classification analysis.

[0077] Furthermore, there may be situations where functionality overlaps among multiple skill bots. This can occur, for example, when intents handled by two skill bots overlap or are very close to each other. In such situations, it may be difficult for the master bot to identify which of the multiple skill bots to select based solely on intent classification analysis. In such scenarios, explicit invocation resolves the ambiguity regarding which specific skill bot should be used.

[0078] In addition to determining that an utterance is an explicit invocation, the EIS230 plays a role in determining whether any part of the utterance should be used as input to the explicitly invoked skill bot. Specifically, the EIS230 can determine whether any part of the utterance is not associated with an invocation. The EIS230 can make this determination through analysis of the utterance and / or analysis of the extracted information 205. Instead of sending the entire utterance that the EIS230 receives, the EIS230 can send the portion of the utterance that is not associated with an invocation to the invoked skill bot. In some examples, the input to the invoked skill bot is formed simply by removing any portion of the utterance that is associated with an invocation. For example, "I want to order a pizza using Pizza Bot" becomes "I ordered a pizza It can be shortened to "i" because "using Pizza Bot" is a shortened version of Pizza Bot's call This is because it relates to output, but not to any processing performed by the pizza bot. In some cases, the EIS230 may reformat the parts to be sent to the invoked bot, for example, to form a complete sentence. Thus, the EIS230 determines not only whether there is an explicit invoke, but also what should be sent to the skill bot when there is an explicit invoke. In some cases, the text to be entered into the invoked bot. There may be cases where this is not possible. For example, if the utterance is "Pizza Bot", EIS230 will respond with "pizza". It may be determined that a bot has been called, but there is no text to process by the pizza bot. In such a scenario, the EIS230 may indicate to the skill bot call unit 240 that there is nothing to send.

[0079] The skillbot invocation unit 240 invokes skillbots in various ways. For example, the skillbot invocation unit 240 may invoke a bot in response to receiving instruction 235 that a particular skillbot has been selected as a result of an explicit invocation. Instruction 235 may be sent by the EIS 230 along with input about the explicitly invoked skillbot. In this scenario, the skillbot invocation unit 240 takes control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot will determine an appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, the initial state of the new conversation depending on the input sent from the EIS 230.

[0080] Another mode of invocation by the skillbot invocation unit 240 is through implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that a given utterance represents a task configured for a particular skillbot to perform. The intent classifier 242 is trained on various classes, with one class corresponding to each skillbot. For example, each time a new skillbot is registered with the master bot, the intent classifier 242 can be trained using a list of exemplary utterances associated with the new skillbot to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. Parameters generated as a result of this training (e.g., a set of values ​​for parameters of a machine learning model) can be stored as part of the skillbot information 254.

[0081] In certain embodiments, the intent classifier 242 is implemented using a machine learning model, as will be described in more detail here. Training the machine learning model may involve inputting at least a subset of utterances from exemplary utterances associated with various skill bots in order to generate inferences as the output of the machine learning model about which bot is the correct bot to process any given training utterance. For each training utterance, instructions for the correct bot to use for that training utterance may be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0082] In certain embodiments, the intent classifier 242 determines a confidence score for each skill bot registered with the master bot, indicating the likelihood that the skill bot can process a given utterance (e.g., an implicit calling utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., Help, Exit). If a particular confidence score satisfies one or more conditions, the skill bot calling unit 240 will call the bot associated with that particular confidence score. For example, a certain threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill bot. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must be higher than the next highest confidence score by a certain win margin. Imposing such a condition is useful when each of the confidence scores of multiple skill bots exceeds a certain threshold confidence score value. This will enable routing to SkillBot.

[0083] After identifying a bot based on the confidence score evaluation, the skill bot call unit 240 hands over processing to the identified bot. In the case of a system intent, the identified bot is the master bot. Otherwise, the identified bot is a skill bot. Furthermore, the skill bot call unit 240 will determine what to provide as input 247 for the identified bot. As described above, in the case of an explicit call, input 247 may be based on a portion of an utterance not associated with the call, or input 247 may be nothing (e.g., an empty string). In the case of an implicit call, input 247 may be the entire utterance.

[0084] The data store 250 comprises one or more computing devices that store data used by various subsystems of the master bot system 200. As described above, the data store 250 includes rules 252 and skill bot information 254. Rules 252 include rules for the MIS 220 to determine, for example, when an utterance represents multiple intents and how to divide an utterance that represents multiple intents. Rules 252 further include rules for the EIS 230 to determine which part of an utterance that explicitly invokes a skill bot should be sent to the skill bot. Skill bot information 254 includes the invocation names of skill bots in the chatbot system, for example, a list of the invocation names of all skill bots registered with a particular master bot. Skill bot information 254 may also include information used by the intent classifier 242 to determine a confidence score for each skill bot in the chatbot system, for example, parameters of a machine learning model.

[0085] Figure 3 is a simplified block diagram of a SkillBot System 300 according to a specific embodiment. The SkillBot System 300 is a computing system that can be implemented with software alone, hardware alone, or a combination of hardware and software. In specific embodiments, such as the embodiment shown in Figure 1, the SkillBot System 300 can be used to implement one or more SkillBots within a digital assistant.

[0086] The SkillBot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to the MIS 220 in Figure 2 and provides similar functionality, including being operable to determine (1) whether an utterance represents multiple intents, and if so, (2) how to divide the utterance into separate utterances for each of the multiple intents, using rules 352 in the data store 350. In a particular embodiment, the rules applied by the MIS 310 to detect multiple intents and divide the utterance are the same as the rules applied by the MIS 220. The MIS 310 receives an utterance 302 and extracted information 304. The extracted information 304 is similar to the extracted information 205 in Figure 1 and can be generated using a language parser 214 or a language parser local to the SkillBot system 300.

[0087] The intent classifier 320 can be trained in a manner similar to the intent classifier 242 described above in relation to the embodiment of Figure 2, as will be described in more detail here. For example, in a particular embodiment, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of exemplary utterances associated with that skill bot as training utterances. The ground truth for each training utterance would be the particular bot intent associated with that training utterance.

[0088] Utterance 302 may be received directly from the user or supplied via the masterbot. If utterance 302 is supplied through the master bot as a result of processing through the MIS220 and EIS230 in the embodiment shown in Figure 2, for example, the MIS310 can be bypassed to avoid repeating processing already performed by the MIS220. However, if utterance 302 is received directly from the user, for example during a conversation that takes place after routing to the skill bot, the MIS310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If utterance 302 represents multiple intents, the MIS310 applies one or more rules to split utterance 302 into separate utterances for each intent, for example, utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, the MIS310 forwards utterance 302 to the intent classifier 320 for intent classification without splitting it.

[0089] The intent classifier 320 is configured to match an received utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As described above, a skillbot can be configured with one or more intents, each intent including at least one exemplary utterance associated with that intent and used to train the classifier. In the embodiment of Figure 2, the intent classifier 242 of the masterbot system 200 is trained to determine confidence scores for individual skillbots and for system intents. Similarly, the intent classifier 320 can be trained to determine confidence scores for each intent associated with the skillbot system 300. The classification performed by the intent classifier 242 is at the bot level, while the classification performed by the intent classifier 320 is at the intent level and therefore of finer granularity. The intent classifier 320 can access intent information 354. The intent information 354 includes, for each intent associated with the skillbot system 300, a list of utterances that represent and exemplify the meaning of that intent and are typically associated with tasks that can be performed by that intent. The intent information 354 may further include parameters that are generated as a result of training on this list of utterances.

[0090] The conversation manager 330 receives, as output from the intent classifier 320, an instruction 322 for a specific intent identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some cases, the intent classifier 320 may be unable to determine any match. For example, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value if the utterance is directed to a system intent or an intent of a different skillbot. When this occurs, the skillbot system 300 may delegate the utterance to the master bot for processing, for example, to route it to a different skillbot. However, if the intent classifier 320 successfully identifies an intent within the skillbot, the conversation manager 330 will initiate a conversation with the user.

[0091] A conversation initiated by the conversation manager 330 is a conversation specific to an intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a certain dialogue flow for an identified intent. The state machine may include a default initiation state (for example, for when the intent is invoked without any additional input) and one or more additional states, in which case each state is associated with an action to be performed by the skillbot (e.g., execute a purchase transaction) and / or a dialogue to be presented to the user (e.g., a question, answer). Thus, when the conversation manager 330 receives an instruction 322 that identifies an intent, it can determine an action / dialogue 335 and decide on additional actions or dialogues in response to subsequent utterances received during the conversation. It is possible.

[0092] The data store 350 comprises one or more computing devices that store data used by various subsystems of the skillbot system 300. As shown in Figure 3, the data store 350 includes rule 352 and intent information 354. In certain embodiments, the data store 350 may be integrated with the data store of the master bot or digital assistant, for example, the data store 250 in Figure 2.

[0093] OOD Data Expansion Models used to classify utterances as intents have been found to have a tendency to underperform and produce poor results with respect to text that is OOD (Unintelligible). To overcome this problem, various embodiments have been directed toward techniques to augment training data with utterances containing OOD examples (negative learning utterances) in order to generalize the model and teach it that OOD utterances should be classified as either accurate intents or ground truths such as unresolved intents. By augmenting training data with OOD examples, the model is better able to focus on the most important parts of the examples in the correct context that link them to each class, including the OOD class. The model, trained on augmented text data, can be implemented in a chatbot system as described with respect to Figures 1, 2, and 3. Advantageously, these models and chatbots perform better with OOD utterances because the model can better focus on the important parts of the utterances in the correct context. Furthermore, since the augmentation is applied automatically in a synthesis-independent manner, customers or clients do not need to worry about adding OOD utterances in various contexts to the training data.

[0094] Figure 4 is a block diagram showing a view of a chatbot system 400 configured to train and utilize a classifier (e.g., the intent classifiers 242 or 320 described in relation to Figures 2 and 3) based on text data 405. As shown in Figure 4, the text classification performed by the chatbot system 400 in this example includes several stages: a predictive model training stage 410; a skillbot call stage 415 to determine the likelihood that an utterance represents a task configured for a particular skillbot to perform; and an intent prediction stage 420 to classify the utterance as one or more intents. The predictive model training stage 410 constructs and trains one or more predictive models 425a to 425n (where "n" represents any natural number) (which may be individually referred to as predictive model 425 and collectively as predictive model 425) to be used by the other stages. For example, the predictive model 425 may include a model for determining the likelihood that an utterance represents a task configured for a particular skillbot to perform, another model for predicting intent from an utterance for a first type of skillbot, and another model for predicting intent from an utterance for a second type of skillbot. Further other types of predictive models may be implemented in other examples pursuant to this disclosure.

[0095] Predictive model 425 is a convolutional neural network (CNN), such as an inception neural network, a residual neural network (Resnet), or a recurrent neural network. Work, for example, long short-term memory (LSTM) models or gated recurrent unit (GRU) models, other variations of deep neural networks (DNN) (e.g., single-in The machine learning (ML) models could be multilevel n-binary DNN classifiers or multiclass DNN classifiers for content classification. Predictive models 425 can also be Naive Bayes classifiers, linear classifiers, support vector machines, and random This could be a bagging model such as a forest model, a boosting model, a shallow neural network, or a combination of one or more such techniques, or any other suitable ML model trained for natural language processing, such as a CNN-HMM or MCNN (Multiscale Convolutional Neural Network). The chatbot system 400 may employ the same or different types of predictive models to determine the likelihood of a task configured for a particular skillbot to perform, to predict intents from utterances for a first type of skillbot, and to predict intents from utterances for a second type of skillbot. Further other types of predictive models may be implemented in other examples according to this disclosure.

[0096] To train various predictive models 425, the training phase 410 consists of three main components: dataset preparation 430, feature engineering 435, and model training 440. Dataset preparation 430 includes the process of loading data assets 445, dividing the data assets 445 into training and validation sets 445a-445n, and performing basic preprocessing so that the system can train and test the predictive models 425. Data assets 445 may include at least a subset of utterances from exemplary utterances associated with various skillbots. As mentioned above, utterances can be provided in various ways, including speech or text. Utterances can be fragments, complete sentences, multiple sentences, etc. For example, if utterances are provided as speech, data preparation 430 may convert speech to text using a speech-to-text converter (not shown) that inserts punctuation, e.g., commas, semicolons, periods, etc., into the resulting text. In some cases, exemplary utterances are provided by a client or customer. In other examples, exemplary utterances are automatically generated from a library of previous utterances (for example, identifying utterances from a library specific to the skills the chatbot learns). The data asset 445 for the predictive model 425 may include input text or speech (or input features of text or speech frames) and labels 450 corresponding to the input text or speech (or input features) as a matrix or table of values. For example, for each training utterance, instructions for the exact bot to use with that training utterance may be provided as ground truth information for the labels 450. The behavior of the predictive model 425 can then be adapted (for example, by backpropagation) to minimize the difference between the generated inferences and the ground truth information. Alternatively, the predictive model 425 may be trained for a particular skill bot using at least a subset of exemplary utterances associated with that particular skill bot as training utterances.The ground truth information for label 450 for each training utterance would be the specific bot intent associated with that training utterance.

[0097] In various embodiments, data preparation 430 includes OOD data augmentation (455) of data asset 445 to include OOD examples of utterances in various contexts in order to make the predictive model 425 more robust to OOD utterances. By augmenting data asset 445 with OOD examples in various contexts, the predictive model 425 better focuses on the most important parts of the examples and the contexts that link these examples to their respective classes, which include OOD classes. Augmentation 455 is implemented using OOD augmentation techniques to incorporate OOD utterances in various contexts along with the original utterances in data asset 445. The OOD augmentation techniques include the following four operations: The four operations generally include (i) generating a dataset containing multiple OOD examples, (ii) filtering out OOD examples with contexts that are too similar to the context of the original utterance, and (iii) feeding OOD examples to the model during training in a batch process to balance OOD examples and in-domain examples, as the number of OOD examples can be much larger than the number of in-domain utterances. Batch supply is, The process begins with batches containing easier OOD cases and continues with batches containing more difficult OOD cases.

[0098] The generation steps may include the following: (i) identifying clusters within a corpus (e.g., internal enterprise-oriented monolingual datasets); (ii) identifying clusters within a vocabulary database such as WordNet; (iii) generating OOD examples across multiple domains (i.e., clusters of data) using a text generation model; (iv) generating OOD examples across multiple domains (i.e., clusters of data) using an adversarial attack model; or any combination thereof; and (iv) selecting OOD examples from each of the identified clusters or domains that should be included in the OOD example dataset. A cluster or domain may be a set of data that is (in some sense) more similar to one another than clusters or domains in other groups, such as a cluster of weather-related OOD examples, a cluster of sports-related OOD examples, a cluster of political-related OOD examples, etc. The identification of clusters or domains may be performed using any known clustering analysis technique (e.g., k-means clustering) as in the examples above.

[0099] 1. Find examples of OOD (Good Occasion) from the corpus. OOD examples are found in corpora (e.g., NLTK's Brown Corpus and Reuters Corpus) Candidates can be identified by searching for sentences. Examples that are too short (less than 5 words) may be filtered out as it may not be beneficial to add examples that do not contain much content. Furthermore, examples that are too easy (measures of difficulty such as ease are explained in detail here) may also be excluded as it may not be beneficial to add examples that are too easy (the model can not learn anything from examples that are too easy).

[0100] 2. Find OOD examples from the vocabulary database. To find examples of Good Occasions (OODs) that cover all possible clusters or domains of data, vocabulary databases such as WordNet can be used.

[0101] 3. Generate OOD examples using a model, such as GPT-2. OOD examples can be generated using text generation models such as Generative Pre-trained Transformer 2 (GPT-2). This can provide more control over the type of examples generated for specific clusters or domains of data.

[0102] 4. Generate OOD examples using an adversarial attack model. A variation of the adversarial attack model can be used to generate OOD statements that are known to be classified as being within a domain by a predictive model. This is a one-time method because it does not require a cluster or domain identification step.

[0103] The filtering step may include filtering out OOD examples that are too similar (virtually similar) to examples in the training data using a distance measure (e.g., Minkowski, Manhattan, Euclidean, Hamming, cosine, etc.) to avoid class mismatches. In some examples, distance filtering is based on the Euclidean distance between the OOD example candidate vector and the examples in the training data. Examples of filtering may include (i) using a global fixed distance threshold for all domains (e.g., using default distance = 1.3), (ii) using percentiles of the distance between training examples and their nearest neighbors, (iii) using the maximum distance from a training example to its nearest neighbor with a fixed positive margin in [-0.1, 0.2], (iv) using the maximum distance from a training example to its nearest neighbor multiplied by some coefficient [0.9~1.2], or (v) any combination thereof.

[0104] In some embodiments, a global fixed distance threshold is used for all clusters or domains to filter out OOD examples. The global fixed distance threshold may be determined using a model for multilingual word embeddings, such as the Multilingual Universal Sentence Encoder (MUSE) single embedding. MUSE is a multi-task and multi-language sentence coding model that can create a single embedding space common to all trained languages. The MUSE filtering process includes the following: This includes (i) using v_i=muse(x_i) as the representation of utterance x_i (i=1→n) in the training data, (ii) using u=muse(ood utterance) as the representation of ood utterances such as "I want to order Coca-Cola," and (iii) using d_i=Euclidean distance(v_i,u). Furthermore, (iv) if min(d_i) < a predetermined threshold (e.g., 1.3), then =>>, for example, the utterance "I want to order Coca-Cola" becomes too close to an intradomain intent (e.g., the order_pizza intent). Therefore, this utterance "I want to order Coca-Cola" will be filtered out of the OOD example dataset.

[0105] The supply step may include a step to ensure that the correct number of OOD examples are present in each training batch using a batch balancing method (too few examples means that the OOD examples will have no impact, and too many OOD examples may wipe out the original examples within the domain). For example, in the case of an unbalanced multi-class dataset for classification containing up to 30% positive examples (examples within the original domain) and up to 70% negative examples (OOD examples), a predetermined number of batch sizes may be selected during training to ensure that each batch used for backpropagation is balanced, for example, so that up to 10 data points are sampled from the positive training data and up to 10 data points are sampled from the negative training data. This avoids the model being biased towards negative data such as OOD examples.

[0106] Furthermore, the OOD examples in the OOD example dataset may be sorted from easiest to hardest and incorporated into each training batch based on the sort. The difficulty (hardness or ease) of an OOD example can be determined using a MUSE representation, for example, by sorting the d_i values ​​from the filtering step. Easy OOD examples will have high d_i values, while difficult OOD examples will have lower d_i values, closer to a predetermined threshold, e.g., 1.3. The OOD examples can be sorted based on difficulty determination / sorting and then introduced into batches of training data (for example, easy OOD examples may be introduced into the first five batches, intermediate OOD examples into the next five batches, and difficult OOD examples into the next five batches). These batches can be fed into the predictive model 425 using protocols ranging from the easiest to the hardest, such as curriculum training techniques. Given the possibility of many OOD examples, the curriculum training technique can be established so that the predictive model 425 is exposed to each OOD example only once. In other words, once these batches are created, they are fed into the predictive model 425 with the constraint that each OOD example (regardless of difficulty) is included in the training data batch only once. The characteristics of the process described above for OOD augmentation can be treated as hyperparameters of the predictive model 425 (e.g., epochs = 28, fixed logit = FALSE, OOD utterances per intent = 14, OOD utterances per batch = 5).

[0107] The introduction of OOD extension technology will allow predictive model 425 to function better for OOD utterances and normal intra-domain utterances because predictive model 425 will be able to focus better on the important parts of the utterance. Because it is applied automatically in a synthesis-independent manner, customers or clients do not need to worry about manually adding OOD utterances to the training data.

[0108] In some cases, additional augmentations may be applied to data asset 445 (along with OOD augmentations). For example, Easy Data Augmentation (EDA) techniques may be used to improve the performance of a text classification task. EDA includes four operations: synonym substitution, random insertion, random exchange, and random deletion, which help prevent overfitting and train a more robust model. In contrast to OOD augmentations, EDA operations generally (i) take words from the original text and (ii) incorporate those words into each data asset 445 relative to the original text. For example, a synonym substitution operation involves randomly selecting n non-stop words from the original sentence (e.g., utterance) and replacing each of these words with one of its randomly selected synonyms. A random insertion operation involves finding n random synonyms of random non-stop words in the original sentence and inserting those synonyms at random positions in the sentence. A random exchange operation involves randomly selecting two words in a sentence n times and swapping their positions. A random deletion operation involves randomly deleting each word in a sentence with probability p.

[0109] In various embodiments, feature engineering 435 involves transforming the data asset 445 into feature vectors (with OOD extensions) and / or creating new features (with OOD extensions) using the data asset 445. Feature vectors may include count vectors as features, TF-IDF vectors as features such as word-level, n-gram-level, or character-level features, word embeddings as features, text / NLP as features, topic models as features, or combinations thereof. A count vector is a matrix representation of the data asset 445 where each row represents an utterance, each column represents a word from the utterance, and each cell represents the frequency count of a particular word within the utterance. TF-IDF scores represent the relative importance of words in an utterance. Word embeddings are a form of representing words and utterances using dense vector representations. The position of a word in vector space is learned from text and is based on the words that surround that word when it is used. Text / NLP-based features may include the number of words in an utterance, the number of characters in an utterance, the average word density, the number of punctuation marks, the number of capital letters, the number of headwords, the frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof. Topic modeling is a technique for identifying groups of words (called topics) that contain the best information within a collection of utterances.

[0110] In various embodiments, model training 440 includes training a classifier using feature vectors and / or new features created in feature engineering 435. In some cases, the training process includes iterative operations to find a set of parameters for the predictive model 425 that minimizes a loss function or error function for the predictive model 425. Each iteration may involve finding a set of parameters for the predictive model 425 such that the value of the loss function or error function using a set of parameters for the predictive model 425 is less than the value of the loss function or error function using a different set of parameters in the previous iteration. The loss function or error function can be constructed to measure the difference between the output predicted using the predictive model 425 and the labels 450 contained in the data asset 445. Once the set of parameters is identified, the predictive model 425 is trained and can be used for prediction as designed.

[0111] In addition to the data assets 445, labels 450, feature vectors and / or new features, other techniques and information can also be employed to improve the training process of the predictive model 425. For example, feature vectors and / or new features can be used in the classifier or These may be combined with each other to help improve the accuracy of the model. Additionally or alternatively, hyperparameters may be tuned or optimized, and for example, several parameters such as tree length, leaves, and network parameters may be fine-tuned to obtain a best-fit model. However, the training mechanisms described herein primarily focus on training predictive models 425. These training mechanisms can also be used to fine-tune existing predictive models 425 trained from other data assets. For example, in some cases, predictive models 425 may have been pre-trained using utterances specific to a different skillbot. In such cases, predictive models 425 can be retrained using data asset 445 (with OOD extensions) as described herein.

[0112] The predictive model training stage 410 outputs a trained predictive model 425, which includes a task predictive model 460 and an intent predictive model 465. The task predictive model 460 may be used in the skillbot invocation stage 415 to determine the likelihood that an utterance represents a task configured for a particular skillbot to perform (470), and the intent predictive model 465 may be used in the intent predictive stage 420 to classify an utterance as one or more intents (475). In some examples, the skillbot invocation stage 415 and the intent predictive stage 420 may proceed independently, using separate models in some examples. For example, the trained intent predictive model 465 may be used in the intent predictive stage 420 to predict intents for a skillbot without first identifying the skillbot in the skillbot invocation stage 415. Similarly, the task prediction model 460 may be used in the skillbot invocation stage 415 to predict the task or skillbot that should be used for the utterance, without identifying the intent of the utterance in the intent prediction stage 420.

[0113] Alternatively, the skillbot invocation stage 415 and the intent prediction stage 420 may be performed sequentially, with one stage using the output of the other stage as input, or one stage invoking a particular skillbot in a particular manner based on the output of the other stage. For example, with respect to a given text data 405, the skillbot invocation unit can invoke a skillbot through an implicit invocation using the skillbot invocation stage 415 and the task prediction model 460. The task prediction model 460 is trained using machine learning and / or rule-based training techniques to determine the likelihood that a given utterance represents a task configured for a particular skillbot 470 to perform. Then, with respect to the identified or invoked skillbot and the given text data 405, the intent prediction stage 420 and the intent prediction model 465 can be used to match the received utterance (e.g., an utterance in a given data asset 445) to an intent 475 associated with the skillbot. As described here, a skillbot can consist of one or more intents, each intent including at least one exemplary utterance associated with that intent and used to train a classifier. In some embodiments, the skillbot call stage 415 and task prediction model 460 used in a masterbot system are trained to determine confidence scores for individual skillbots and for system intents. Similarly, the intent prediction stage 420 and intent prediction model 465 may be trained to determine a confidence score for each intent associated with the skillbot system. The classification performed by the skillbot call stage 415 and task prediction model 460 is at the bot level, while the classification performed by the intent prediction stage 420 and intent prediction model 465 is at the intent level and therefore has a finer granularity.

[0114] OOD Data Augmentation and Speech Classification Technologies Figure 5 is a flowchart of process 500 for augmenting a training dataset with an OOD example, according to several embodiments. The process shown in Figure 5 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 5 and described below are intended to be illustrative and non-limiting. Figure 5 shows various processing steps performed in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, those steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, such as the embodiments shown in Figures 1 to 4, the process shown in Figure 5 may be performed by a preprocessing subsystem (e.g., preprocessing subsystem 210 or predictive model training stage 410) to generate an OOD augmented dataset for training with one or more predictive models (e.g., intent classifiers 242 or 320 or predictive model 425).

[0115] In step 505, a training set of utterances is received by a data processing system (for example, the chatbot system 400 described in relation to Figure 4). In some examples, the training set of utterances is used to train a machine learning model to identify one or more intents for one or more utterances.

[0116] In 510, the data processing system expands the speech training set with OOD examples to obtain an expanded speech training set. In various embodiments, the expansion step includes (i) generating a dataset of OOD examples; (ii) filtering out OOD examples from the speech training set that have contexts substantially similar to the contexts of the speeches; (iii) determining a difficulty value for each OOD example remaining in the filtered OOD example dataset; and (iv) generating a batch of expanded speeches, which includes speeches from the speech training set and speeches from the filtered OOD example dataset, based on the difficulty value for each OOD. When used herein, if an action is “based on” something, it means that the action is at least partially based on at least a part of something. When used herein, the terms “substantially,” “almost,” and “about” are defined as being largely (and fully) identified, not necessarily completely, as understood by those skilled in the art. In any of the disclosed embodiments, the terms “substantially,” “almost,” or “about” may be replaced with “within [percentage]” of the specified, where the percentages include 0.1 percent, 1 percent, 5 percent, and 10 percent. The substantial similarity between the context of an OOD example and the context of an utterance in a training set of utterances is determined on a distance scale to avoid class mismatches.

[0117] Optionally, the substantial similarity between the context of an OOD example and the context of an utterance in the utterance training set is determined using a Multilingual Universal Sentence Encoder (MUSE) single embedding, and the context of the OOD example and the context of an utterance in the utterance training set are determined to be substantially similar if min(d_i) < a predetermined threshold (d_i = Euclidean distance(v_i,u)).

[0118] In some cases, the difficulty value is such that easy OOD examples have a high d_i value, while difficult OOD examples have a lower d_i value that is closer to a predetermined threshold. This leads to the conclusion that batches of augmented utterances can be generated based on the following constraints: (i) a predetermined batch size, (ii) inclusion of a predetermined number of in-domain examples of utterances from a training set of utterances in each batch, (iii) inclusion of a predetermined number of OOD examples of utterances from a filtered OOD dataset in each batch, (iv) selection of a predetermined number of in-domain examples and a predetermined number of OOD examples of utterances to maintain a predetermined batch size, (v) OOD examples of utterances included in each batch having substantially similar difficulty values, and optionally, (vi) each ODD example from a filtered OOD dataset being included in only a single batch of augmented utterances.

[0119] In 515, a machine learning model is trained using a batch of augmented utterances to determine the likelihood that an utterance or message represents a task configured for a skillbot to perform, or to match an utterance or message to an intent associated with a skillbot. Training may be performed according to a curriculum training protocol that includes feeding the machine learning model batches of augmented utterances based on the difficulty values ​​of the OOD examples in each batch of augmented utterances. Then, in 520, the trained machine learning model may be deployed in a chatbot system (e.g., as part of a skillbot) to determine the likelihood that an utterance or message represents a task configured for a skillbot to perform, or to match an utterance or message to an intent associated with a skillbot. For example, an utterance may be received and analyzed to determine whether it contains a skillbot invocation name. If no invocation name is found, the utterance is considered implicitly invocation, and the process proceeds to an intent classifier, such as the trained machine learning model. If a call name is determined to exist, the utterance is considered an explicit call, and the process proceeds to determine which part of the utterance is associated with the call name.

[0120] When a trained model is invoked, the entire received utterance is provided as input to the intent classifier. The intent classifier receiving the utterance may be the master bot's intent classifier (e.g., intent classifier 242 in Figure 2). The intent classifier may be a machine learning-based or rule-based classifier trained with OOD-enhanced data to determine whether the intent of the utterance matches a system intent (e.g., Exit, Help) or a specific skill bot. As described here, the intent analysis performed by the master bot may be limited to matching to a specific skill bot without determining which intent within that skill bot is the best match for the utterance. Thus, the intent classifier receiving the utterance may identify a specific skill bot to be invoked. Alternatively, if the utterance represents a specific system intent (e.g., the utterance contains the words "Exit" or "Help"), the intent classifier receiving the utterance may identify that specific system intent to trigger a conversation between the master bot and the user based on a dialogue flow configured for that specific system intent.

[0121] If a call name exists, one or more explicit call rules are applied to determine which part of the utterance is associated with the call name. This determination may be based on an analysis of the utterance's sentence structure, using POS tags, dependency information, and / or other extracted information received with the utterance. For example, a part associated with a call name could be a noun phrase containing the call name, or a prepositional object corresponding to the call name. Any part associated with a call name, as determined based on the processing, is removed. Other parts of the utterance that are not necessary to convey the meaning of the utterance (e.g., prepositional words) may also be removed. Removal of specific parts of the utterance generates input about the skill bot associated with the call name. If any portion of the received utterance remains after removal, that remaining portion will form a new utterance for input to the skillbot, for example, as a text string. Otherwise, if the received utterance is completely removed, the input may be an empty string.

[0122] Subsequently, the skillbot associated with the invocation name is invoked, and the generated input is provided to that skillbot. Upon receiving the generated input, the invoked skillbot will process it by performing intent analysis, for example, using the skillbot's intent classifier trained on OOD-enhanced data, to identify bot intents that match the user intents represented in the input. As a result of identifying matching bot intents, the skillbot may perform a specific action or initiate a conversation with the user according to the dialogue flow associated with the matching bot intent. For example, if the input is an empty string, the conversation may begin in a default state defined with respect to the dialogue flow, which could be a welcome message. Alternatively, if the input is not an empty string, for example, if the input contains a value about an entity or some other information that the skillbot received as part of the input and no longer needs to ask the user for, the conversation may begin in some intermediate state. As another example, the skillbot may determine that it cannot process the input (for example, because the confidence scores of all bot intents configured with respect to the skillbot fall below a certain threshold). In this situation, the skill bot may either pass the input back to the master bot for processing (e.g., intent analysis using the master bot's intent classifier), or the skill bot may prompt the user for clarification.

[0123] In various embodiments, the deployment and use of an intent classifier within a chatbot system includes the chatbot system receiving utterances generated by a user interacting with the chatbot system, classifying the utterances into intent categories corresponding to intents using an intent classifier deployed within the chatbot system, and outputting intents based on the classification using the intent classifier. The intent classifier includes multiple model parameters identified using training data, which includes a training set of augmented utterances for training the intent classifier to identify one or more intents for one or more utterances, and a training set of augmented utterances for training the intent classifier to identify one or more intents for one or more utterances, the training set of augmented utterances being artificially generated from the training set of utterances to include augmented utterances, in which case the augmented utterances are OOD utterances having a substantially different context from the context of the utterances in the training set of utterances, the substantial difference between the context of the OOD example and the context of the utterances in the training set of utterances being determined using a Multilingual Universal Sentence Encoder (MUSE) single embedding, where min(d_i) => a predetermined threshold (d_i = Euclidean distance(v_i,u)) is determined to be substantially different from the context of the OOD example and the context of the utterances in the training set of utterances. Multiple model parameters are identified using the training data based on minimizing the loss function. Deploying and using an intent classifier within a chatbot system further involves using the intent classifier to output intents based on the classification.

[0124] The substantial difference between the context of an OOD example and the context of an utterance in a training set can be determined based on a distance scale to avoid class mismatches.

[0125] Exemplary System Figure 6 shows a simplified diagram of the distributed system 600. In the illustrated example, the distributed system The M600 includes one or more client computing devices 602, 604, 606, and 608 connected to the server 612 via one or more communication networks 610. The client computing devices 602, 604, 606, and 608 may be configured to run one or more applications.

[0126] In various examples, server 612 may be adapted to run one or more services or software applications that enable one or more embodiments of the embodiments described herein. In certain examples, server 612 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be web-based services or cloud services, such as under the Software as a Service (SaaS) model, to users of client computing devices 602, 604, 606 and / or 608. These may be provided as follows. Users operating client computing devices 602, 604, 606, and / or 608 can utilize the services provided by these components by interacting with server 612 using one or more client applications.

[0127] In the configuration shown in Figure 6, server 612 may include one or more components 618, 620, and 622 that implement the functions performed by server 612. These components may include one or more processors, hardware components, or software components that can be executed by a combination thereof. It should be recognized that a wide variety of system configurations are possible, which may differ from the distributed system 600. Therefore, the example shown in Figure 6 is an example of a distributed system for implementing an exemplary system, and is not intended to be limiting.

[0128] A user may use client computing devices 602, 604, 606, and / or 608 to run one or more applications, models, or chatbots that can generate one or more events or models that can be realized or processed in accordance with the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. The client device may also output information to the user through this interface. Although Figure 6 shows only four client computing devices, any number of client computing devices may be supported.

[0129] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, game systems, thin clients, various messaging devices, sensors or other sensing devices. These computing devices may use various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®). (Mark), Windows Phone (registered trademark), Android (registered trademark), BlackBerry (registered trademark), Palm Various types and versions of software applications and operating systems, including OS(registered trademark), such as Microsoft Windows(registered trademark), Apple Macintosh(registered trademark), UNIX(registered trademark), or UNIX-like operating systems, Linux(registered trademark). This may include registered trademarks or Linux-based operating systems, such as Google Chrome® OS. Portable handheld devices include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), and personal digital assistants (PDAs). This may include, etc. Wearable devices may include Google Glass® head-mounted displays and other devices. Game systems may include various handheld game devices, internet-connected game devices (e.g., Kinect® gestures) This may include Microsoft Xbox® game consoles with or without input devices, Sony PlayStation® systems, and various game systems provided by Nintendo®. Client devices may be capable of running a wide variety of applications, such as various internet-related applications and communication applications (e.g., email applications, short message service (SMS) applications), and may use various communication protocols.

[0130] Network 610 may be any type of network known to those skilled in the art, capable of supporting data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. This is merely an example. Network 610 includes local area networks (LANs), Ethernet®-based networks, Token Ring, wide area networks (WANs), the Internet, virtual networks, virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (for example, networks operating under any of the IEEE 802.11 protocol suites, Bluetooth®, and / or any other wireless networks). Linear protocols, and / or any combination of these and / or other networks.

[0131] Server 612 may consist of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, rack-mount servers, etc.), server farms, server clusters, or other appropriate configurations and / or combinations. Server 612 may include one or more virtual machines running a virtual operating system, or other computing architectures with virtualization, such as one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the server. In various examples, Server 612 may be adapted to run one or more services or software applications that provide the functions described above.

[0132] The computing system within server 612 may run one or more operating systems, including any of the above operating systems, and any commercially available server operating system. Server 612 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA® servers, database servers, etc. Exemplary database servers include those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc. Not limited.

[0133] In some implementations, server 612 may include one or more applications for analyzing and organizing data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. For example, the data feeds and / or event updates may include real-time events related to sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring. Twitter may receive data from one or more third-party sources and continuous data streams. The server 612 may also include, but is not limited to, a (registered trademark) feed, Facebook (registered trademark) updates, or real-time updates. The server 612 may also include one or more applications for displaying the data feed and / or real-time events via one or more display devices of client computing devices 602, 604, 606, and 608.

[0134] The distributed system 600 may also include one or more data repositories 614, 616. In a particular example, these data repositories may be used to store data and other information. For example, one or more of the data repositories 614, 616 may be used to store information such as information related to chatbot performance or generated models for use by a chatbot used by the server 612 when performing various functions according to various embodiments. The data repositories 614, 616 may reside in various locations. For example, the data repository used by the server 612 may be located locally with the server 612, or it may be located remotely from the server 612 and may communicate with the server 612 via a network-based connection or a dedicated connection. The data repositories 614, 616 may be of different types. In a particular example, the data repository used by the server 612 may be a database, such as a relational database, such as a database provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to commands in SQL format.

[0135] In a particular example, one or more of the data repositories 614, 616 may be used by the application to store application data. The data repositories used by the application may be of various types, such as a key-value store repository, an object store repository, or a general-purpose storage repository supported by the file system.

[0136] In a particular example, the functions described herein may be provided as services through a cloud environment. Figure 7 is a simplified block diagram of a cloud-based system environment in a particular example where various services may be provided as cloud services. In the example shown in Figure 7, the cloud infrastructure system 702 may provide one or more cloud services that a user may request using one or more client computing devices 704, 706, and 708. The cloud infrastructure system 702 may include one or more computers and / or servers, which may include those described above with respect to server 612. The computers within the cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0137] Network 710 can facilitate the communication and exchange of data between clients 704, 706, and 708 and the cloud infrastructure system 702. Network 710 may include one or more networks. The networks may be of the same type or different types. Network 710 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0138] The example shown in Figure 7 is merely one example of a cloud infrastructure system and is not intended to be limiting. In some other examples, the cloud infrastructure system 702 may have more or fewer components than shown in Figure 7, or it may be a combination of two or more components, or different components. It should be understood that the components may have a configuration or arrangement such as the above. For example, Figure 7 shows three client computing devices, but in alternative examples, any number of client computing devices may be supported.

[0139] The term "cloud service" is generally used to refer to services made available to users on demand via communication networks such as the Internet, through a service provider's system (e.g., cloud infrastructure system 702). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can order and use applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, some cloud services, such as middleware services, database services, and Java® cloud services, are offered by Oracle Corporation® in Redwood Shores, California.

[0140] In a particular example, the cloud infrastructure system 702 may provide one or more cloud services using various models, including a hybrid service model, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, and an Infrastructure as a Service (IaaS) model. The cloud infrastructure system 702 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0141] The SaaS model enables the delivery of applications or software as a service to customers over a communication network such as the internet, without requiring customers to purchase the underlying hardware or software for the application. For example, the SaaS model can be used to allow customers to access on-demand applications hosted on a cloud infrastructure system 702. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0142] The IaaS model is generally used to provide flexible computing and storage capabilities by offering infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation®.

[0143] The PaaS model is generally used to provide a platform and environmental resources as a service, enabling customers to develop, run, and manage applications and services without having to procure, build, or maintain those resources themselves. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0144] Cloud services are generally delivered in an on-demand, self-service, subscription-based, flexibly scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by the cloud infrastructure system 702 through a subscription order. The cloud infrastructure system 702 then provides the services requested in the customer's subscription order by performing processing. For example, a user may use utterances to request the cloud infrastructure system to take a specific action (e.g., an intent) as described above, and / or to provide services for a chatbot system as described herein. The cloud infrastructure system 702 may be configured to provide one cloud service or more cloud services.

[0145] The cloud infrastructure system 702 can provide cloud services through various deployment models. In a public cloud model, the cloud infrastructure system 702 may be owned by a third-party cloud service provider, and the cloud services are provided to a general public customer. This customer may be an individual or a company. In another example, under a private cloud model, the cloud infrastructure system 702 may function within an organization (e.g., a corporate organization), and the services are provided to customers within this organization. For example, this customer may be various departments within the company, such as the human resources department or the payroll department, or an individual within the company. In yet another example, under a community cloud model, the cloud infrastructure system 702 and the services provided may be shared among various organizations within the relevant community. Other models, such as hybrid models of the above models, may also be used.

[0146] The client computing devices 704, 706, and 708 may be of different types (e.g., client computing devices 602, 604, 606, and 608 shown in Figure 6) and may be capable of operating one or more client applications. Users may interact with the cloud infrastructure system 702, such as requesting services provided by the cloud infrastructure system 702, by using the client devices. For example, a user may use a client device to request information or actions from a chatbot, as described in this disclosure.

[0147] In some examples, the processing performed by the cloud infrastructure system 702 to provide services may include model training and deployment. This analysis may include training and deploying one or more models by using, analyzing, and processing a dataset. This analysis may be performed by one or more processors, which may, in some cases, process the data in parallel and run simulations using the data. For example, big data analysis may be performed by the cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used in this analysis may be structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)). It may include the Inari Large Object.

[0148] As shown in the example in Figure 7, the cloud infrastructure system 702 may include infrastructure resources 730 used to facilitate the provisioning of various cloud services provided by the cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In a particular example, a storage virtual machine available to handle storage requested by an application may be part of the cloud infrastructure system 702. In other examples, the storage virtual machine may be part of a different system.

[0149] In certain examples, resources may be grouped into resource sets or resource modules (also referred to as "pods") to facilitate the efficient provisioning of these resources to support various cloud services provided by the cloud infrastructure system 702 to different customers. Each resource module or pod may contain a pre-integrated and optimized combination of one or more types of resources. In certain examples, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for a database service, and a second set of pods, which may contain different resource combinations than the pods in the first set, may be provisioned for a Java service, etc. For some services, resources allocated to provision these services may be shared among the services.

[0150] The cloud infrastructure system 702 itself may internally use services 732 that are shared by various components of the cloud infrastructure system 702 and facilitate the provisioning of services by the cloud infrastructure system 702. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services that enable cloud support, email services, notification services, and file transfer services.

[0151] The cloud infrastructure system 702 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in Figure 7, the subsystem may include a user interface subsystem 712 that enables users or customers of the cloud infrastructure system 702 to interact with the cloud infrastructure system 702. The user interface subsystem 712 may include various different interfaces, such as a web interface 714, an online store interface 716 where cloud services provided by the cloud infrastructure system 702 are advertised and available for purchase by consumers, and other interfaces 718. For example, a customer may use a client device to request (service request 734) one or more services provided by the cloud infrastructure system 702 using one or more of interfaces 714, 716, and 718. For example, a customer may access an online store, browse the cloud services provided by the cloud infrastructure system 702, and place a subscription order for one or more services provided by the cloud infrastructure system 702 that the customer wishes to subscribe to. This service request may include information identifying the customer and the one or more services that the customer wishes to subscribe to. For example, a customer can place an order for services provided by the cloud infrastructure system 702. As part of the order, the customer will provide information identifying the chatbot system on which the service will be provided, and optionally, chatbot information. You may provide one or more credentials regarding the chatbot system.

[0152] In a specific example, such as the one shown in Figure 7, the cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this process, the OMS 720 may be configured to prepare the order for provisioning by creating a customer account if one does not already exist, receiving billing and / or account information from the customer to be used to charge the customer in order to provide the requested services to the customer, verifying the customer information, reserving the order for the customer after verification, and coordinating various workflows.

[0153] If properly validated, OMS720 may invoke an order provisioning subsystem (OPS)724 configured to provision resources for this order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring those resources to facilitate the services requested by the customer order. The method of provisioning resources for an order and the types of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a certain workflow, OPS724 may be configured to determine the specific cloud service being requested and identify the number of pods that would have been pre-configured for that particular cloud service. The number of pods allocated for an order may depend on the size / volume / level / scope of the requested service. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to suit the specific customer making the request in order to provide the requested service.

[0154] In certain cases, the setup phase process may be performed by the cloud infrastructure system 702 as part of the provisioning process, as described above. The cloud infrastructure system 702 may generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by the cloud infrastructure system 702 itself, or from storage virtual machines provided by other systems other than the cloud infrastructure system 702.

[0155] The cloud infrastructure system 702 may send a response or notification 744 to the requesting customer to indicate when the requested service will be available. In some examples, the customer may be sent information (e.g., a link) that enables the customer to begin using and utilizing the benefits of the requested service. In certain examples, the response to the customer requesting the service may include a chatbot system ID generated by the cloud infrastructure system 702 and information identifying a chatbot system selected by the cloud infrastructure system 702 for the chatbot system corresponding to the chatbot system ID.

[0156] The cloud infrastructure system 702 may provide services to multiple customers. For each customer, the cloud infrastructure system 702 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and is responsible for providing the requested services to the customer. The cloud infrastructure system 702 may also collect usage statistics about the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information may be used to charge the customer. For example, It can be done monthly.

[0157] The cloud infrastructure system 702 may provide services to multiple customers in parallel. The cloud infrastructure system 702 may store information about these customers, including, in some cases, copyright information. In a particular example, the cloud infrastructure system 702 includes an Identity Management Subsystem (IMS) 728 configured to manage customer information and separate the managed information so that information about one customer cannot be accessed by another customer. The IMS 728 may be configured to provide various security-related services, such as identity services including information access management, authentication and authorization services, and services for managing customer identities and roles and related capabilities.

[0158] Figure 8 shows an example of computer system 800. In some examples, computer system 800 can be used to implement any digital assistant or chatbot system in a distributed environment, as well as the various servers and computer systems described above. As shown in Figure 8, computer system 800 includes various subsystems, including a processing subsystem 804 that communicates with several other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. The storage subsystem 818 may include a non-temporary computer-readable storage medium, including a storage medium 822 and system memory 810.

[0159] The bus subsystem 802 provides a mechanism for various components and subsystems of the computer system 800 to communicate with each other as intended. While the bus subsystem 802 is schematically shown as a single bus, alternative examples of the bus subsystem may utilize multiple buses. The bus subsystem 802 may be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, local bus, etc., using one of various bus architectures. For example, such architectures include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and those manufactured according to the IEEE P1386.1 standard. Peripheral component interconnects can be implemented as mezzanine buses. This may include components such as the PCI bus.

[0160] The processing subsystem 804 controls the operation of the computer system 800 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 804 may include one or more dedicated coprocessors, such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of the processing subsystem 804 may be implemented using customized circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0161] In some examples, the processing unit within the processing subsystem 804 is located in the system memory. Instructions stored in 810 or the computer-readable storage medium 822 may be executed. In various examples, the processing unit may execute various program or code instructions and maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in system memory 810 and / or the computer-readable storage medium 822, which may contain one or more storage devices. Through appropriate programming, the processing subsystem 804 may provide the various functions described above. In an example where the computer system 800 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0162] In a particular example, a processing acceleration unit 806 may be optionally provided to perform customized processing to accelerate the overall processing performed by the computer system 800, or to offload a portion of the processing performed by the processing subsystem 804.

[0163] The I / O subsystem 808 may include devices and mechanisms for inputting information into and / or outputting information from or through the computer system 800. Generally, the use of the term “input device” is intended to include all conceivable types of devices and mechanisms for inputting information into the computer system 800. User interface input devices may include, for example, pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors, Microsoft Xbox® 360 game controllers, and devices that provide interfaces for receiving input using gestures and voice commands, enabling users to control and interact with input devices. The user interface input device may also include an eye gesture recognition device, such as the Google Glass® blink detector, which detects eye movements from the user (e.g., blinking while taking a picture and / or making a menu selection) and translates the eye gestures into input to an input device (e.g., Google Glass®). In addition, the user interface input device may also include a voice recognition sensing device that enables the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0164] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as auditory / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. In addition, user interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0165] Generally, the use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from the computer system 800 to a user or other computer. User interface output devices include displays. The display subsystem may include non-visual displays such as subsystems, indicator lights, or audio output devices. The display subsystem may also include flat panel devices such as displays using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, and touchscreens. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0166] The storage subsystem 818 provides a repository or datastore for storing information and data used by the computer system 800. The storage subsystem 818 provides a tangible, non-temporary, computer-readable storage medium for storing basic programming and data configurations that provide some example functionality. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 804, provides the above-described functionality may be stored in the storage subsystem 818. The software may be executed by one or more processing units of the processing subsystem 804. The storage subsystem 818 may also provide authentication in accordance with the teachings of this disclosure.

[0167] The storage subsystem 818 may include one or more non-temporary memory devices, including volatile and non-volatile memory devices. As shown in Figure 8, the storage subsystem 818 includes system memory 810 and computer-readable storage medium 822. The system memory 810 may include several memories, including volatile primary random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS), which includes basic routines to assist in the transfer of information between elements within the computer system 800 during startup, etc., may typically be stored in ROM. The RAM contains data and / or program modules currently being operated and executed by the processing subsystem 804. In some implementations, the system memory 810 may include several different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0168] As an example, and not an exhaustive one, as shown in Figure 8, the system memory 810 may load running application programs 812, program data 814, and the operating system 816, which may include various applications such as web browsers, middle-tier applications, and relational database management systems (RDBMS). As an example, the operating system 816 may be Microsoft Windows (registered trademark). Apple Macintosh® and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or iOS®, Windows Phone, Android® OS, BlackBerry® OS, Palm® OS. This could include various versions of mobile operating systems, such as ping systems.

[0169] The computer-readable storage medium 822 may store programming and data structures that provide several example functions. The computer-readable storage medium 822 may provide storage for computer-readable instructions, data structures, program modules, and other data for the computer system 800. When executed by the processing subsystem 804 Software (programs, code modules, instructions) that provides the functions may be stored in the storage subsystem 818. For example, the computer-readable storage medium 822 may include non-volatile memory such as hard disk drives, magnetic disk drives, optical disk drives such as CD-ROMs, DVDs, and Blu-ray® discs, or other optical media. The computer-readable storage medium 822 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, and digital videotapes. The computer-readable storage medium 822 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, and solid-state ROMs, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0170] In certain examples, the storage subsystem 818 may also include a computer-readable storage medium reader 820 which may be further connected to the computer-readable storage medium 822. The reader 820 may be configured to receive and read data from memory devices such as disks, flash drives, etc.

[0171] In certain cases, computer system 800 may support virtualization technologies, including but not limited to the virtualization of processing and memory resources. For example, computer system 800 may provide support for running one or more virtual machines. In certain cases, computer system 800 may run programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing resources (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating system run by other virtual machines run by computer system 800. Thus, potentially multiple operating systems may run simultaneously by computer system 800.

[0172] The communication subsystem 824 provides interfaces to other computer systems and networks. It functions as an interface for sending and receiving data between other systems and the computer system 800. For example, the communication subsystem 824 may enable the computer system 800 to establish communication channels to one or more client devices via the internet in order to send and receive information with one or more client devices. For instance, if the computer system 800 is used to implement the bot system 120 shown in Figure 1, the communication subsystem may be used to communicate with a chatbot system selected for the application.

[0173] The communication subsystem 824 may support both wired and / or wireless communication protocols. In certain examples, the communication subsystem 824 may include (e.g., cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (High Speed ​​Data Rate for Global Evolution), radio frequency (RF) transceiver components for accessing wireless voice and / or data networks using WiFi (IEEE 802.XX family standards, or other mobile communication technologies, or a combination thereof), global positioning system (GPS) receiver components, and / or other components. In some examples, the communication subsystem 824 may also include wireless In addition to or instead of a linear interface, it may provide a wired network connection (e.g., Ethernet®).

[0174] The communication subsystem 824 can receive and transmit data in various formats. In some examples, in addition to other formats, the communication subsystem 824 may receive input communications in the form of structured data feeds and / or unstructured data feeds 826, event streams 828, event updates 830, etc. For example, the communication subsystem 824 may receive social media network and / or Twitter® feeds. Facebook® updates, Rich Site Summary (RSS) feeds, etc. It may be configured to receive (or send) data feeds 826 in real time from users of other communication services, such as web feeds and / or real-time updates from one or more third-party sources.

[0175] In certain examples, the communication subsystem 824 may be configured to receive data in the form of a continuous data stream, which may include an event stream 828 and / or event update 830 of real-time events that are inherently continuous or infinite and do not have a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0176] The communication subsystem 824 may be configured to transmit data from computer system 800 to other computer systems or networks. This data may be transmitted in various different formats, such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., to one or more databases that can communicate with one or more streaming data source computers connected to computer system 800.

[0177] Computer system 800 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phone, iPad® computing tablet, PDA), wearable devices (e.g., Google Glass® head-mounted display), personal computers, workstations, mainframes, kiosks, server racks, or other data processing systems. Because the nature of computers and networks is constantly changing, the description of computer system 800 shown in Figure 8 is intended only as a specific example. Many other configurations with more or fewer components than the system shown in Figure 8 are possible. Based on the disclosures and teachings herein, it should be recognized that there are other embodiments and / or methods for realizing various examples.

[0178] While specific examples have been described, various variations, modifications, alternative configurations, and equivalent examples are possible. The examples are not limited to operation within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, while some examples have been illustrated using specific sets of transactions and steps, it should be apparent to those skilled in the art that this is not intended as limitation. Some flowcharts illustrate operations as sequential processes, but many of these operations may be performed in parallel or concurrently. In addition, the order of operations may be re-specified. Processes may have additional steps not shown in the diagrams. The various features and aspects of the above examples may be used individually or together.

[0179] Furthermore, while specific examples have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are also possible. The specific examples may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0180] Where it is stated that a device, system, component, or module is configured to perform a particular operation or function, such configuration may be achieved, for example, by designing electronic circuits to perform the operation; by programming programmable electronic circuits (such as a microprocessor) to perform the operation; by executing computer instructions or code programmed to execute code or instructions stored in a non-temporary memory medium, or by executing a processor or core, or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication; different pairs of processes may use different techniques; or the same pair of processes may use different techniques at different times.

[0181] This disclosure provides specific details to ensure that the examples are fully understood. However, the examples can be implemented without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary details in order to avoid ambiguity in the examples. This specification provides only exemplary examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples will provide a description that enables the implementation of various examples for those skilled in the art. Various modifications can be made within the scope of the function and configuration of the elements.

[0182] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be clear that additions, reductions, deletions, and other modifications and changes may be made to them without departing from the broader spirit and scope as set forth in the claims. Thus, specific examples have been described, but these are not intended to be limiting. Various variations and equivalents are within the scope of the attached claims.

[0183] While the above specification illustrates aspects of the disclosure with reference to specific examples, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the above disclosure may be used individually or together. Furthermore, the examples can be used in a variety of environments and applications beyond those described herein, without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0184] In the above description, the method is presented in a specific order for illustrative purposes. It should be understood that in alternative examples, the method may be executed in a different order than described. It should also be understood that the above method may be executed by hardware components or embodied in a sequence of machine-executable instructions, which, when used, can be used to cause a machine, such as a general-purpose or dedicated processor or logic circuit programmed with such instructions, to execute the method. These machine-executable instructions may be stored on one or more machine-readable media, such as CD-ROMs or other types of optical discs, floppy disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, these methods may be executed by a combination of hardware and software.

[0185] Where components are described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing an electronic circuit or other hardware to perform a particular operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform a particular operation, or by any combination thereof.

[0186] While specific examples of this application are described in detail here, it should be understood that the concept of the present invention can be embodied and adopted in various other forms, and that the claims are intended to be interpreted to include such variations, unless limited by the prior art.

Claims

1. It is a method, A data processing system includes the steps of receiving a training set of utterances to train a machine learning model to identify one or more intents for one or more utterances, The aforementioned data processing system generates the aforementioned data in out-of-domain (OOD) examples. This includes a step of expanding the training set of speeches, the expanding step of, Steps to generate a dataset of OOD examples, The steps include filtering out and removing OOD examples from the OOD example dataset that have a context substantially similar to the context of the utterance, from the training set of utterances, The steps include determining the difficulty value for each OOD example remaining in the filtered OOD example dataset, The method further includes the step of generating a batch of augmented utterances, which includes utterances from a training set of utterances and utterances from a filtered dataset of OODs, based on the difficulty value for each OOD, A method comprising the step of training the machine learning model using the expanded utterances by the data processing system, wherein the training step includes supplying the expanded utterances to the machine learning model based on the difficulty values ​​of the OOD examples in each batch of the expanded utterances.

2. The method according to claim 1, wherein the substantial similarity between the context of the OOD example and the context of the utterance in the training set of utterances is determined on a distance scale to avoid class mismatches.

3. The substantial similarity between the context of the aforementioned OOD example and the context of the aforementioned utterance in the training set of utterances was determined using a Multilingual Universal Sentence Encoder (MUSE) single embedding. The method according to claim 2, wherein if min(d_i) < a predetermined threshold (d_i = Euclidean distance(v_i, u)), the context of the OOD example and the context of the utterance in the training set of utterances are determined to be substantially similar.

4. The method according to claim 3, wherein the difficulty value is derived from the d_i value such that easy OOD examples have a high d_i value, while difficult OOD examples have a lower d_i value that is closer to the predetermined threshold.

5. The method according to claim 4, wherein the batches of the expanded utterances are generated subject to the following constraints, the following constraints: (i) a predetermined batch size; (ii) inclusion of a predetermined number of in-domain examples of utterances from a training set of the utterances in each batch; (iii) inclusion of a predetermined number of OOD examples of utterances from a filtered OOD dataset in each batch; (iv) the predetermined number of in-domain examples and the predetermined number of OOD examples of utterances are selected to maintain the predetermined batch size; (v) the OOD examples of utterances included in each batch have substantially similar difficulty values; and optionally, (vi) each ODD example from the filtered OOD dataset is included in only one batch of the batches of expanded utterances.

6. The method according to any one of the prior claims, further comprising the step of deploying the trained machine learning model in a chatbot system.

7. The method according to any one of the preceding claims, wherein the batch of extended utterances is generated using a batch balancing scheme.

8. It is a system, One or more processors, The memory includes one or more processors, the memory stores a plurality of instructions that can be executed by the one or more processors, and the plurality of instructions, when executed by the one or more processors, include instructions that cause the one or more processors to perform the following operations, the following operations are: The process involves receiving a training set of utterances to train a machine learning model to identify one or more intents for one or more utterances, This includes an action to extend the training set of utterances with an out-of-domain (OOD) example, the extension action being: The process of generating a dataset of OOD examples, The operation of filtering out and removing OOD examples from the OOD example dataset that have a context substantially similar to the context of the utterance, from the training set of the utterances, The operation of determining the difficulty value for each OOD example remaining in the filtered OOD example dataset, The operation includes generating a batch of augmented utterances, which includes utterances from the training set of utterances and utterances from the filtered OOD dataset, based on the difficulty value for each OOD, and further includes: A system comprising the operation of training the machine learning model using the aforementioned batches of augmented utterances, wherein the training operation comprises the operation of supplying the batches of augmented utterances to the machine learning model based on the difficulty values ​​of the OOD examples in each batch of the batches of augmented utterances.

9. The system according to claim 8, wherein the substantial similarity between the context of the OOD example and the context of the utterance in the training set of utterances is determined on a distance scale to avoid class mismatches.

10. The system according to claim 9, wherein the substantial similarity between the context of the OOD example and the context of the utterance in the training set of utterances is determined using a multilingual universal sentence encoder (MUSE) single embedding, and the context of the OOD example and the context of the utterance in the training set of utterances are determined to be substantially similar if min(d_i) < a predetermined threshold (d_i = Euclidean distance (v_i, u)).

11. The system according to claim 10, wherein the difficulty value is derived from the d_i value such that easy OOD examples have a high d_i value, while difficult OOD examples have a lower d_i value that is closer to the predetermined threshold.

12. The batches of the expanded utterances are generated based on the following constraints: (i) a predetermined batch size; (ii) inclusion of a predetermined number of in-domain examples of utterances from the training set of the utterances in each batch; (iii) inclusion of a predetermined number of OOD examples of utterances from the filtered OOD dataset in each batch; (iv) the predetermined number of in-domain examples and the predetermined number of OOD examples of utterances are selected to maintain the predetermined batch size; and (v) the OOD examples of utterances included in each batch have substantially similar difficulty values, and are arbitrary. The system according to claim 11, wherein (vi) each ODD example from the filtered OOD dataset is incorporated into only one batch of the extended utterances.

13. The system according to any one of claims 8 to 12, wherein the aforementioned operations further include deploying the trained machine learning model in the chatbot system.

14. The system according to any one of claims 8 to 13, wherein the batch of extended utterances is generated using a batch balancing scheme.

15. A non-temporary computer-readable memory that stores a plurality of instructions that can be executed by one or more processors, wherein the plurality of instructions, when executed by the one or more processors, include instructions that cause the one or more processors to perform the following operations, and the following operations are: The process involves receiving a training set of utterances to train a machine learning model to identify one or more intents for one or more utterances, This includes an action to extend the training set of utterances with an out-of-domain (OOD) example, the extension action being: The process of generating a dataset of OOD examples, The operation of filtering out and removing OOD examples from the OOD example dataset that have a context substantially similar to the context of the utterance, from the training set of the utterances, The operation of determining the difficulty value for each OOD example remaining in the filtered OOD example dataset, The operation includes generating a batch of augmented utterances, which includes utterances from the training set of utterances and utterances from the filtered OOD dataset, based on the difficulty value for each OOD, and further includes: Non-temporary computer-readable memory, which includes an operation to train the machine learning model using the aforementioned batches of augmented utterances, the training operation including an operation to supply the machine learning model with the batches of augmented utterances based on the difficulty values ​​of the OOD examples in each batch of the batches of augmented utterances.

16. Non-temporary computer-readable memory according to claim 15, wherein the substantial similarity between the context of the OOD example and the context of the utterance in the training set of utterances is determined on a distance scale to avoid class mismatches.

17. Non-temporary computer-readable memory according to claim 16, wherein the substantial similarity between the context of the OOD example and the context of the utterance in the training set of utterances is determined using a multilingual universal sentence encoder (MUSE) single embedding, and if min(d_i) < a predetermined threshold (d_i = Euclidean distance (v_i, u)), then the context of the OOD example and the context of the utterance in the training set of utterances are determined to be substantially similar.

18. The non-temporary computer-readable memory according to claim 17, wherein the difficulty value is derived from the d_i value such that easy OOD examples have a high d_i value, while difficult OOD examples have a lower d_i value that is closer to the predetermined threshold.

19. The batches of the extended utterances are generated based on the following constraints: (i) a predetermined batch size, and (ii) the training set of the utterances within each batch. Non-temporary computer-readable memory according to claim 18, comprising: (iii) incorporating a predetermined number of intradomain examples of utterances from; (iv) incorporating a predetermined number of OOD examples of utterances from the filtered OOD dataset in each batch; (iv) the predetermined number of intradomain examples and the predetermined number of OOD examples of utterances are selected to maintain the predetermined batch size; (v) the OOD examples of utterances incorporated in each batch have substantially similar difficulty values; and optionally, (vi) each ODD example from the filtered OOD dataset is incorporated into only one batch of the expanded batches.

20. The non-temporary computer-readable memory according to any one of claims 15 to 19, further comprising the operation of deploying the trained machine learning model in a chatbot system.

Images