Multiple feature balancing for natural language processors

Abstract

Techniques are disclosed for a system that includes techniques for multiple feature balancing for a natural language processor.SOLUTION: The method includes receiving a natural language query to be processed by a machine learning model. The machine learning model utilizes a dataset of natural language phrases to process the natural language query. The method further includes determining, based on the machine learning model and the natural language query, a feature dropout value, generating, based on the natural language query, one or more contextual features and one or more expressive features that can be input to the machine learning model, modifying at least one of the one or more contextual features and the one or more expressive features based on the feature dropout value to generate a set of input features for the machine learning model, and processing the set of input features to generate an output data set for responding to the natural language query.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] Priority claim This application is a non-provisional application of U.S. Provisional Patent Application No. 63 / 139,695, entitled "MULTI-FACTOR BALANCING FOR TRAINING NATURAL LANGUAGE PROCESSORS," filed on January 20, 2021. This provisional application claims the benefit of and priority to, and is incorporated herein by reference in its entirety for all purposes.

[0002] FIELD OF THE INVENTION The present disclosure relates generally to chatbot systems, and more particularly to techniques for multi-feature balancing for training and implementing chatbot systems in natural language processing systems. [Background technology]

[0003] background Instant messaging capabilities and automated chat platforms are efficient solutions to modern customer service problems. Organizations can leverage these implementations to provide timely and agile service to customers without investing valuable human capital in individual user inquiries. These chatbots are configured to process queries, sometimes called utterances, written in a natural language format that simulates human speech patterns. Unlike human-to-human oral speech patterns, written natural language phrases often do not capture important elements inherent in phrases, such as speech intonation, context, emphasis, and other elements. Therefore, it can be difficult for computer systems to process written natural language queries and determine appropriate responses thereto. This is particularly problematic for chatbot systems configured to generate responses to natural language queries. Humans interacting with chatbots may become frustrated or annoyed with, or even stop using, a chatbot system if the chatbot does not respond appropriately to their natural language queries.

[0004] Minor differences in the context, intonation, spelling, tone, and / or setting of a customer inquiry can result in the selection of the wrong chatbot / skill for a particular task. Errors in selecting a chatbot can rapidly compound when an organization performs hundreds or thousands of automated inquiry responses daily. Simple methods for selecting a chatbot, such as a one-to-one mapping of words to a specific chatbot, may not take into account proper contextual analysis and do not consider the complexity of the conversation. To aid in skill selection, a chatbot may use a machine learning model to process an utterance and output the most likely skill to respond to the utterance. The selection of skills to assist in answering an inquiry can be based on a contextual and lexical analysis of the inquiry provided to the organization. To train these models to process a wide range of utterances, the machine learning model is extensively trained using a training dataset of utterances, which allows it to refine its operating parameters and better “recognize” linguistic patterns within the utterance.

[0005] It is extremely difficult to train a machine learning model using every natural language phrase the model may encounter. Doing so would require extensive training sets and require enormous amounts of training time. In addition, such training can lead to "overfitting," a situation in which a natural language processing model associates natural language phrases with accurate ground truth or "gold" labels in the training data. This risks being overlooked, meaning that the model will have difficulty processing phrases for which it has not been trained. To ameliorate these problems, a natural language processor may utilize a dataset of natural language phrases that correspond to a category of labels, called a "gazetteer." While processing the natural language phrase, the natural language model may recognize certain words that are also contained in the gazetteer. When processing the natural language phrase, the model may weight the presence of the phrase in the gazetteer and the corresponding category label associated with the gazetteer. For example, the natural language processor may convert the utterance "I would like to see a map of Colerain, Ohio" into a natural language phrase. The utterance may be input into a machine learning model to predict the chatbot skill for processing the utterance. Although the training dataset used to train the model is highly unlikely to contain the word "Colerain, Ohio," a corresponding gazetteer with an associated category label "Location" may contain that word. Thus, when the model processes the utterance to determine the category label associated with the utterance, it may weight the fact that the "Location" gazetteer contained the same word.

[0006] The use of gazetteer introduces expressive features to the natural language processor that aid in the prediction of associated labels for input natural language queries. However, the expressive properties of gazetteer may cause inappropriate weighting of certain natural language phrases toward gazetteer-based expressive features relative to contextual features determined by the natural language processor. For example, the utterance "please mark these papers" contains the word "mark," which corresponds to a gazetteer list of common names with the category label "Names," because "Mark" is a common name. However, the word "mark" is used as a verb in this given utterance. Although this utterance is not related to a name, the model's reliance on the expressive features generated by the gazetteer would cause the model to incorrectly predict that the label "Name" is associated with the utterance. Thus, the absence of a gazetteer requires the natural language processing model to process phrases that are unlikely to be present in the training data used to train the model, and the introduction of a gazetteer may cause the model to rely too heavily on the expressive features of the gazetteer at the expense of the contextual features generated by the model. Summary of the Invention [Means for solving the problem]

[0007] overview A technique for multi-factor balancing for training chatbot systems in natural language processing is disclosed.

[0008] In certain exemplary embodiments, a computer-implemented method includes: a computing device receiving an indication of a first coverage value corresponding to a desired overlap between a dataset of natural language phrases and a training dataset for training a machine learning model; the computing device determining a second coverage value corresponding to a measured overlap between the dataset of natural language phrases and the training dataset; the computing device determining a coverage delta value based on a comparison between the first coverage value and the second coverage value; the computing device modifying at least one of the dataset of natural language phrases and the training dataset based on the coverage delta value; and the computing device processing an input dataset including a set of input features using a machine learning model including the modified dataset of natural language phrases, wherein the machine learning model processes the input dataset based at least in part on the dataset of natural language phrases to generate an output dataset.

[0009] In some examples, the method further includes determining a second coverage value by determining a number of natural language phrases from the dataset of natural language phrases that are also present in the training data, wherein each of the natural language phrases that are also in the dataset corresponds to a category that matches a category associated with the dataset of natural language phrases. In some further examples, modifying at least one of the dataset of natural language phrases and the training dataset includes modifying the dataset of natural language phrases by updating the dataset of natural language phrases to include one or more natural language phrases associated with categories from the training data, wherein the updated dataset of natural language phrases includes a number of natural language phrases that are also present in the training data in a proportion that is equal to or greater than the first coverage value. In other further examples, modifying at least one of the dataset of natural language phrases and the training dataset includes modifying the training dataset by updating the training dataset to include one or more natural language phrases from the dataset of natural language phrases and associating the one or more natural language phrases with categories, wherein the dataset of natural language phrases includes a number of natural language phrases that are also present in the updated training data in a proportion that is equal to or greater than the first coverage value.

[0010] In some further examples, updating the training dataset to include one or more natural language phrases from the dataset of natural language phrases includes generating one or more training pairs from the one or more natural language phrases, the one or more training pairs including a natural language query generated from the natural language phrase and a gold label category that matches a category of the dataset of natural language phrases. In some further examples, processing the input dataset includes the machine learning model processing the updated training dataset to retrain the machine learning model.

[0011] In some examples, processing the input dataset includes a machine learning model processing a natural language query received by the chatbot system, wherein the machine learning model is configured to generate an output dataset including at least one of skills and intents associated with the chatbot for responding to the natural language query. In some examples, the machine learning model is a convolutional neural network machine learning model, and the set of input features corresponds to input nodes of the convolutional neural network.

[0012] In another particular illustrative embodiment, a computer-implemented method includes: a computing device receiving a natural language query to be processed by a machine learning model, the machine learning model utilizing a dataset of natural language phrases to process the natural language query; the method further includes: the computing device determining a feature dropout value based on the machine learning model and the natural language query; the computing device generating, based on the natural language query, one or more contextual features and one or more expressive features that can be input to the machine learning model; the computing device modifying at least one of the one or more contextual features and the one or more expressive features based on the feature dropout value to generate a set of input features for the machine learning model; and the computing device utilizing the machine learning model to process the set of input features to generate an output dataset corresponding to the natural language query.

[0013] In some examples, the feature dropout value is a first contextual feature dropout value corresponding to a percentage of the contextual features of the one or more contextual features, and the method further includes modifying the one or more contextual features by removing a percentage of the contextual features from the one or more contextual features based on the first contextual feature dropout value, and the set of input features is a set of the modified one or more contextual features and one or more expressive features. In some further examples, the feature dropout value further includes a second contextual feature dropout value corresponding to a percentage of contextual features of the one or more contextual features that correspond to a natural language phrase in the dataset of natural language phrases, the method further including determining a subset of contextual features, each contextual feature of the subset of contextual features corresponding to a natural language phrase in the dataset of natural language phrases, the method further including modifying the subset of contextual features by removing a percentage of contextual features that correspond to the second contextual feature dropout value from the subset of contextual features, and modifying the one or more contextual features includes removing a percentage of contextual features that comprise the modified subset of contextual features from the one or more contextual features based on the first contextual feature dropout value.

[0014] In some examples, the feature dropout value is a first expressive feature dropout value corresponding to a percentage of expressive features among the one or more expressive features, and the method further includes modifying the one or more expressive features by removing a percentage of expressive features from the one or more expressive features based on the first expressive feature dropout value, and the set of input features is generated from the one or more contextual features and the modified one or more expressive features.

[0015] In some examples, the method further includes comparing the dataset of natural language phrases to a training dataset used to train the machine learning model and determining a noise value based on the comparison, the noise value corresponding to a number of natural language phrases associated with the same particular category in the dataset of natural language phrases and the training dataset and a number of natural language phrases associated with different categories in the dataset of natural language phrases and the training dataset, and the feature dropout value is determined based at least in part on the noise value. In some examples, the machine learning model is a convolutional neural network machine learning model, and the set of input features corresponds to input nodes of the convolutional neural network.

[0016] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0017] The techniques described above and below can be implemented in several ways and in several contexts. Some example implementations and contexts are provided, as described in more detail below and with reference to the following drawings. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a simplified block diagram of a distributed environment incorporating an illustrative embodiment; [Figure 2] FIG. 1 is a simplified block diagram of a parent bot (MB) system according to one embodiment. [Figure 3] FIG. 1 is a simplified block diagram of a Skillbot system according to an embodiment. [Figure 4] FIG. 1 is a simplified block diagram of a computing system implementing a language processing system. [Figure 5] FIG. 1 illustrates a process flow for managing a dataset of natural language phrases and a training dataset for a machine learning model, according to various embodiments. [Figure 6] FIG. 1 illustrates an exemplary dataset of natural language phrases and a training dataset utilized as part of a multi-factor model for natural language processing, according to various embodiments. [Figure 7] FIG. 1 illustrates a process flow for performing feature dropout as part of multi-feature balancing for a natural language processor, according to various embodiments. [Figure 8] FIG. 1 is a simplified block diagram of a skill classifier artificial neural network machine learning model that utilizes feature dropout, according to an embodiment. [Figure 9] FIG. 1 illustrates a process flow for performing noise-based feature dropout as part of multi-feature balancing for a natural language processor, according to various embodiments. [Figure 10] 1 shows a simplified diagram of a distributed system for implementing various embodiments. [Figure 11] FIG. 1 is a simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment system may be offered as cloud services, according to various embodiments. [Figure 12] FIG. 1 illustrates an exemplary computer system that can be used to implement various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0019] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0020] As described above, chatbots are useful tools for interacting with and communicating with human clients in a natural language format. Chatbot operators strive to improve their chatbots to interact as closely as possible with another human. Humans who interact with a well-configured chatbot have a more pleasant experience interacting with the chatbot, and their queries are answered more quickly. Therefore, it is highly advantageous for a chatbot to quickly and accurately process and respond to natural language speech generated by humans. A chatbot processes input utterances, including natural language queries, from humans and, in response, generates an output, e.g., a selection of a skill for responding to the natural language query. A skill is, for example, a subroutine of a chatbot specifically trained or configured to accomplish some task in response to or on behalf of a human. In some examples, a chatbot processes a natural language query including the utterance and outputs a predicted label for the utterance that corresponds to a predicted category associated with the utterance and can be used to select a corresponding skill to respond to the utterance. The skill subroutine then performs some response action to resolve the query.

[0021] Natural language processing inherently involves many complexities that make predicting corresponding labels from natural language queries difficult to implement. For example, the contextual or "lexical" features of a sentence are not easily parsed by computer-based natural language processors. The phrase "I will subscribe to this service when pigs fly" is a simple satirical phrase that most humans will recognize, but that many natural language processors will not recognize. A simple natural language processor would be unable to interpret the partial phrase "I will subscribe to th A chatbot may only recognize the phrase "I will subscribe to this service" and ignore the rest of the phrase, thus predicting that a client (e.g., a human using the chatbot service via an automated virtual assistant program) wishes to subscribe to the service, when in fact the opposite is true. Chatbots utilize machine learning models that are trained to consider context and other complex language-based features in natural language productions. These machine learning models are trained using a set of training data and associated with several "gold labels" or "ground truth labels" that represent the correct labels that the machine learning model should predict given a natural language query as input. The parameters of the machine learning model are adjusted during training to more accurately predict labels given at least somewhat similar natural language queries as input. For example, a trained machine learning model may also recognize the phrase "I will subscribe to this service," but not "when pigs fly." It will also recognize phrases, determine that they correspond to certain conditions, and predict labels that will determine that the conditions are almost certainly false and prevent the chatbot from automatically subscribing the user to the service.

[0022] Machine learning models often receive one or more features as input. Features represent aspects of a natural language phrase and are data processed by the model to ultimately output a prediction, such as a predicted label for the natural language phrase. For example, a natural language phrase is preprocessed to generate several features that correspond to input nodes of an artificial neural network (ANN) machine learning model. The ANN may process the feature input through a series of hidden layers until an output prediction is generated. A larger number of input nodes, and therefore a larger number of features, allows the machine learning model to process multiple aspects of a natural language phrase, such as context, intonation, mood, and semantic information.

[0023] The accuracy of predictions made by a machine learning model is largely based on the quality and quantity of training data used to train the model. However, due to the vast number of words present in natural language, it is very difficult to train a machine learning model with a wide range of natural language vocabulary. Attempting to create training data for every possible natural language word would be highly inefficient, and doing so for every possible inflection, context, etc. for each word would be nearly impossible. Instead, a machine learning model can supplement the creation of contextual features with the creation of expressive features using a pre-generated list of natural language phrases, referred to herein as a "gazetteer." For example, rather than generating a training dataset with every possible proper noun name in English, a gazetteer containing an extensive list of known English names may be utilized by the machine learning model. Received natural language phrases can similarly be preprocessed to generate a set of expressive features that can be input into the machine learning model to generate output. For example, given the received natural language query "I would like to visit Colerain, Ohio," it is highly unlikely that the machine learning model has been trained to recognize the location "Colerain, Ohio." However, a gazetteer associated with a word known to be "Towns" may contain the word "Colerain, Ohio," and a set of expressive features is included as input to the machine learning model that weights the model in favor of outputting the label "Towns."

[0024] Gazetteers and other expressive lists of natural language phrases have drawbacks that can harm machine learning models even more than they would without them. For example, homophone words and phrases present great difficulties to natural language processors. The phrase "Mark is a friend of mine" uses the word "Mark" as a proper noun, but "Mark these papers and return them to me" The phrase uses the word "Mark" as a verb. The context, intonation, spelling, and tone of the customer inquiry , and / or slight variations in configuration may result in the selection of the wrong chatbot / skill for a particular task. A gazetteer utilizing a list of words will result in the generation of expressive features that are weighted in favor of classifying a natural language phrase as "Name" when in fact the correct label may be "Test Grading."

[0025] To overcome these and other challenges, this document describes techniques for multi-feature balancing for natural language processing for training and deploying chatbots / skills to process and respond to natural language-based queries. As described herein, multi-feature balancing refers to the use of contextual information, including contextual features, and expressive information, including expressive features, to generate more accurate and efficient predictions and improve machine learning models. More specifically, the techniques described herein relate to improvements for balancing the use of contextual features generated by a trained machine learning model with expressive features generated by gazetteers and other expressive lists. The described techniques include processes for altering the composition of gazetteers and training data to improve natural language processing and direct manipulation of features generated therefrom to promote more accurate and efficient predictions by machine learning models.

[0026] For example, as described above, machine learning models that rely solely on contextual and lexical features to process natural language phrases are often not adequately trained on training datasets that are comprehensive enough to recognize most natural language phrases with sufficient accuracy. The introduction of gazetteers to compensate for this deficiency tends to overfit model predictions to expressive features that do not fully consider context. Balancing the use of contextual and expressive features to improve model predictions, as well as direct manipulation of training data and gazetteer configuration, as part of multi-feature balancing for natural language processing, enables machine learning techniques to appropriately balance both types of features, thus improving chatbot accuracy and client interaction.

[0027] Exemplary Chatbot System A bot (also referred to as a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. Bots are generally capable of responding to natural language messages (e.g., questions or comments) through messaging applications using natural language messages. Businesses may use one or more bot systems to communicate with end users through messaging applications. The messaging application, sometimes referred to as a channel, may be the end user's preferred messaging application that the end user already has installed and is familiar with. Thus, end users do not need to download and install a new application to chat with a bot system. Messaging applications may be, for example, over-the-top (OTT) messaging channels (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (e.g., Amazon Dot, Echo, or Show, Google® Home, Apple HomePod, etc.), or messaging services. ), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat functionality, or voice-based input (e.g., Siri, Cortana, Google Voice, or other voice input for interaction) This may include a device or app with an interface for using the device.

[0028] In some examples, a bot system can be associated with a Uniform Resource Identifier (URI). The URI can identify the bot system using a string of characters. The URI can be used as a webhook for one or more messaging application systems. The URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The bot system may be designed to receive a message (e.g., a HyperText Transfer Protocol (HTTP) postcall message) from a messaging application system. The HTTP postcall message may be directed to a URI from the messaging application system. In some embodiments, the message may differ from the HTTP postcall message. For example, the bot system may receive a message via Short Message Service (SMS). Although the discussion herein may refer to a communication received by the bot system as a message, it should be understood that the message may be an HTTP postcall message, an SMS message, or any other type of communication between the two systems.

[0029] End users can interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), much like interactions between people. In some cases, interactions begin with the end user saying "Hello" to the bot. , the bot may respond with "Hi" and ask the end user how it can assist them. In some cases, the interaction may also be a transactional interaction with a banking bot, e.g., transferring money from one account to another; an informational interaction with an HR bot, e.g., checking a vacation balance; or an interaction with a retail bot, e.g., discussing returning a purchased item or seeking technical support.

[0030] In some embodiments, the bot system can intelligently handle end-user interactions without interaction with an administrator or developer of the bot system. For example, an end-user may send one or more messages to the bot system to achieve a desired goal. The messages may include content such as text, emojis, audio, images, video, or other methods of conveying a message. In some embodiments, the bot system converts the content into a standardized format (e.g., a representational state transfer (REST) ​​call to an enterprise service with appropriate parameters) and provides a natural language response. The bot system may generate an answer. The bot system may also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system may also initiate communication with the end user rather than passively responding to end user utterances. Various techniques are described herein for identifying explicit invocations of the bot system and determining inputs for the invoked bot system. In some embodiments, explicit invocation analysis is performed by the parent bot based on detecting a call name in the utterance. In response to detecting the call name, the utterance may be refined for input to a skill bot associated with the call name.

[0031] A conversation with a bot can follow a specific conversational flow that includes multiple states. The flow can define what happens next based on input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in and from states. A conversation can take different paths based on end-user input, which can affect the decisions the bot makes about the flow. For example, at each state, based on the end-user input or utterance, the bot can determine the end-user's intent and decide the appropriate action to take next. Here, and in the context of utterances, the term "intent" refers to the intent of the user who gave the utterance. For example, a user intends to engage a bot in a conversation to order a pizza, and the user's intent may be expressed by the utterance "order a pizza." A user's intent can be directed to a specific task that the user wants the chatbot to perform on their behalf. Thus, an utterance can be expressed as a question, command, request, etc. that reflects the user's intent. An intent is a goal that the end user wants to achieve. may include:

[0032] In the context of chat configuration, the term “intent” is used herein to refer to configuration information for mapping a user’s utterance to a specific task / action or category of task / action that a chatbot can perform. To distinguish between an utterance intent (i.e., a user’s intent) and a chatbot’s intent, the latter may be referred to herein as a “bot intent.” A bot intent may include a set of one or more utterances associated with the intent. For example, an intent for ordering a pizza may have various permutations of utterances expressing a desire to place a pizza order. These associated utterances may be used to train the chatbot’s intent classifier, which can then determine whether an input utterance from a user matches the pizza ordering intent. A bot intent may be associated with one or more dialog flows for initiating a conversation with a user in a certain state. For example, the first message for a pizza ordering intent may be the question, “What kind of pizza would you like?” In addition to the associated utterance, a bot intent may further include a named entity associated with the intent. For example, a pizza ordering intent may include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, pizza quantity, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0033] FIG. 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102, which enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as shown in FIG. 1, a user 104 representing a particular business can use the DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, a bank can use the DABP 102 to create one or more digital assistants for use by the bank's customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0034] For purposes of this disclosure, a "digital assistant" is an entity that helps a user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant may be implemented solely using software (e.g., a digital assistant is a digital entity implemented using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as a computer, a mobile phone, a watch, an appliance, a vehicle, etc. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0035] A digital assistant, such as digital assistant 106 built using DABP 102, can be used to perform a variety of tasks through a natural language-based conversation between the digital assistant and its user 108. As part of the conversation, the user provides one or more user inputs 110 to the digital assistant 106, which in turn communicates with the digital assistant. The digital assistant 106 may provide a response 112 to the user. A conversation can include one or more of the input 110 and the response 112. Through these conversations, the user can request that one or more tasks be performed by the digital assistant 106, and in response, the digital assistant 106 is configured to perform the user-requested task and respond to the user with an appropriate response.

[0036] User input 110 is generally in the form of natural language and is referred to as speech. User utterance 110 can be in text form, such as when a user types a sentence, a question, a piece of text, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user utterance 110 can be in the form of voice input or speech, such as when a user says or speaks something that is provided as input to the digital assistant 106. The speech is typically in the language spoken by the user 108. For example, the speech may be in English or some other language. If the speech is in voice form, the voice input is converted into textual speech in that particular language, and the textual speech is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert the voice or auditory input into textual speech, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion may be performed by the digital assistant 106 itself.

[0037] The utterance, which may be a text utterance or a voice utterance, may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing on the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intent. For purposes of this disclosure, it is assumed that the utterance is a text utterance provided directly by a user 108 of the digital assistant 106 or is the result of converting an input voice utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0038] For example, user 108's input may request that a pizza be ordered by providing an utterance such as, "I want to order a pizza." Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action may include responding to the user with a question requesting user input regarding, for example, the type of pizza the user wants to order, the size of the pizza, any toppings on the pizza, etc. The responses provided by digital assistant 106 may also be in natural language format, typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 may perform natural language generation (NLG). Through a conversation between the user and digital assistant 106, for the user to order a pizza, the digital assistant may guide the user to provide all necessary information to order the pizza and then, at the end of the conversation, have the user order the pizza. Digital assistant 106 may end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0039] At a conceptual level, digital assistant 106 performs various operations in response to utterances received from a user. In some embodiments, this operation includes, for example, understanding the meaning of the input utterance (sometimes called natural language understanding (NLU)), determining an action to be taken in response to the utterance, and, if appropriate, carrying out the action. NLU processing involves a series or pipeline of processing steps, including triggering a user utterance, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing may include parsing a received input utterance to understand the structure and meaning of the utterance, and refining and restructuring the utterance to develop a more understandable form (e.g., a logical form) or structure for the utterance. Generating a response may include using NLG techniques.

[0040] NLU processing performed by a digital assistant, such as digital assistant 106, may include various NLP-related processes, such as sentence analysis (e.g., tokenization, reordering, identifying part-of-speech tags for sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, dividing sentences into clauses, analyzing individual clauses, resolving anaphora, performing chunking, etc.). In an embodiment, the NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 can use other resources to perform portions of the NLU processing. For example, the syntax and structure of an input spoken sentence may be identified by processing the sentence using syntactic parsing, part-of-speech tagging, and / or named entity recognition. In one implementation, for English, syntactic parsing, part-of-speech tagging, and named entity recognition, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0041] Although the various examples provided in this disclosure show English utterances, this is meant as an example only. In certain embodiments, the digital assistant 106 can also process utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes NLU processing flexible and extensible for each language, including allowing for different orders of processing. Language packs may be provided for individual languages, and the language packs can register a list of subsystems that can be served from the NLU core server.

[0042] 1 can be made available or accessible to its user 108 through a variety of different channels, such as, but not limited to, through an application, through a social media platform, through various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for it, so that it can run on and be accessed by different services simultaneously.

[0043] A digital assistant or chatbot system typically includes or is associated with one or more skills. In one embodiment, these skills are individual chatbots (referred to as skillbots) that interact with a user and are configured to fulfill specific types of tasks, such as tracking inventory, submitting a timecard, creating an expense report, ordering food, checking a bank account, making a reservation, or purchasing a widget. For example, in the embodiment shown in FIG. 1 , the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of this disclosure, the term "skill" is used synonymously with the term "skillbot."

[0044] Each skill associated with a digital assistant helps the digital assistant user complete a task through a conversation with the user, where the conversation is a combination of text or auditory input provided by the user and responses provided by the skill bot. These responses may be in the form of text or audio messages to the user and / or with simple user interface elements (e.g., selection lists) presented to the user for selection.

[0045] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In one example, a skillbot may be developed by a company and then added to a digital assistant using DABP 102. In another example, a skillbot may be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet another example, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills aimed at a wide range of tasks. Skills offered through the skill store may also expose various cloud services. To add a skill to a digital assistant created using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and indicate that the selected skill be added to the digital assistant created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in modified form (e.g., a user of DABP102 may select and clone a particular skillbot provided by the skill store, customize or modify the selected skillbot, and then add the modified skillbot to a digital assistant created with DABP102).

[0046] A variety of different architectures may be used to implement a digital assistant or chatbot system. For example, in some embodiments, a digital assistant created and deployed using DABP 102 may be implemented using a parent bot / child (or sub)bot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a parent bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in FIG. 1, digital assistant 106 includes parent bot 114 and skill bots 116-1, 116-2, etc., that are child bots of parent bot 114. In certain embodiments, digital assistant 106 itself may act as a parent bot.

[0047] A digital assistant implemented according to the parent-child bot architecture allows users of the digital assistant to interact with multiple skills through a unified user interface, i.e., through a parent bot. When a user engages with the digital assistant, user input is received by the parent bot. The parent bot then performs processing to determine the meaning of the user input utterance. The parent bot then determines whether the task requested by the user in the utterance can be handled by the parent bot itself. If not, the parent bot selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows users to converse with the digital assistant through a common, single interface while still providing the ability to use several skill bots configured to perform specific tasks. For example, in a digital assistant developed for an enterprise, the parent bot of the digital assistant can interface with skill bots with specific functions, such as a CRM bot for performing functions related to customer relationship management (CRM), an ERP bot for performing functions related to enterprise resource planning (ERP), and an HCM bot for performing functions related to human capital management (HCM). In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common parent bot interface, and behind the scenes, multiple skill bots are provided to handle user requests.

[0048] In one embodiment, in a parent bot / child bot infrastructure, the parent bot is configured to recognize an available list of skill bots. The parent bot may have access to various available skill bots and, for each skill bot, metadata identifying each skill bot's capabilities, including the tasks that can be performed by each skill bot. Upon receiving a user request in the form of an utterance, the parent bot is configured to identify or predict a specific skill bot from multiple available skill bots that can best accommodate or process the user request. The parent bot then routes the utterance (or a portion of the utterance) to that specific skill bot for further processing. Thus, control flows from the parent bot to the skill bot. A parent bot can support multiple input and output channels.

[0049] 1 illustrates a digital assistant 106 with a parent bot 114 and skill bots 116-1, 116-2, and 116-3, but this is not intended to be limiting. A digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be realized solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an implementation using a combination of software and hardware.

[0050] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants, including one or more skillbots associated with the digital assistant. In some cases, a skillbot can be created by cloning an existing skillbot, for example, by cloning a skillbot provided by a skill store. As described above, DABP 102 provides a skill store or skill catalog that offers multiple skillbots for performing various tasks. Users of DABP 102 can clone a skillbot from the skill store. They may modify or customize the cloned skillbot as needed. In some other cases, users of DABP 102 created a skillbot from scratch using tools and services provided by DABP 102. As described above, the skill store or skill catalog provided by DABP 102 may offer multiple skillbots for performing various tasks.

[0051] In certain embodiments, at one high level, creating or customizing a skillbot includes the following steps: (1) Set up the settings for the new skill bot (2) Configure one or more intents for the skill bot (3) Set one or more entities for one or more intents. (4) Training the SkillBot (5) Create a dialog flow for your skill bot (6) Add custom components to your skill bot as needed (7) Test and deploy the skill bot. Each step will be briefly described below.

[0052] (1) Set Settings for a New Skillbot—Various settings may be set for a skillbot. For example, a skillbot designer can specify one or more call names for the skillbot being created. These call names can then be used by a user of the digital assistant to explicitly call the skillbot. For example, a user can explicitly call the corresponding skillbot by entering a call name in the user's utterance. It can be called explicitly.

[0053] (2) Configure one or more intents and associated example utterances for the skillbot—A skillbot designer specifies one or more intents (also called bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes that the skillbot is trained to infer about input utterances. Upon receiving an utterance, the trained skillbot infers the intent of the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes an appropriate action to respond to the utterance based on the intent inferred for the utterance. In some cases, the intents for a skillbot represent tasks that the skillbot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot trained for banking, the specified intents for the skillbot might be "CheckBalance," "TransferMoney," "DepositCheck," etc. It may include.

[0054] For each intent defined for a skillbot, the skillbot designer may also provide one or more example utterances that are representative of and demonstrate that intent. These example utterances are meant to represent utterances a user may input to the skillbot for that intent. For example, for a balance inquiry intent, example utterances might include, "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account?", etc. Thus, various permutations of typical user utterances may be specified as example utterances for an intent.

[0055] The intents and their associated example utterances are used as training data to train the skill bot. A variety of different training techniques may be used. As a result of this training, a predictive model is generated that is configured to take an utterance as input and output an intent inferred for the utterance by the predictive model. In some cases, the input utterance is provided to an intent analysis engine that is configured to predict or infer an intent for the input utterance using the trained model. The skill bot may then take one or more actions based on the inferred intent.

[0056] (3) Configuring One or More Entities for One or More Intents - In some instances, additional context may be required to enable the skill bot to respond appropriately to a user utterance. For example, there may be situations where user input utterances resolve to the same intent in the skill bot. For example, in the example above, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same balance check. Although each utterance resolves to the same intent, these utterances are different requests that want different things. To clarify such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "saving" could be used to resolve the skill. This may enable the robot to parse the user request and respond appropriately. In the example above, the utterance resolves to the same intent, but the value associated with the AccountType entity is different. , which is different for the two utterances. This allows the skill bot to determine whether the two utterances have the same intent. Even though the two utterances resolve to the same entity, they may perform different actions in some cases. One or more entities may be specified for a particular intent configured for a skill bot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and enable the skill bot to complete the user request.

[0057] In one embodiment, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that may be specified by a skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, circulation period, currency, phone number, URL, etc. Custom entities are used for more customized uses. For example, for a banking skill, the AccountType entity may be specified. Entities may be defined by the skillbot designer to enable various banking transactions by checking user input for keywords such as checking, savings, and credit card.

[0058] (4) Train the Skillbot—The skillbot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select an intent associated with the received user input. As described above, the skillbot must be trained for this. In one embodiment, the skillbot is trained based on the intents configured for the skillbot and example utterances associated with those intents (collectively, training data), thereby enabling the skillbot to resolve user input utterances to one of the skillbot's configured intents. In particular embodiments, the skillbot is trained using the training data and uses a predictive model that enables the skillbot to identify what the user is saying (or, in some cases, what they are trying to say). DABP 102 provides a variety of different training techniques that can be used by the skillbot designer to train the skillbot, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In one embodiment, a portion (e.g., 80%) of the training data is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes called a trained skill bot) can then be used to process and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and no further conversation. To address such situations, a Q&A (Question and Answer) intent may be defined for the skill bot. This allows the skill bot to output a response to a user request without having to update the dialog definition. A Q&A intent is created similarly to a regular intent. The dialog flow for a Q&A intent may differ from the dialog flow for a regular intent.

[0059] (5) Create a dialog flow for the skill bot—The dialog flow specified for the skill bot describes how the skill bot reacts when different intents for the skill bot are resolved in response to received user input. The dialog flow defines the behavior or actions that the skill bot takes, for example, how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. The dialog flow is like a flowchart that the skill bot follows. The skill bot designer specifies the dialog flow using a language such as Markdown language. In one embodiment, OBotML Dialog flow for skill bots can be specified using a version of YAML called Dialog Flow Definition. The dialog flow definition for a skill bot acts as a model of the conversation itself, allowing skill bot designers to choreograph the interactions between the skill bot and the users it serves.

[0060] In one embodiment, a skill bot's dialog flow definition includes three sections: (a) Context Section (b) Default transition section (c) State section.

[0061] Context Section - In the context section, the skill bot designer can define variables used in the conversation flow. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to recognize and persist user preferences, etc.

[0062] Default Transition Section - Transitions for a skill bot can be defined in the dialog flow state section or the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there is no applicable transition defined in a state or when the conditions required to trigger a state transition cannot be met. The default transition section can be used to define routing that allows the skill bot to gracefully handle unexpected user actions.

[0063] State Section - A dialog flow and its associated behavior are defined as a series of temporary states that govern the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality needed at that point in the dialog. In this way, you build states around components. States contain component-specific characteristics and define transitions to other states that are triggered after the component executes.

[0064] Special case scenarios can be handled using the state section. For example, you might want to give a user the option to temporarily exit a first skill they're working on and do something in a second skill within the digital assistant. For example, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), the user might want to jump to a banking skill (e.g., the user might want to verify that they have enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in a first skill can be configured to initiate an interaction with a second, different skill in the same digital assistant and then return to the original flow.

[0065] (6) Adding Custom Components to a Skillbot - As described above, a state specified in a dialog flow for a skillbot nominates a component that provides the required functionality corresponding to that state. The component enables the skillbot to perform the function. In one embodiment, DABP 102 provides a set of pre-configured components to perform a wide range of functions. A skillbot designer can select one or more of these pre-configured components and associate them with states in the dialog flow for the skillbot. A skillbot designer can also add custom or new components using the tools provided by DABP 102. You can create a custom component and associate it with one or more states in the dialog flow for your skill bot.

[0066] (7) Testing and Deploying Skillbots - DABP 102 provides several features that allow skillbot designers to test the skillbots they are developing. The skillbots can then be deployed and included in a digital assistant.

[0067] While the above description describes how to create a skill bot, similar techniques can also be used to create a digital assistant (or parent bot). At the parent bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the parent bot) can handle without invoking a skill bot associated with the digital assistant. Examples of system intents defined for a parent bot include: (1) Exit, which applies when a user signals to the digital assistant that they want to end the current conversation or context; (2) Help, which applies when a user asks for help or direction; and (3) Unresolved Intent, which applies to user input that does not match the Exit and Help intents. The digital assistant also stores information about one or more skill bots associated with the digital assistant. This information allows the parent bot to select a specific skill bot to process an utterance.

[0068] At the parent bot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to process the utterance and associated conversation to determine how to route the utterance. The digital assistant makes this determination using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance should be routed to a specific skill for processing, handled by the digital assistant or parent bot itself according to built-in system intents, or handled as a different state in the current conversation flow.

[0069] In certain embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skill bot corresponding to the invocation name. In such a scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In the absence of a specific or explicit invocation, in some embodiments, the digital assistant evaluates the received user input utterance and calculates confidence scores for system intents and skill bots associated with the digital assistant. The calculated scores for the skill bots or system intents represent the likelihood that the user input represents a task that the skill bot is configured to perform or represents a system intent. System intents or skill bots whose associated calculated confidence scores exceed a threshold (e.g., a Confidence Threshold routing parameter) are selected as candidates for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In certain embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill), and a confidence score is determined for each intent. Generally, intents with a confidence score above a threshold (e.g., 70%) are treated as candidate intents. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the parent bot itself according to the selected system intent.

[0070] FIG. 2 is a simplified block diagram of a parent bot (MB) system 200, according to one embodiment. The MB system 200 can be implemented solely in software, solely in hardware, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multiple-intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill bot invocation unit 240, and a data store 250. The MB system 200 shown in FIG. 2 is merely an example of an arrangement of components in a parent bot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have subsystems in a different configuration or arrangement.

[0071] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As described above, the utterance may be provided in a variety of ways, including as audio or text. The utterance 202 may be a fragment, a complete sentence, multiple sentences, etc. The utterance 202 may include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the audio to text using a speech-to-text converter (not shown), which inserts punctuation, e.g., commas, semicolons, periods, etc., into the resulting text.

[0072] The language detection unit 212 detects the language of the utterance 202 based on the text of the utterance 202. Because each language has its own grammar and semantics, the way in which the utterance 202 is processed depends on the language. Language differences are taken into account when analyzing the syntax and structure of the utterance.

[0073] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and lemmatize the words. A lemma is the primary form of a set of words represented in a dictionary (e.g., "run" is the lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking of complex expressions, for example, combining "credit" and "card" into a single expression "credit_card". The language parser 214 may also identify relationships between words in utterance 202. For example, in some embodiments, language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., particular nouns) are direct objects, which parts of the utterance are prepositions, etc. The results of the processing performed by language parser 214 form extracted information 205, which, along with utterance 202 itself, are provided as inputs to MIS 220.

[0074] As described above, utterance 202 may contain multiple sentences. For purposes of multiple intent and explicit invocation detection, utterance 202 may be treated as a single unit even if it contains multiple sentences. However, in some embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210, to identify single sentences within multiple sentences for multiple intent analysis and explicit invocation analysis. In general, the results generated by MIS 220 and EIS 230 are consistent regardless of whether utterance 202 is processed at the individual sentence level or as a single unit containing multiple sentences. are essentially the same.

[0075] The MIS 220 determines whether the utterance 202 expresses multiple intents. While the MIS 220 can detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not involve determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the processing for determining whether the intent of the utterance 202 matches a bot intent may be performed by the intent classifier 242 of the MB system 200 (e.g., as shown in the embodiment of FIG. 3) or by an intent classifier of the skill bot. The processing performed by the MIS 220 assumes that a bot (e.g., a particular skill bot or the parent bot itself) exists that can process the utterance 202. Thus, the processing performed by the MIS 220 does not require knowledge of what bots are in the chatbot system (e.g., the identities of the skill bots registered with the parent bot) or what intents are configured for a particular bot.

[0076] To determine that utterance 202 contains multiple intents, MIS 220 applies one or more rules from a set of rules 252 in data store 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern may include a conjunction connecting two parts of a sentence (e.g., coordinates), both of which correspond to separate intents. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents directed to different bots or different intents within the same bot). Instead, an utterance may have separate instances of the same intent, such as "order a pizza using payment account X, then order a pizza using payment account Y."

[0077] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines what portions of utterance 202 are associated with each intent. For each intent expressed in the multiple-intent utterance, MIS 220 constructs a new utterance for separate processing in place of the original utterance, e.g., utterance “B” 206 and utterance “C” 208, as shown in FIG. 2 . Thus, original utterance 202 may be split into two or more separate utterances that are handled one at a time. MIS 220 determines which of the two or more utterances should be processed first using extracted information 205 and / or from an analysis of utterance 202 itself. For example, MIS 220 may determine that utterance 202 includes a marker word indicating that a particular intent should be handled first. The newly formed utterance corresponding to this particular intent (e.g., utterance 206 or one of utterances 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (or been temporarily interrupted), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) may then be sent to EIS 230 for processing.

[0078] The EIS 230 determines whether the received utterance (e.g., utterance 206 or utterance 208) includes a call name for the skillbot. In one embodiment, each skillbot in the chatbot system is assigned a unique call name that distinguishes the skillbot from other skillbots in the chatbot system. A list of call names can be maintained as part of the skillbot information 254 in the data store 250. When the utterance includes words that match the call name, the utterance is considered to be an explicit call. If the bot is not explicitly called, the utterance received by the EIS 230 is considered an implicit call utterance 234 and is passed to the parent bot's intent classifier (e.g., intent classifier 242). The intent classifier 242 determines which bot to use to process the utterance. In some examples, the intent classifier 242 determines that an utterance implicitly invoked by a parent bot should be processed. In other examples, the intent classifier 242 determines which skill bot to route the utterance to for processing.

[0079] The explicit invocation feature provided by EIS 230 has several advantages. It can reduce the amount of processing that a parent bot must perform. For example, when there is an explicit invocation, the parent bot may not have to perform any intent classification analysis (e.g., using intent classifier 242) or may have to perform reduced intent classification analysis to select a skill bot. Thus, explicit invocation analysis may enable the selection of a specific skill bot without relying on intent classification analysis.

[0080] There may also be situations where there is overlap in functionality among multiple skill bots. This can occur, for example, when the intents handled by two skill bots overlap or are very close to each other. In such situations, it may be difficult for a parent bot to identify which of multiple skill bots to select based solely on intent classification analysis. In such scenarios, explicit invocation disambiguates the specific skill bot to be used.

[0081] In addition to determining that an utterance is an explicit invocation, EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the explicitly invoked skillbot. In particular, EIS 230 may determine whether any portion of the utterance is not associated with an invocation. EIS 230 may make this determination through analysis of the utterance and / or analysis of extracted information 205. Instead of sending the entire utterance received by EIS 230, EIS 230 may send the portion of the utterance that is not associated with an invocation to the invoked skillbot. In some examples, the input to the invoked skillbot is formed simply by removing any portion of the utterance that is associated with an invocation. For example, "I would like to order a pizza using Pizza Bot" becomes can be shortened to "I want to order a pizza" because "Using Pizza Bot" is related to the invocation of PizzaBot, but not to any processing performed by PizzaBot. In some instances, EIS 230 may reformat the portion to be sent to the invoked bot, for example to form a complete sentence. Thus, EIS 230 determines not only that there is an explicit invocation, but also what to send to the skill bot when there is an explicit invocation. In some instances, there may be no text to input to the invoked bot. For example, if the utterance was "PizzaBot", In this case, the EIS 230 may determine that the PizzaBot is being invoked, but that there is no text to be processed by the PizzaBot. In such a scenario, the EIS 230 may indicate to the SkillBot invoker 240 that there is nothing to send.

[0082] The skillbot invoker 240 invokes a skillbot in various manners. For example, the skillbot invoker 240 can invoke the bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invoke. The indication 235 can be sent by the EIS 230 along with input for the explicitly invoked skillbot. In this scenario, the skillbot invoker 240 hands over control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response can be to perform a particular action or to start a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0083] Another manner in which the skillbot invoker 240 can invoke a skillbot is through implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 is trained with different classes, one class for each skillbot. For example, each time a new skillbot is registered with a parent bot, a list of example utterances associated with the new skillbot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. Parameters generated as a result of this training (e.g., a set of values ​​for parameters of a machine learning model) can be stored as part of the skillbot information 254.

[0084] In one embodiment, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training the machine learning model may include inputting at least a subset of utterances from example utterances associated with various skill bots to generate, as an output of the machine learning model, an inference about which bot is the correct bot to process any particular training utterance. For each training utterance, an indication of the correct bot to use for that training utterance may be provided as ground truth information. The behavior of the machine learning model may then be adapted (e.g., through backpropagation) to minimize the discrepancy between the generated inference and the ground truth information.

[0085] In particular embodiments, the intent classifier 242 determines a confidence score for each skill bot registered with the parent bot, indicating the likelihood that the skill bot can process an utterance (e.g., an implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skill bot invoker 240 will invoke the bot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either the identification of a system intent or the identification of a particular skill bot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next highest confidence score by a particular winning margin. Imposing such a condition enables routing to a particular skill bot when the confidence scores of multiple skill bots each exceed the threshold confidence score value.

[0086] After identifying the bot based on the evaluation of the confidence score, the skillbot invoker 240 hands over processing to the identified bot. In the case of a system intent, the identified bot is a parent bot. Otherwise, the identified bot is a skill bot. Furthermore, the skillbot invoker 240 determines what to provide as input 247 to the identified bot. As described above, in the case of an explicit invoke, the input 247 can be based on a portion of the utterance not associated with the invoke, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invoke, the input 247 can be the entire utterance.

[0087] Data store 250 comprises one or more computing devices that store data used by various subsystems of parent bot system 200. As described above, data store 250 includes rules 252 and skill bot information 254. Rules 252 include, for example, rules for determining, by MIS 220, when an utterance expresses multiple intents and how to split an utterance that expresses multiple intents. Rules 252 further include rules for determining, by EIS 230, which portions of an utterance that explicitly invokes a skillbot should be sent to the skillbot. Skillbot information 254 includes the call names of skillbots in the chatbot system, e.g., a list of the call names of all skillbots registered to a particular parent bot. Skillbot information 254 may also include information used by intent classifier 242 to determine a confidence score for each skillbot in the chatbot system, e.g., parameters of a machine learning model.

[0088] 3 is a simplified block diagram of a Skillbot system 300 according to certain embodiments. Skillbot system 300 is a computing system that may be implemented solely in software, solely in hardware, or a combination of hardware and software. In some embodiments, such as the embodiment shown in FIG. 1, Skillbot system 300 can be used to implement one or more Skillbots within a digital assistant.

[0089] Skillbot system 300 includes MIS 310, intent classifier 320, and conversation manager 330. MIS 310 is similar to MIS 220 of FIG. 2 and provides similar functionality, including being operable to determine (1) whether an utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents using rules 352 in data store 350. In one embodiment, the rules applied by MIS 310 to detect multiple intents and split the utterance are the same as the rules applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. Extracted information 304 is similar to extracted information 205 of FIG. 1 and can be generated using language parser 214 or a language parser local to skillbot system 300.

[0090] The intent classifier 320 may be trained in a manner similar to the intent classifier 242 discussed above in connection with the embodiment of FIG. 4, and as described in more detail herein. For example, in particular embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of example utterances associated with that particular skill bot as training utterances. The ground truth for each training utterance will be the particular bot intent associated with that training utterance.

[0091] The utterance 302 may be received directly from a user or may be provided through a parent bot. When the utterance 302 is provided through a parent bot, for example, as a result of processing through the MIS 220 and the EIS 230 in the embodiment shown in FIG. 4, the MIS 310 may be bypassed to avoid repeating processing already performed by the MIS 220. However, if the utterance 302 is received directly from a user, for example, during a conversation that occurs after routing to a skill bot, the MIS 310 may process the utterance 302 to determine whether the utterance 302 represents multiple intents. If the utterance 302 represents multiple intents, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, for example, utterance “D” 306 and utterance “E” 308. If the utterance 302 does not represent multiple intents, the MIS 310 forwards the utterance 302, without segmentation, to the intent classifier 320 for intent classification.

[0092] The intent classifier 320 is configured to match a received utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As explained above, a skillbot may be configured with one or more intents. Each intent includes at least one example utterance associated with that intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the parent bot system 200 is trained to determine a confidence score for each skill bot and a confidence score for the system intent. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skill bot system 300. While the classification performed by the intent classifier 242 is at the bot level, the classification performed by the intent classifier 320 is at the intent level and therefore has finer granularity. The intent classifier 320 has access to intent information 354. For each intent associated with the skill bot system 300, the intent information 354 includes a list of utterances that represent the meaning of the intent and are typically associated with tasks that can be performed by the intent. The intent information 354 may further include parameters generated as a result of training on this list of utterances.

[0093] The conversation manager 330 receives as output from the intent classifier 320 an indication 322 of the particular intent identified by the intent classifier 320 as the best match for the utterance input to the intent classifier 320. In some instances, the intent classifier 320 is unable to determine any match. For example, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value if the utterance is directed to a system intent or to the intent of a different skill bot. When this occurs, the skill bot system 300 may refer the utterance to a parent bot for processing, e.g., routing to a different skill bot. However, if the intent classifier 320 successfully identifies the intent within the skill bot, the conversation manager 330 begins a conversation with the user.

[0094] The conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine may include a default starting state (e.g., for when the intent is invoked without any additional input) and one or more additional states, each having associated therewith an action to be performed by the skill bot (e.g., perform a purchase transaction) and / or a dialog to be presented to the user (e.g., question, response). Thus, the conversation manager 330 may determine an action / dialog 335 upon receiving an instruction 322 identifying an intent, and may determine the additional action or dialog in response to subsequent utterances received during the conversation.

[0095] Data store 350 comprises one or more computing devices that store data used by various subsystems of skillbot system 300. As shown in Figure 3, data store 350 includes rules 352 and intent information 354. In certain embodiments, data store 350 can be integrated with a parent bot or digital assistant data store, such as data store 250 of Figure 2.

[0096] Exemplary Data Processing System 4 is a simplified block diagram of a computing system that implements a language processing system. Language processing system 400 may be any system, device, hardware, software, computer-readable medium, or other entity that performs or assists in the performance of the methods described herein. Language processing system 400 includes a pre-processing subsystem 412. The pre-processing system may be any system capable of capturing input that may be relevant to the procedures of language processing system 400. For example, pre-processing subsystem 412 may include: , may be programmed to take utterances, such as utterance A402A, and determine how to process and route the utterance.

[0097] In one embodiment, the utterance A402A is a user query from a user interacting or attempting to interact with the chatbot system. In another embodiment, the utterance A402A is training data for training a machine learning model. The pre-processing subsystem 412 includes a language detector 414 that enables the pre-processing system 412 to detect the language input thereto. The pre-processing subsystem 412 includes a training / query manager 416. The training / query manager 416 can detect whether the utterance A402A is a training set of data or a query from a human client and determine how the pre-processing subsystem 412 should route the utterance. For example, if the utterance is training data, the pre-processing subsystem 412 routes the training data to the training subsystem to retrain the machine learning model. If the utterance is a natural language query from a client seeking a solution to it, the pre-processing subsystem 412 routes the utterance in a manner that receives features of the utterance as input so that the machine learning model generates predicted data output. In one embodiment, the preprocessing subsystem 412 will convert data from utterance A 402A into language or data that is more easily parsed by the language analysis subsystem, such as utterance B 402B. For example, the preprocessing subsystem 412 may include a natural language preprocessor system that converts natural language queries into one or more features that will be input into a machine learning model. Such features may include contextual features determined based on lexical information generated based on natural language phrases and expressive features generated based on gazetteer or other expressive phrase lists.

[0098] Language processing system 400 further includes a feature balancing subsystem 420. In various embodiments, feature balancing subsystem 420 is an entity within language processing system 400 that can process, train, or otherwise use natural language data to perform the methods described herein. In various embodiments, feature balancing subsystem 420 includes one or more subsystems that include instructions for performing multi-feature balancing for a natural language processor.

[0099] The feature balancing subsystem 420 includes a coverage balancing subsystem 422. The coverage balancing subsystem 422 is a subsystem configured and implemented to perform a coverage balancing method such as that described herein. Examples of coverage balancing methods are discussed further below with reference to Figures 5 and 6. The coverage balancing subsystem 422 includes coverage balancing instructions 423, which include steps for performing the coverage balancing steps described herein, as well as values, ranges, or any other type of parameters that affect how the coverage balancing subsystem 422 operates.

[0100] Feature balancing subsystem 420 includes dropout balancing subsystem 424, which is a subsystem configured and implemented to perform a dropout-based balancing method such as that described herein. Examples of dropout-based balancing methods are discussed further below with reference to Figures 7 and 8. Dropout balancing subsystem 424 includes dropout balancing instructions 425, which include steps for performing the dropout balancing steps described herein, as well as values, ranges, or any other type of parameters that affect how dropout balancing subsystem 424 operates.

[0101] The feature balancing subsystem 420 includes a noise balancing subsystem 426. The noise balancing subsystem 426 implements a noise-based balancing method, such as that described herein. 426 is a subsystem configured and implemented to utilize noise balancing instructions 427 to perform a noise-based balancing method. An example of a noise-based balancing method is discussed further below with reference to FIGURE 9. Noise balancing subsystem 426 includes noise balancing instructions 427 that include steps for performing the noise-based balancing steps described herein, as well as values, ranges, or any other type of parameters that affect how noise balancing subsystem 426 operates.

[0102] Feature balancing subsystem 420 includes model combination subsystem 428. Model combination subsystem 428 is a subsystem configured and implemented to combine one or more models for multi-feature balancing according to methods such as those described herein. Model combination subsystem 428 includes combination instructions 429, which may include instructions, formulas, transforms, or any other type of combination criteria that determine how model combination subsystem 428 is implemented. For example, model combination subsystem 428 may implement a combination of balancing instructions from subsystems 422, 424, and / or 426 as part of a process for multi-feature balancing for natural language processing.

[0103] Language processing system 400 includes a data store 430. Data store 430 may be any type of storage, memory, repository, or other entity capable of storing data and instructions for performing the methods described herein. In some embodiments, data store 430 stores multiple models for natural language processing that may be used for chatbot responses or training. Data store 430 includes a machine learning model catalog 432, which is a catalog of machine learning models that may be used to process natural language queries. Data store 430 also includes a gazetteer catalog 434, which is a catalog of lists of natural language phrases (i.e., gazetteers) that may be implemented by any number of machine learning models to generate expressive features as part of natural language processing.

[0104] The language processing system 400 includes a skillbot selector 440. In various embodiments, the skillbot selector 440 is a subsystem that determines the skill or chatbot that is best suited to respond to a query or that would be selected from an input training dataset. For example, the skillbot selector 440 incorporates data output by a machine learning model, such as predicted labels, to determine a skill for processing and / or responding to a natural language query. The skillbot selector 440 includes skillbot selection instructions 442, which are instructions for selecting a skill, chatbot, skillbot, or any other type of matching entity based on the input data. The skillbot selector 440 further includes a repository 444, which is a store of skillbots or representations of skillbots that can be selected by the skillbot selector 440.

[0105] Scope of application language training As explained above, machine learning models may use both contextual features and regular expression features during natural language processing. The contextual features input to the machine learning model are generated by a trained machine learning model trained on a particular training dataset. The expressive features input to the machine learning model are generated, at least in part, by a gazetteer list of natural language phrases corresponding to a label, e.g., a list of "names." The machine learning model uses both contextual and expressive features as inputs to trigger the generation of output predictions. However, not all gazetteers are sufficient for use with a particular machine learning model. For example, a machine learning model may be trained using a training dataset containing various medical-based commands and responses. A corresponding gazetteer that may be used to generate expressive features may be associated with the label "Locations." If the training dataset is In situations that do not include many gold labels associated with locations, the model may rely too heavily on gazetteer-based expressive features. For example, given the utterance "My arm is hurt," the gazetteer may include the location "Hurt, Virginia." The corresponding expressive features generated by the gazetteer may cause the model to incorrectly predict that the utterance corresponds to a request for information about "Hurt, Virginia" instead of a request for medical attention.

[0106] A method for multi-factor balancing includes determining and automatically utilizing a gazetteer with a machine learning model based on a coverage metric associated with natural language words in the gazetteer and a training dataset used to train the model. The more words included in the gazetteer that also match corresponding words and labels in the training dataset used to train the machine learning model, the more likely the input of expressive and contextual features to the model is balanced. For example, utilizing a gazetteer that includes many words not in the training dataset may cause the model to overly rely on expressive features from the gazetteer because the model does not generate strong contextual features corresponding to the same words. To remedy this shortcoming, a coverage metric is determined based on the gazetteer and the training dataset, and the gazetteer is utilized or not utilized based on the coverage metric. The coverage metric can also be used to automatically modify the gazetteer or the training dataset to provide an appropriate level of coverage for utilizing the gazetteer with the corresponding trained model.

[0107] FIG. 5 illustrates a process flow for managing a dataset of natural language phrases and a training dataset for a machine learning model, according to various embodiments. The process illustrated in FIG. 5 is implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software is stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 5 and described below is intended to be exemplary and non-limiting. While FIG. 5 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel. In various embodiments, the steps detailed in process 500 are performed by the systems and models discussed herein with respect to FIGS. 1-5.

[0108] Process 500 begins at 510, where a desired coverage value for a dataset of natural language phrases and a training dataset for a machine learning model is received. The dataset of natural language phrases is, for example, a gazetteer of natural language phrases used to generate expressive features for input to the machine learning model. The desired coverage value relates to the relative "coverage" for the gazetteer and the training dataset. For example, the coverage value may represent a certain percentage of natural language phrases in the gazetteer that match corresponding natural language phrases in the training data and whose corresponding labels (e.g., "Name") in the gazetteer match gold labels in the training data. For example, in a training dataset used to train a machine learning model, there may be 1,000 gold labels corresponding to the label "Name." A corresponding gazetteer containing a list of natural language phrases may correspond to "Names," and the gazetteer may be used to generate expressive features for input to the machine learning model. Of the 1000 phrases corresponding to the 1000 gold labels in the gazetteer data, 600 of those natural language phrases may be included. Therefore, the coverage of the gazetteer for the training data may be 60% (600 / 1000). Gazetteer's greater "coverage" of training data means that the expressive features generated using Gazetteer more closely resemble the contextual features generated by contextual machine learning models, causing the model to not overfit its predictions to the expressive features it generates.

[0109] The desired coverage value received at 510 relates to an optimal or minimum coverage value at which the model can trust the gazetteer. This value may be received, for example, by a data scientist implementing the machine learning model as part of the chatbot features or determined during hyperparameter tuning of the machine learning model. For example, a desired coverage value of 80% may indicate that the contextual machine learning model should not utilize gazetteers that do not have coverage of at least 80% of the training dataset.

[0110] At 520, the dataset of natural language phrases is compared to a training dataset for the machine learning model to determine the number of corresponding natural language phrases. As described above, gazetteers are compared to the training dataset. In various embodiments, this step involves analyzing the training dataset to determine the number of gold labels in the training dataset that correspond to a certain type of gazetteer (e.g., to determine the actual coverage value of a gazetteer for a particular training dataset, the gazetteer's label may first be determined to be "Locations" and then the number of gold labels in the training dataset may be determined to be "Locations"). The Gold label in the GIS dataset is all Gold locations with the label "Locations". (The phrases are analyzed to determine the gold label.) Once a gold label is matched to a gazetteer, the corresponding natural language phrase in the training dataset is compared to the gazetteer to determine the percentage of "overlap" between the gazetteer and the training dataset. In some embodiments, phrases are considered to overlap only if they are an exact character-for-character match. In other embodiments, a sub-phrase in the training dataset will be considered to match a larger natural language phrase in the gazetteer if the phrase in the gazetteer contains the sub-phrase as a subset of another phrase.

[0111] At 530, an actual coverage value is determined based on a comparison of the list of natural language phrases with the training dataset. For example, as described above, the actual coverage value may be the proportion of phrases in the training dataset that match the gazetteer in both label and phrase. For example, if the gazetteer matches 58 phrases in the training dataset out of 100 possible matching labels in both phrase and label, then the coverage is 58% (58 / 100).

[0112] At 540, a determination is made as to whether the actual coverage value determined at 530 is greater than or equal to the desired coverage value received at 510. Specifically, the comparison determines whether the actual coverage value is greater than or equal to the desired coverage value, meaning that the gazetteer sufficiently covers the training data up to the tolerance level specified by the desired coverage value. For example, given a desired coverage value of 80% specified by a data scientist, only gazetteers with at least 80% coverage are used in conjunction with a machine learning model to process natural language queries. A gazetteer with an actual coverage value of 94% is acceptable, but a gazetteer with 44% coverage is not acceptable. In some embodiments, when a gazetteer's actual coverage value is less than the desired coverage value, another gazetteer is selected to determine its coverage value with the training dataset.

[0113] At 550, if the actual coverage value is not greater than or equal to the desired coverage value, the actual coverage value is The gazetteer and / or the training dataset may be modified so that the coverage value meets or exceeds the desired coverage value. For example, at 550, one or more natural language phrases are selected from the training dataset that are not present in the dataset of natural language phrases (i.e., the gazetteer). The selected natural language phrases are missing phrases that, if included in the gazetteer, would increase the gazetteer's actual coverage value when compared to the training dataset.

[0114] In some embodiments, instead of or in addition to selecting one or more natural language phrases from the training dataset, some natural language phrases are selected from the gazetteer. Selected phrases from the gazetteer that correspond to phrases not included in the training dataset would increase the actual coverage rate if they were included in the training dataset. An example of this is provided below with reference to FIG. 6.

[0115] At 560, one or more selected natural language phrases selected at 550 are added to a dataset of natural language phrases to increase actual coverage. Specifically, natural language phrases from the training dataset that are not in the gazetteer but have a corresponding gold label for the gazetteer label are added to the gazetteer. In some embodiments, selected portions of phrases from the training dataset are added to the gazetteer instead of the entire natural language phrase selected from the training dataset. For example, one set of training phrases from the training dataset may be "I would like to go to Sydney Opera House." The corresponding gold label may include the phrase "Sydney Opera House." Instead of adding the entire natural language phrase above to the location gazetteer, only the phrase "Sydney Opera House" is selected from the training dataset and added to the gazetteer. In various embodiments, the one or more selected natural language phrases are sufficient such that, when added to the gazetteer, the newly determined actual coverage rate meets or exceeds the desired coverage rate.

[0116] As described above, in addition to or instead of 550, one or more natural language phrases are selected from the gazetteer for inclusion in the training dataset. For example, to increase practical coverage, phrases in the gazetteer that are not included in the training dataset may be added to the training dataset. For example, a gazetteer of names may include the phrase “Mortimer,” which corresponds to the English name. The training dataset may not include the name Mortimer in association with at least the gold label “Name.” The phrase “Mortimer” may then be selected from the gazetteer and added to the training dataset along with the gold label “Name,” and the newly modified training dataset will be used to retrain the contextual machine learning model. In various embodiments, phrases from the gazetteer may be modified before inclusion in the training dataset. For example, a stock symbol-based utterance generation may convert the phrase “Mortimer” into the complete utterance “Hello, my name is Mortimer” for inclusion in the training dataset.

[0117] At 570, once the one or more selected natural language phrases are added to the dataset of natural language phrases at 560, or if the actual coverage value is already equal to or greater than the desired coverage value as determined at 540, the natural language query is processed using the machine learning model that includes the dataset of natural language phrases. For example, at this point, the actual coverage value is equal to or greater than the desired coverage value, and the gazetteer is configured to generate expressive features of the machine learning model without overriding the contextual features generated by the machine learning model. Gazetteer can be "trusted." Thus, Gazetteer is now used alongside machine learning models to process natural language queries. In various embodiments, the natural language queries processed are natural language queries contained in utterances submitted by clients for resolution by the chatbot.

[0118] 6 is a diagram illustrating an example dataset of natural language phrases and a training dataset utilized as part of a multi-factor model for natural language processing, according to various embodiments. Specifically, FIG. 6 illustrates Gazetteer and a training dataset that may be modified to increase the actual coverage value between both datasets, allowing Gazetteer to be used in conjunction with a machine learning model trained using the training dataset.

[0119] 6 shows a gazetteer 600. The gazetteer 600 includes gazetteer attributes 602. The gazetteer attributes 602 relate to labels for all phrases included in the gazetteer, e.g., "Location," "Person," etc. Each phrase in the gazetteer 600 is associated with a label in the gazetteer attribute 602. The gazetteer 600 includes a gazetteer phrase list 604 that includes several phrases 606(1) to 606(N). Each of the phrases 606 corresponds to a gazetteer attribute 602 (e.g., a city name corresponding to the gazetteer attribute of "Location").

[0120] 6 illustrates a training dataset 610. The training dataset 610 includes sets of training data, each of which includes a training phrase 612 and a gold label 614. As shown, the training dataset 610 includes several sets of training data, ranging from a training phrase 612(1) and a corresponding gold label 614(1) to a training phrase 612(N) and a corresponding gold label 614(N).

[0121] As described above with reference to FIG. 5 , when modifying either or both gazetteer 600 and training dataset 610 to increase the actual coverage value associated with the dataset, phrases from training dataset 610 may be added to gazetteer 600, or vice versa. For example, as shown in FIG. 6 , training phrases 612(1) and 612(2) are added to gazetteer phrase list 604 as phrases N+1 and N+2. For example, it may be determined that gold labels 614(1) and 614(2) have a label type corresponding to gazetteer attribute 602, meaning that each of phrases 612(1) and 612(2) will be accepted into gazetteer phrase list 604. Furthermore, it may be determined that gazetteer phrase list 604 does not include phrases 612(1) and 612(2). Thus, training phrases may be added to gazetteer to increase gazetteer 600's actual coverage with the training dataset.

[0122] As shown in FIG. 6 , phrases from Gazetteer 600 may be added to training dataset 610. For example, phrase 606(1) in Gazetteer phrase list 604 is determined not to be included in training dataset 610. Phrase 606(1) is added to the training dataset as training phrase N+1. The corresponding gold label is copied from Gazetteer attributes 602 to form a complete training data set. The training dataset can be used to retrain machine learning models for use with Gazetteer 600 according to an acceptable coverage rate.

[0123] Model Feature Dropout As discussed above, the coverage ratio can be used to determine how a gazetteer and a contextually trained machine learning model work together to generate expressive and contextual features for input to the machine learning model. Coverage balancing is an effective tool for determining the appropriate combination of a gazetteer and a machine learning model to process natural language phrases using the features of the machine learning model.

[0124] Contextual features may be generated by feeding the utterance into a pre-trained language model, such as a multilingual sentence encoder, BERT, etc. Expressive features may be generated by gazetteer matching using various techniques, including but not limited to: Extracting expressive or gazetteer features for each token of the input utterance using a neural classifier trained solely on gazetteer, which classifies subsequences of tokens in the input utterance as belonging to a certain entity class with a certain degree of confidence; The output of the classifier is then used as a feature to be integrated into a natural language model, such as an existing BiLSTM-CRF architecture; Training a separate gazetteer classifier with the augmented gazetteer data, and integrating this classifier into a natural language model, such as an existing BiLSTM-CRF architecture; Matching and encoding expressive or gazetteer features via a self-attention mechanism, then concatenating with other features (e.g., Glove, ELMO, etc.). Expressive or gazetteer features are matched and encoded as additional representations, which are then added to an existing natural language model, such as an existing BiLSTM-CRF architecture. Finally, gazetteers may be used to perform data augmentation on existing training data, which may be used to fine-tune or retrain an existing natural language model, such as an existing BiLSTM-CRF architecture.

[0125] Once features are obtained for input to a machine learning model, the features are input and a predicted output is generated from the model. For example, in embodiments where the machine learning model is an ANN, the generated features are mapped to input nodes of a neural network. The values ​​generated at the input nodes are processed by several hidden layers, and then the output layer of the nodes is used to generate an output, e.g., a prediction of a skill label used to respond to a natural language query. An example ANN machine learning model is described below with reference to FIG. 8.

[0126] The generated features can be further refined to increase or decrease the weight of specific terms input to the machine learning model. For example, expressive features generated using a gazetteer may be more accurate than contextual features generated by processing natural language queries with a contextual model, and therefore the expressive features should be given more weight when input to the machine learning model. This can be achieved with lexical dropout or contextual dropout to reduce the weight or number of contextual features input to the machine learning model. Percentage-based contextual dropout randomly removes some contextual features from the input to the machine learning model. For example, a 20% contextual dropout rate would randomly remove up to one in five contextual features from the pool of features input to the machine learning model. However, the general contextual dropout approach can be improved using targeted contextual dropout, which drops out specific contextual features that are already well represented by the corresponding expressive features.

[0127] For example, given the utterance "I would like to visit the Eiffel Tower," contextual and expressive features are generated for the subphrase "Eiffel Tower." The expressive features generated from the location word gazetteer may be more important than the contextual features generated from the same phrase. Because the Eiffel Tower is a fairly constant location, gazetteer can be relied upon with utmost certainty to generate accurate expressive features. Therefore, the contextual features associated with the Eiffel Tower Signatures may be dropped out because they will only serve to reduce the weight of expressive features. Such use of targeted expressive dropout will improve the overall accuracy of the model.

[0128] Conversely, gazetteer-based models may tend to overfit their output predictions to expressive features. For example, "Mark these papers, please" Given the utterance "Mark these papers, please," an English name gazetteer might generate an expressive feature for the name "Mark" when the word "Mark" is clearly not used as a name. Therefore, an expressive dropout rate similar to contextual dropout may be utilized to give less weight to confusing expressive features input to a machine learning model. For example, a contextual analysis of the utterance "Mark these papers, please" might generate an expressive feature for the word "Mark" when the word "Mark" is clearly not used as a name. The model would determine that "Mark" is almost certainly a verb. Therefore, targeted expressive dropout of expressive features associated with nouns (e.g., English names) may be appropriate to avoid overfitting predictions with inaccurate expressive features. Therefore, either expressive features or contextual features can be targeted for dropout to improve machine learning model processing of natural language phrases. In various embodiments, the targeted dropout discussed herein may be used in combination with a general random dropout rate for features input to the machine learning model. For example, two percentage rates for dropout may be utilized: a first targeted dropout rate that drops out features at a first rate, and a second general dropout rate that randomly drops out features after the targeted dropout has occurred.

[0129] FIG. 7 illustrates a process flow for performing feature dropout as part of multi-feature balancing for a natural language processor, according to various embodiments. The process illustrated in FIG. 7 is implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or a combination thereof. The software is stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 7 and described below is intended to be exemplary and non-limiting. While FIG. 7 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel. In various embodiments, the steps detailed in process 700 are performed by the systems and models discussed herein with respect to FIGS. 1-6.

[0130] Specifically, process 700 describes a process for targeted contextual dropout of contextual features. It will be appreciated that in addition to or instead of the contextual dropout process described in FIG. 7, a corresponding expressive dropout process may be performed to drop out expressive features. Process 700 begins at 710, where a natural language query to be processed by a machine learning model utilizing a natural language processing model and a dataset of natural language phrases is received. The natural language query may be received, for example, from a client interacting with an automated digital assistant to resolve the query using a chatbot. For example, the received natural language query may be utterance 402A as shown in FIG. 4 and may be further processed according to dropout balancing subsystem 424.

[0131] At 720, in response to receiving the natural language query, a contextual dropout rate corresponding to the machine learning model is determined. The contextual dropout rate may be, for example, a targeted contextual dropout rate for dropping out contextual features before inputting the generated pool of features into the machine learning model. In some embodiments, the contextual dropout rate is pre-set by a data scientist or operator of the machine learning model according to a desired level of contextual dropout. In some embodiments, the contextual dropout rate is determined during hyperparameter tuning of the machine learning model.

[0132] At 730, a set of contextual features and a set of expressive features are determined. The contextual features and expressive features are generated from a natural language query to be input to the machine learning model. Some contextual features correspond to expressive features due to specific features generated based on similar portions of the natural language query, and vice versa. For example, in the utterance "I would like to see the Eiffel Tower," the subphrase "Eiffel Tower" may be used to generate some expressive features and some contextual features. The expressive features, perhaps generated using a gazetteer of locations, may allow the model to make more accurate predictions than the corresponding contextual features.

[0133] At 740, a subset of contextual features corresponding to expressive features for a portion of the natural language query is determined. As described above with reference to 730, a subset of contextual features may correspond to expressive features when both features are generated from the name portion of the natural language query. For example, some contextual features may correspond to expressive features when both sets of features are generated by the subphrase "Eiffel Tower" in the example query above. Contextual features are less likely to affect accurate predictions than expressive features and are therefore prime candidates for targeted contextual dropout.

[0134] At 750, at least a portion of the contextual features from the subset of contextual features determined in 740 are removed from the pool of contextual features at a rate corresponding to a contextual dropout rate to form a set of modified contextual features. For example, assuming a 20% contextual dropout rate determined in 720, one in five features from the subset of features would be removed from the pool of contextual features prior to inputting the features into the machine learning model. Thus, targeted dropout of contextual features from the subset of contextual features corresponding to high accuracy expressive features occurs to prevent those contextual features from inappropriately weighting the output of the model toward inaccurate contextual predictions.

[0135] An optional step of additional general contextual dropout may occur at 760. For example, additional general contextual dropout may occur in addition to the targeted contextual dropout that occurs as part of 740 and 750. The general contextual dropout at 760 includes removing at least a portion of the contextual features from the set of modified contextual features generated at 750 at a rate corresponding to a distinct contextual dropout rate to form a further modified set of contextual features. For example, a random selection of contextual features may be dropped out from the pool of contextual features prior to inputting all features into the machine learning model. This would result in the two-stage dropout described above: first targeted dropout and second general random dropout.

[0136] At 770, the machine learning model processes the modified set of contextual features and the set of expressive features. The machine learning model takes the modified set of contextual features and the set of expressive features as inputs, causing feature processing and generating output predictions based on the processing. For example, each of the features in the modified set of contextual features and the set of expressive features may map to an input node of an ANN in the machine learning model, but a particular input node of the ANN may receive a reduced or null value from the dropped-out contextual features, thus affecting the processing of the value at the input node.

[0137] 8 is a simplified block diagram of a skill classifier artificial neural network machine learning model that utilizes feature dropout, according to one embodiment. Specifically, FIG. 8 illustrates an exemplary embodiment of a skill classifier machine learning model that is configured to utilize an artificial neural network to take as input a number of contextual and expressive features and output a predicted skill for processing a natural language query.

[0138] As shown in Figure 8, skill classifier 800 takes as input several contextual features 810(1)-810(4) and several expressive features 820(1)-820(N). It will be understood that the number of feature inputs of either kind is not limited by the exemplary embodiment shown in Figure 8 and, in fact, may be a very large number of features to accurately reflect the various contextual and expressive features of natural language queries that may be generated. Each of the input features is mapped to an input node 830(A)-830(N) of the ANN.

[0139] As part of the dropout process described herein, one or more features may be dropped from the pool of inputs before the machine learning model skill classifier 800 processes the inputs. For example, as shown in FIG. 8, some contextual features, including contextual features 810(2) and 810(4), are separated from corresponding input nodes 830(B) and 830(D). This illustrates how input features can be dropped out to reduce the weight that certain features would have on the machine learning model processing and increase the weight of other features. For example, by dropping out contextual features 810(2) and 810(4), the corresponding input nodes 830(B) and 830(D) will not generate initial values ​​and will contribute little, if any, node weights to successive hidden layers of node processing in the ANN.

[0140] Input nodes 830(A)-830(N) are mapped to additional layers of the artificial neural network via one or more edges. For example, some edges emanating from input nodes 830(A)-830(N) may lead to additional nodes in hidden layers, each of which may have corresponding node parameter / weight values. For example, as shown in FIG. 8, input nodes 830(A)-830(N) are mapped to hidden layer 840(A) of the artificial neural network. It will be understood that an artificial neural network is a network that includes a one-to-one edge relationship of nodes in each successive layer or any subset of edges therein.

[0141] Values ​​passed from input nodes 830(A)-830(N) are passed sequentially through multiple hidden layers 840(A)-840(N) to progress through the neural network. It will be appreciated that node parameters / weights, and possibly edge weights between nodes, are modified according to one or more machine learning training methods. Once hidden layer 840(N) processes data from the previous hidden layer, the data is passed to score selector 850. Score selector 850 may utilize one or more score selection criteria and values ​​received from hidden layer 840(N) to determine the predicted skill 860 output by skill classifier 800. For example, score selector 850 may receive one or more score values ​​for several possible skills from which predicted skill 860 may be selected, and utilize criteria specifying that the highest score be selected to select predicted skill 860.

[0142] Noise Compensation Training As mentioned above, dropout rates for contextual and expressive features can be used to balance feature processing by the machine learning model and produce more accurate predictions. In some cases, a contextual feature may be less likely to produce accurate results than the corresponding expressive feature, and vice versa, meaning that excluding a phrase entirely would be beneficial. As explained above, using a gazetteer to generate expressive features can result in the generation of features that correspond to false positives for matching natural language words to the gazetteer. For example, the phrase "Mark these exams as soon as possible" utilizes the word "Mark" as a verb, but a gazetteer with a label type of "Name" may generate an expressive feature that weights the model to recognize the word as "Name." The problem increases as the gazetteer includes more common words in natural language. For example, the words "An" and "The" are names that can be recognized by the gazetteer, but are not common in English. They are also always common connecting words. Expressive features generated for these words could overwhelm the model with the implication that a phrase using these words corresponds to a name, even if this is unlikely. In this way, expressive features can cause a very high proportion of "noise" or discrimination of expressive features in a given natural language query. Noise can be defined using the following formula: N=a / (a+b) Formula (1) where N is a score representation of the generated noise, a is the number of violating phrases (i.e., phrases in the natural language query that match phrases in the gazetteer but whose correct labels do not match the gazetteer's labels), and b is the number of matching phrases (phrases that match phrases in the gazetteer and have correct labels that match the gazetteer's labels). For example, a gazetteer for "Names" containing the words "An" and "The" generates a very large amount of noise (value a is very high and value b is very low). The noise will slow down the training of the machine learning model and the predictions made by the machine learning model due to all the false positives generated as expressive features.

[0143] Conversely, if the noise is very small (i.e., the expressive features are generated only sparingly or not highly weighted), the precision of the expressive features will be high (i.e., the expressive features will not generate many false positives for a phrase), but the recall of the features will be very low (i.e., the expressive features will be underutilized and may produce many false negatives). Therefore, a certain level of noise is desirable when considering expressive features for input to a machine learning model. The level of noise can be controlled by performing contextual or expressive dropout of features to meet a noise threshold range. The number of violating phrases and matching phrases can be determined by comparing the gazetteer to a training dataset used to train the machine learning model. The training dataset provides a good opportunity to determine predicted noise values, since the training dataset was used to train the machine learning model, allowing for more accurate predictions of how much noise the gazetteer will generate when used to generate expressive features for natural language queries.

[0144] FIG. 9 illustrates a process flow for performing noise-based feature dropout as part of multi-feature balancing for a natural language processor, according to various embodiments. The process illustrated in FIG. 9 is implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or a combination thereof. The software is stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 9 and described below is intended to be exemplary and non-limiting. While FIG. 9 illustrates various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel. In various embodiments, the steps detailed in process 900 are performed by the systems and models discussed herein with respect to FIGS. 1-8.

[0145] Process 900 begins at 730 of process 700 where a set of contextual features and a set of expressive features of the natural language query are determined.

[0146] At 910, after the expressive and contextual features have been determined, the training dataset is compared with the dataset of natural language phrases to determine the number of partial matches between the phrase words and categories and the number of exact matches between the phrase words and categories. The comparison of words and categories corresponds to determining overlapping phrases between the training data and the gazetteer and determining whether the overlapping words correspond to the same category label (e.g., "Location"). A partial match occurs when a phrase is included in both the training data and the gazetteer, but the gold label associated with the phrase in the training data does not match the attribute label in the gazetteer. An exact match occurs when a phrase is included in both the training data and the gazetteer, and the gold label associated with the phrase in the training data matches the attribute label in the gazetteer.

[0147] At 920, a noise score for the comparison is generated based on the number of partial matches and the number of exact matches. For example, in equation (1), the noise score N is generated by dividing the number of partial matches a by the sum of the partial matches and exact matches a+b.

[0148] At 930, it is determined whether the noise score is within an acceptable range. A specified range of noise may be tolerated to avoid excessive noise that would interfere with the processing of the machine learning model, while ensuring that an appropriate level of recall is achieved by the model when utilizing expressive features. The acceptable range may be provided by a data scientist, for example, or may be achieved during hyperparameter tuning of the machine learning model.

[0149] At 940, if the noise score is not within the acceptable range, the contextual / expressive dropout rate is updated according to the noise score generated at 920. For example, given an acceptable noise range of 0.05 to 0.15, the generated noise score may be 0.2. Therefore, when training a machine learning model or generating predicted output in response to a natural language query from a client, some expressive features should be dropped out prior to input to the machine learning model to avoid generating excessive noise. Reducing the expressive features reduces the number of partial matches at prediction time, thus reducing the noise score. Alternatively, given the same acceptable noise range, the generated noise score may be 0.02. Therefore, to increase the weight given to the expressive features, some contextual features should be dropped out prior to input to the machine learning model. This may increase the noise rate, but also improve the reproducibility of the machine learning model.

[0150] At 950, contextual / expressive dropout is performed according to the updated contextual / expressive dropout rates. The dropout performed is described above with reference to FIGS. 7 and 8.

[0151] At 960, following the implementation of contextual / expressive dropout, or if the noise score is already within the tolerance range determined at 930, the machine learning model processes the modified set of contextual features and the modified set of expressive features that may have been modified by dropout at 950.

[0152] Model combination method In various embodiments, multiple balancing techniques, including different ratios, may be used to generate predictions by a machine learning model. Two or more balancing techniques may be combined in any manner necessary to achieve more efficient training and prediction results according to the embodiments discussed herein. For example, the coverage balancing subsystem 422 may perform coverage balancing to modify the training dataset and / or gazetteer prior to generating expressive features and contextual features. The resulting modified training dataset may be used to retrain the machine learning model. The modified gazetteer would then be utilized to generate expressive features alongside the contextual features that are input to the retrained machine learning model. The dropout balancing subsystem 424 and / or noise balancing subsystem 426 may then perform rate-based and / or noise-based dropout of features before inputting them into the retrained machine learning model. It will be understood that any combination of the balancing processes described herein may be used in any combination or ratio to achieve more accurate prediction and recall for the machine learning model.

[0153] Exemplary System 10 shows a simplified diagram of a distributed system 1000. In the illustrated example, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 may be configured to run one or more applications.

[0154] In various examples, server 1012 may be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 1012 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some examples, these services may be provided as web-based services or cloud services, such as under a Software as a Service (SaaS) model, to users of client computing devices 1002, 1004, 1006, and / or 1008. A user operating a client computing device 1002, 1004, 1006, and / or 1008 may utilize one or more client applications to interact with the server 1012 to utilize the services provided by these components.

[0155] 10, server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by server 1012. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be appreciated that a wide variety of system configurations are possible that may differ from distributed system 1000. Thus, the example shown in FIG. 10 is an example of a distributed system for implementing the example system and is not intended to be limiting.

[0156] A user uses client computing devices 1002, 1004, 1006, and / or 1008 to execute one or more applications, models, or chatbots, which may generate one or more events or models, which may then be implemented or processed according to the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. The client device may also output information to the user through this interface. FIG. 10 illustrates four client computing devices. Although only one client computing device is shown, any number of client computing devices may be supported.

[0157] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems), various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, etc.), and the like. Portable handheld devices may include cellular phones, smartphones (e.g., iPhone (registered trademark), tablet (e.g., iPad (registered trademark)), personal digital assistant (PDA) and the like. Wearable devices may include Google Glass® head-mounted displays and other devices. Gaming systems may include various handheld gaming devices and Internet-connectable gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.). Client devices may be capable of running a wide variety of applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use a variety of communication protocols.

[0158] Network 1010 may be any type of network known to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc., by way of example only. As such, the network 1010 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a wireless network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, Bluetooth and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0159] The servers 1012 may be comprised of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 1012 may include one or more virtual machines running a virtual operating system or other computing architecture involving virtualization, such as one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the servers. In various examples, the servers 1012 may be adapted to run one or more services or software applications that provide the functionality described in the above disclosure.

[0160] The computing systems within server 1012 may run one or more operating systems, including any of the operating systems described above, as well as commercially available server operating systems. Server 1012 may also run any of a variety of other server and / or middle-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA server, a database server, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc. It will not be done.

[0161] In some implementations, server 1012 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 1012 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0162] The distributed system 1000 may also include one or more data repositories 1014, 1016. In certain examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 1014, 1016 may be used to store information, such as information related to chatbot performance or generated models for use by the chatbot used by the server 1012 when performing various functions according to various embodiments. The data repositories 1014, 1016 may reside in a variety of locations. For example, the data repository used by the server 1012 may be local to the server 1012 or may be remote from the server 1012 and communicate with the server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 may be of different types. In certain examples, the data repository used by the server 1012 may be a database, for example, a relational database such as those provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0163] In particular examples, one or more of the data repositories 1014, 1016 may be used by an application to store application data. The data repositories used by the application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general-purpose storage repository supported by a file system.

[0164] In certain examples, the functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 11 illustrates a diagram of a particular example of providing various services as cloud services. 11 is a simplified block diagram of a cloud-based system environment that may include a cloud infrastructure system 1102. In the example shown in FIG. 11, a cloud infrastructure system 1102 may provide one or more cloud services that users may request using one or more client computing devices 1104, 1106, and 1108. Cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above with respect to server 1012. The computers in cloud infrastructure system 1102 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0165] The network 1110 may facilitate communication and exchange of data between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network 1110 may include one or more networks. The networks may be of the same type or different types. The network 1110 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0166] The example shown in Figure 11 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other examples, cloud infrastructure system 1102 may have more or fewer components than those shown in Figure 11, may combine two or more components, or may have components in a different configuration or arrangement. For example, while Figure 11 shows three client computing devices, in alternative examples, any number of client computing devices may be supported.

[0167] The term cloud service is generally used to refer to services made available to users on demand via a communications network, such as the Internet, by a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that comprise the cloud service provider's system are distinct from a customer's own on-premise servers and systems. The cloud service provider's systems are managed by the cloud service provider. Thus, customers can use cloud services offered by the cloud service provider without purchasing separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system may host applications, and users can order and use the applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation of Redwood Shores, California.

[0168] In particular examples, cloud infrastructure system 1102 may provide one or more cloud services using various models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including hybrid service models. Cloud infrastructure system 1102 may provide applications, middleware, databases, and other related services that enable the provisioning of various cloud services. It may also contain a suite of other resources.

[0169] The SaaS model allows applications or software to be delivered as a service to customers over a communications network such as the Internet, without the customer having to purchase the underlying application hardware or software. For example, the SaaS model may be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 1102. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0170] The IaaS model is commonly used to provide flexible computing and storage capabilities by providing infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation (registered trademark).

[0171] The PaaS model is generally used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or manage the environment resources. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0172] Cloud services are generally provided on an on-demand, self-service basis, on a subscription basis, and in a flexible, scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services offered by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take a particular action (e.g., intent), as described above, and / or to provide a service for a chatbot system, as described herein. Cloud infrastructure system 1102 may be configured to provide one cloud service or even multiple cloud services.

[0173] Cloud infrastructure system 1102 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 1102 may be owned by a third-party cloud service provider, and cloud services are offered to general public customers. These customers may be individuals or businesses. In another example, under a private cloud model, cloud infrastructure system 1102 may function within an organization (e.g., within a corporate organization), and services are offered to customers within the organization. For example, these customers may be various departments within a company, such as the human resources department, payroll department, or individuals within the company. In another example, under a community cloud model, cloud infrastructure system 1102 and the services it offers may be shared among various organizations within an associated community. Various other models, including hybrids of the above models, may also be used.

[0174] Client computing devices 1104, 1106, and 1108 may be of different types (e.g., client computing devices 1002, 1004, 1006, and 1008 shown in FIG. 10) and may be capable of operating one or more client applications. Users may use the client devices to interact with cloud infrastructure system 1102, such as to request services provided by cloud infrastructure system 1102. For example, users may use client devices to request information or actions from a chatbot, as described in this disclosure.

[0175] In some examples, the processing performed by cloud infrastructure system 1102 to provide services may include model training and deployment. This analysis may include using, analyzing, and processing data sets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing the data in parallel, running simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 1102 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0176] 11 , cloud infrastructure system 1102 may include infrastructure resources 1130 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc. In particular examples, storage virtual machines available to handle storage requested by applications may be part of cloud infrastructure system 1102. In other examples, the storage virtual machines may be part of a different system.

[0177] In certain examples, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 1102 to different customers, resources may be organized into resource sets or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated, optimized combination of one or more types of resources. In certain examples, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, while a second set of pods may be provisioned for Java services, etc., which may include a different combination of resources than the pods in the first set of pods. For some services, the resources allocated for provisioning these services may be shared between services.

[0178] Cloud infrastructure system 1102 itself may use services 1132 internally that are shared by different components of cloud infrastructure system 1102 and that facilitate the provisioning of services by cloud infrastructure system 1102. These internal shared services include security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services enabling cloud support, email services, and the like. This may include, but is not limited to, services, notification services, file transfer services, etc.

[0179] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 11 , the subsystems may include a user interface subsystem 1112 that allows users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 through which cloud services offered by cloud infrastructure system 1102 are advertised and available for consumer purchase, and other interfaces 1118. For example, a customer may use a client device to request one or more services (service request 1134) offered by cloud infrastructure system 1102 using one or more of interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 1102, and place a subscription order for one or more services offered by cloud infrastructure system 1102 and for which the customer wishes to subscribe. The service request may include information identifying the customer and one or more services for which the customer wishes to subscribe. For example, a customer may submit an order to subscribe to services provided by cloud infrastructure system 1102. As part of the order, the customer may provide information identifying the chatbot system for which the service will be provided, and optionally one or more credentials for the chatbot system.

[0180] 11 , cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this processing, OMS 1120 may be configured to create an account for the customer if not already created, receive billing and / or account information from the customer to use for billing the customer for providing the requested services to the customer, verify the customer information, and, once verified, reserve the order for the customer and prepare the order for provisioning by coordinating various workflows.

[0181] Upon proper validation, the OMS 1120 may invoke an order provisioning subsystem (OPS) 1124 configured to provision resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a workflow, the OPS 1124 may be configured to determine the specific cloud service being requested and identify the number of pods that will be pre-configured for this specific cloud service. The number of pods allocated for an order may depend on the size / amount / level / scope of the service requested. For example, the number of pods to allocate may be determined based on the number of users the service is to support, the duration for which the service is requested, etc. The allocated pods may then be customized to the specific requesting customer to provide the requested service.

[0182] In a particular example, the setup phase may involve the cloud infrastructure, as described above. This may be performed as part of the provisioning process by cloud infrastructure system 1102. Cloud infrastructure system 1102 may generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by cloud infrastructure system 1102 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 1102.

[0183] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service will be available for use. In some examples, the customer may be sent information (e.g., a link) that enables the customer to begin using and utilizing the benefits of the requested service. In particular examples, for a customer requesting a service, the response may include a chatbot system ID generated by cloud infrastructure system 1102 and information identifying the chatbot system selected by cloud infrastructure system 1102 for the chatbot system corresponding to the chatbot system ID.

[0184] Cloud infrastructure system 1102 may provide services to multiple customers. For each customer, cloud infrastructure system 1102 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and is responsible for providing the requested services to the customer. Cloud infrastructure system 1102 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information may be used to bill the customer. Billing may be on a monthly basis, for example.

[0185] Cloud infrastructure system 1102 may provide services to multiple customers in parallel. Cloud infrastructure system 1102 may store information about these customers, possibly including copyright information. In particular examples, cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage customer information and separate the managed information so that information about one customer is not accessed from information about another customer. IMS 1128 may be configured to provide various security-related services, such as identity services such as information access management, authentication and authorization services, services for managing customer identities and roles and associated capabilities, etc.

[0186] 12 illustrates an example computer system 1200. In some examples, the computer system 1200 may be used to implement any of the digital assistant or chatbot systems in a distributed environment, as well as the various servers and computer systems described above. As shown in FIG. 12, the computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with several other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 may include non-transitory computer-readable storage media, including a storage medium 1222 and a system memory 1210.

[0187] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. Although bus subsystem 1202 is shown schematically as a single bus, the bus subsystem Alternatives may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a bus conforming to the IEEE P1386.1 standard. The bus may include a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured by

[0188] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1204 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 1204 may use customized circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0189] In some examples, processing units within processing subsystem 1204 may execute instructions stored in system memory 1210 or computer-readable storage medium 1222. In various examples, the processing units may execute various program or code instructions and maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in system memory 1210 and / or computer-readable storage medium 1222, potentially including one or more storage devices. Through appropriate programming, processing subsystem 1204 may provide the various functions described above. In examples in which computer system 1200 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0190] In certain examples, a processing acceleration unit 1206 may optionally be provided to accelerate the overall processing performed by the computer system 1200, to perform customized processing, or to offload some of the processing performed by the processing subsystem 1204.

[0191] I / O subsystem 1208 may include devices and mechanisms for inputting information into computer system 1200 and / or outputting information from or through computer system 1200. In general, use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information into computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. The user interface input devices may also include motion sensing and / or gesture recognition devices, such as a Microsoft Kinect® motion sensor that allows a user to control and interact with the input device, a Microsoft Xbox® 360 game controller, or a device that provides an interface for receiving input using gestures and voice commands. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector that detects eye movements from the user (e.g., "blinks" while taking a picture and / or making a menu selection) and translates the eye gestures as input to the input device (e.g., Google Glass®). The user interface input devices may also include voice recognition sensing devices that allow a user to interact with a voice recognition system (e.g., Siri® navigator) via voice commands.

[0192] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0193] In general, the use of the term output device(s) is intended to encompass all conceivable types of devices and mechanisms for outputting information from computer system 1200 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, plotting devices, touch screens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0194] Storage subsystem 1218 provides a repository or data store for storing information and data used by computer system 1200. Storage subsystem 1218 provides a tangible, non-transitory, computer-readable storage medium for storing the basic programming and data constructs that provide some example functionality. Software (e.g., programs, code modules, instructions) that, when executed by processing subsystem 1204, provide the functionality described above may be stored in storage subsystem 1218. The software may be executed by one or more processing units of processing subsystem 1204. Storage subsystem 1218 may also provide authentication in accordance with the teachings of the present disclosure.

[0195] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in Figure 12, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include a volatile main random access memory (RAM) for storing instructions and data during program execution, and a non-volatile read-only memory (ROM) in which fixed instructions are stored. The computer system 1200 may include several memory modules, including ROM (Read Only Memory) or flash memory. In some implementations, the basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1200, such as during start-up, may typically be stored in ROM. Typically, RAM contains data and / or program modules currently operated on and executed by processing subsystem 1204. In some implementations, system memory 1210 may include multiple different types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0196] 12, system memory 1210 may load running application programs 1212, program data 1214, and operating system 1216, which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc. By way of example, operating system 1216 may be a Microsoft Windows®, Apple Macintosh®, and / or Linux operating system. rating systems, various commercially available UNIX or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome OS, etc.), and / or iOS ), Windows Phone, various versions of mobile operating systems such as Android® OS, BlackBerry® OS, Palm® OS operating systems, and the like.

[0197] The computer-readable storage medium 1222 may store programming and data constructs that provide some example functionality. The computer-readable storage medium 1222 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1200. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the above-described functionality may be stored in the storage subsystem 1218. By way of example, the computer-readable storage medium 1222 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 1222 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disc, a digital video tape, etc. The computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0198] In particular examples, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may be further connected to a computer-readable storage medium 1222. Reader 1220 may be configured to receive and read data from a memory device such as a disk, flash drive, or the like.

[0199] In certain examples, computer system 1200 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, computer system 1200 may provide support for running one or more virtual machines. In certain examples, computer system 1200 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically executes independently from other virtual machines. A virtual machine typically executes its own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 1200. Thus, potentially multiple operating systems may be executed simultaneously by computer system 1200.

[0200] The communications subsystem 1224 provides an interface to other computer systems and networks. The communications subsystem 1224 serves as an interface for sending and receiving data between other systems and the computer system 1200. For example, the communications subsystem 1224 may enable the computer system 1200 to establish a communications channel to one or more client devices over the Internet to send and receive information to and from the one or more client devices. For example, if the computer system 1200 is used to implement the bot system 120 shown in FIG. 1, the communications subsystem may be used to communicate with a chatbot system selected for the application.

[0201] The communications subsystem 1224 may support both wired and / or wireless communication protocols. In certain examples, the communications subsystem 1224 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (High Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, the communications subsystem 1224 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0202] The communications subsystem 1224 may receive and transmit data in a variety of formats. In some examples, the communications subsystem 1224 may receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, and the like, among other formats. For example, the communications subsystem 1224 may receive incoming communications from social media networks and / or Twitter (registered users). (trademark) Feeds, Facebook® Updates, Rich Site Summary (RSS) Feeds The network may be configured to receive (or transmit) data feeds 1226 in real time from users of other communications services, such as web feeds, such as Yahoo! News, and / or real-time updates from one or more third-party sources.

[0203] In particular examples, the communications subsystem 1224 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230 that may be continuous or infinite in nature without a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0204] The communications subsystem 1224 communicates data from the computer system 1200 to other computers. The data may be configured to be communicated to one or more computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1200.

[0205] Computer system 1200 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of computer system 1200 shown in FIG. 12 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 12. It should be recognized, based on the disclosure and teachings herein, that there are other aspects and / or methods for implementing the various examples.

[0206] While specific examples have been described, various variations, modifications, alternative configurations, and equivalents are possible. The examples are not limited to operation in a particular data processing environment, but may freely operate in multiple data processing environments. Furthermore, while the examples have been described using a particular sequence of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. Additionally, the order of operations may be re-specified. A process may have additional steps not included in the figures. Various features and aspects of the above examples may be used individually or together.

[0207] Additionally, while particular examples have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are possible. Particular examples may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or any combination of different processors.

[0208] Where a device, system, component, or module is described as being configured to perform a particular operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, by executing, for example, computer instructions or code, or a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0209] In this disclosure, specific details are provided to ensure that the examples are thoroughly understood. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail so as not to obscure the examples. This specification provides illustrative examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the examples The above description provides one skilled in the art with an enabling description for implementing various examples. Various changes are possible within the function and arrangement of elements.

[0210] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the claims. Thus, while specific examples have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0211] While the foregoing specification describes aspects of the disclosure with reference to specific examples thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above disclosure may be used individually or together. Moreover, the examples can be utilized in a variety of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive.

[0212] In the above description, the methods are described in a particular order for purposes of illustration. It should be understood that in alternative examples, the methods may be performed in an order different from that described. It should also be understood that the methods described above may be performed by hardware components or embodied in a sequence of machine-executable instructions that, when used, may cause a machine, such as a general-purpose or special-purpose processor or logic circuitry programmed with such instructions, to perform the method. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, floppy disk, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0213] Where a component is described as being configured to perform particular operations, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the particular operations, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the particular operations, or any combination thereof.

[0214] While illustrative examples of the present application have been described in detail herein, it is to be understood that the concepts of the present invention may be variously embodied and employed in other forms, and that the claims are intended to be construed to include such variations except insofar as limited by the prior art.

Claims

1. 1. A computer-implemented method comprising: receiving, at a computing device, an indication of a first coverage value corresponding to a desired overlap between a dataset of natural language phrases and a training dataset for training a machine learning model; determining, by the computing device, a second coverage value corresponding to the measured overlap between the data set of natural language phrases and the training data set; determining, by the computing device, a coverage delta value based on a comparison between the first coverage value and the second coverage value; the computing device revising at least one of the natural language phrase dataset and the training dataset based on the coverage delta value; and processing an input dataset comprising a set of input features using a machine learning model including the modified dataset of natural language phrases, the machine learning model processing the input dataset based at least in part on the dataset of natural language phrases to generate an output dataset.

2. 2. The method of claim 1, further comprising determining the second coverage value by determining a number of natural language phrases from the dataset of natural language phrases that are also present in the training data, wherein each of the natural language phrases that are also in the dataset corresponds to a category that matches a category associated with the dataset of natural language phrases.

3. 3. The method of claim 2, wherein modifying at least one of the dataset of natural language phrases and the training dataset comprises modifying the dataset of natural language phrases by updating the dataset of natural language phrases to include one or more natural language phrases associated with the category from the training data, wherein the updated dataset of natural language phrases includes some natural language phrases that are also present in the training data at a rate equal to or greater than the first coverage value.

4. 3. The method of claim 2, wherein modifying at least one of the dataset of natural language phrases and the training dataset comprises modifying the training dataset by updating the training dataset to include one or more natural language phrases from the dataset of natural language phrases and associating the one or more natural language phrases with the categories, wherein the dataset of natural language phrases includes some natural language phrases that are also present in the updated training data at a rate equal to or greater than the first coverage value.

5. 5. The method of claim 4, wherein updating the training dataset to include the one or more natural language phrases from the dataset of natural language phrases comprises generating one or more training pairs from the one or more natural language phrases, the one or more training pairs including natural language queries generated from the natural language phrases and gold label categories that match the categories in the dataset of natural language phrases.

6. Processing the input dataset causes the machine learning model to 6. The method of claim 5, further comprising processing a training dataset to retrain the machine learning model.

7. 10. The method of claim 1, wherein processing the input dataset includes the machine learning model processing a natural language query received by a chatbot system, and the machine learning model is configured to generate an output dataset including at least one of a skill and an intent associated with the chatbot for responding to the natural language query.

8. 2. The method of claim 1 , wherein the machine learning model is a convolutional neural network machine learning model, and the set of input features corresponds to input nodes of the convolutional neural network.

9. 1. A computer-implemented method comprising: The method includes receiving, by a computing device, a natural language query to be processed by a machine learning model, the machine learning model utilizing a dataset of natural language phrases to process the natural language query, the method further comprising: determining, by the computing device, a feature dropout value based on the machine learning model and the natural language query; generating, based on the natural language query, one or more contextual features and one or more expressive features that can be input to the machine learning model; modifying, by the computing device, at least one of the one or more contextual features and the one or more expressive features based on the feature dropout value to generate a set of input features for the machine learning model; and the computing device utilizing the machine learning model to process the set of input features to generate an output dataset corresponding to the natural language query.

10. the feature dropout value is a first contextual feature dropout value corresponding to a percentage of contextual features of the one or more contextual features; The method further includes modifying the one or more contextual features by removing a percentage of contextual features from the one or more contextual features based on the first contextual feature dropout value; The method of claim 9 , wherein the set of input features is generated from the modified one or more contextual features and the one or more expressive features.

11. the feature dropout values ​​further include a second contextual feature dropout value corresponding to a percentage of contextual features of the one or more contextual features corresponding to a natural language phrase in the dataset of natural language phrases; The method further includes determining a subset of contextual features, each contextual feature of the subset of contextual features corresponding to a natural language phrase in the dataset of natural language phrases; The method further includes modifying the subset of contextual features by removing a percentage of contextual features from the subset of contextual features that corresponds to the second contextual feature dropout value; 11. The method of claim 10, wherein modifying the one or more contextual features comprises removing a percentage of contextual features from the one or more contextual features, the percentage comprising the modified subset of contextual features, based on the first contextual feature dropout value.

12. the feature dropout value is a first expressive feature dropout value corresponding to a percentage of expressive features of the one or more expressive features; the method further includes modifying the one or more expressive features by removing a percentage of expressive features from the one or more expressive features based on the first expressive feature dropout value; The method of claim 9 , wherein the set of input features is generated from the one or more contextual features and the modified one or more expressive features.

13. comparing the dataset of natural language phrases with a training dataset used to train the machine learning model; determining a noise value based on the comparison, the noise value corresponding to a number of natural language phrases associated with the same particular category in the data set of natural language phrases and the training data set, and a number of natural language phrases associated with different categories in the data set of natural language phrases and the training data set; The method of claim 9 , wherein the feature dropout value is determined based at least in part on the noise value.

14. 10. The method of claim 9, wherein the machine learning model is a convolutional neural network machine learning model, and the set of input features corresponds to input nodes of the convolutional neural network.

15. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions, when executed by the one or more processors, causing the one or more processors to: and causing a computing device to receive a natural language query to be processed by a machine learning model, the machine learning model utilizing a dataset of natural language phrases to process the natural language query, the instructions, when executed by the one or more processors, to cause the one or more processors to: determining, by the computing device, a feature dropout value based on the machine learning model and the natural language query; generating, by the computing device, one or more contextual features and one or more expressive features based on the natural language query, which can be input into the machine learning model; modifying, by the computing device, at least one of the one or more contextual features and the one or more expressive features based on the feature dropout value to generate a set of input features for the machine learning model; and (c) processing the set of input features using the machine learning model to generate an output dataset corresponding to the natural language query.

16. the feature dropout value is a first contextual feature dropout value corresponding to a percentage of contextual features of the one or more contextual features; the one or more processors further perform modifying the one or more contextual features by removing a percentage of contextual features from the one or more contextual features based on the first contextual feature dropout value; The non-transitory computer-readable memory of claim 15 , wherein the set of input features is generated from the modified one or more contextual features and the one or more expressive features.

17. the feature dropout values ​​further include a second contextual feature dropout value corresponding to a percentage of contextual features of the one or more contextual features corresponding to a natural language phrase in the dataset of natural language phrases; the one or more processors are further configured to determine a subset of contextual features, each contextual feature of the subset of contextual features corresponding to a natural language phrase in the dataset of natural language phrases; the one or more processors further perform modifying the subset of contextual features by removing a percentage of contextual features from the subset of contextual features that corresponds to the second contextual feature dropout value; 17. The non-transitory computer-readable memory of claim 16, wherein modifying the one or more contextual features comprises removing a percentage of contextual features from the one or more contextual features, the percentage comprising the subset of modified contextual features based on the first contextual feature dropout value.

18. the feature dropout value is a first expressive feature dropout value corresponding to a percentage of expressive features of the one or more expressive features; the one or more processors further perform modifying the one or more expressive features by removing a percentage of expressive features from the one or more expressive features based on the first expressive feature dropout value; The non-transitory computer-readable memory of claim 15 , wherein the set of input features is generated from the one or more contextual features and the modified one or more expressive features.

19. The one or more processors further comparing the dataset of natural language phrases with a training dataset used to train the machine learning model; determining a noise value based on the comparison, the noise value corresponding to a number of natural language phrases associated with the same particular category in the data set of natural language phrases and the training data set and a number of natural language phrases associated with different categories in the data set of natural language phrases and the training data set; The non-transitory computer-readable memory of claim 15 , wherein the feature dropout value is determined based at least in part on the noise value.

20. 16. The non-transitory computer-readable memory of claim 15, wherein the machine learning model is a convolutional neural network machine learning model, and the set of input features corresponds to input nodes of the convolutional neural network.

Images