Noise Data Augmentation for Natural Language Processing

Abstract

Noisy data augmentation techniques for training chatbot systems in natural language processing. In one particular aspect, a method is provided, comprising: receiving a set of training utterances for training an intent classifier to recognize one or more intents of one or more utterances; augmenting the set of training utterances with noise text to generate an augmented set of training utterances; and training the intent classifier using the augmented set of training utterances. The augmentation comprises: obtaining noise text unrelated to original text in utterances of the training utterance set from a word list, a text corpus, a publication, a dictionary, or any combination thereof, and merging the noise text into the utterances at a predefined augmentation ratio relative to the original text in the utterances of the training utterance set to generate the augmented utterances.

Description

[0001] Priority claim

[0002] This application is a non-provisional application to and claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 63 / 002,066, filed on March 30, 2020, entitled “NOISE DATA AUGMENTATION FOR NATURAL LANGUAGE PROCESSING.” The entire contents of the aforementioned application are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to chatbot systems, and more particularly to noisy data augmentation techniques for training chatbot systems in natural language processing. Background Art

[0004] To get an immediate response, many users around the world use instant messaging or chat platforms. Organizations often use these instant messaging or chat platforms to conduct real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots or robots have begun to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with robots through messaging applications that end users have already installed and use. Intelligent robots (usually powered by artificial intelligence (AI)) can communicate more intelligently and contextually in real-time conversations and therefore can allow more natural conversations between robots and end users to improve the conversation experience. Instead of the end user having to learn a fixed set of keywords or commands that the robot knows how to respond to, the intelligent robot can understand the end user's intentions based on the user's utterances in natural language and respond accordingly.

[0005] However, chatbots are difficult to build because these automated solutions require specific domain knowledge and the application of certain techniques that may only be within the capabilities of professional developers. As part of building such a chatbot, developers can first understand the needs of the business and end users. Developers can then analyze and make decisions related to, for example: selecting a dataset to analyze; preparing the input dataset for analysis (e.g., cleaning, extracting, formatting, and / or transforming the data before analysis, performing data feature engineering, etc.); identifying appropriate machine learning (ML) techniques or model(s) for performing the analysis; and refining the techniques or models to improve results / performance based on feedback. The task of identifying an appropriate model can include developing multiple models (possibly in parallel), iteratively testing and experimenting with these models, and then identifying a specific model (or models) to use. Furthermore, supervised learning-based solutions typically involve a training phase, followed by an application (i.e., inference) phase, and an iterative cycle between the training and application phases. Developers can be responsible for carefully implementing and monitoring these phases to achieve an optimal solution. For example, to train ML techniques or models, accurate training data is needed to enable the algorithms to understand and learn certain patterns or features (e.g., for chatbots—intent extraction and careful syntactic analysis are required, not just raw language processing), which the ML techniques or models will use to predict desired outcomes (e.g., infer intent from utterances). To ensure that the ML techniques or models correctly learn these patterns and features, developers may be responsible for selecting, enriching, and optimizing the training datasets for the ML techniques or models. Summary of the Invention

[0006] The technology disclosed herein generally relates to chatbots. More specifically and without limitation, the technology disclosed herein relates to techniques for noisy data augmentation of training datasets for training chatbot systems in natural language processing. The chatbot can classify user utterances into different categories, such as the user's predefined intents. The chatbot's classifier may include a trained ML model that generates outputs (e.g., intents) based on inputs (e.g., user utterances). When there is insufficient training data for training the trained ML model, the chatbot may more frequently determine incorrect intents. The technology disclosed herein can provide a noisy augmented dataset for training the ML model, making the ML model more resilient to irrelevant contexts and more accurately learning patterns or boundaries of intents.

[0007] In various embodiments, a computer-implemented method is provided, the computer-implemented method comprising: receiving, at a data processing system, a set of training utterances for training an intent classifier to identify one or more intents of one or more utterances; augmenting, by the data processing system, the set of training utterances with noise text to generate an augmented set of training utterances, wherein the augmenting comprises: obtaining noise text unrelated to original text in utterances of the training utterance set from a word list, a text corpus, a publication, a dictionary, or any combination thereof, and incorporating the noise text into the utterances at a predefined augmentation ratio relative to the original text in the utterances of the training utterance set to generate the augmented utterances; and training, by the data processing system, the intent classifier using the augmented set of training utterances.

[0008] In some embodiments, the noise text is set with a predefined expansion ratio of 1:0.5 to 1:5 in the following manner: (i) in front of the original text in the utterance, (ii) behind the original text of the utterance, (iii) on both sides of the original text of the utterance, (iv) integrated in the original text of the utterance, (v) or a combination of the foregoing.

[0009] In some embodiments, the noise text is a random text string automatically generated from a stop word list.

[0010] In some embodiments, the noise text is a text sentence automatically copied verbatim from a corpus.

[0011] In some embodiments, the noise text is a random text string automatically generated from a corpus.

[0012] In some embodiments, the noise text is a random text string automatically generated from a dictionary, wherein the frequency of words or characters selected for the random text string is not considered.

[0013] In some embodiments, the noise text is a random text string automatically generated from a dictionary, taking into account the frequency of words or characters selected for the random text string.

[0014] In various embodiments, a computer-implemented method is provided, comprising: receiving, by a chatbot system, an utterance generated by a user interacting with the chatbot system; classifying the utterance into an intent category corresponding to an intent using an intent classifier deployed within the chatbot system, wherein the intent classifier comprises a plurality of model parameters identified using training data, the training data comprising: an expanded training utterance set for training the intent classifier to identify one or more intents of one or more utterances, wherein the expanded training utterance set is artificially generated to include expanded utterances from the training utterance set, wherein noise text is merged into an utterance from the training utterance set at a predefined expansion ratio relative to original text in the utterance to generate the expanded utterance, and wherein the plurality of model parameters are identified using the training data based on minimizing a loss function; and outputting the intent based on the classification using the intent classifier.

[0015] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0016] In various embodiments, a computer program product is provided, the computer program product being tangibly embodied in a non-transitory machine-readable storage medium and comprising instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0017] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system comprises a non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes.

[0018] The technology described above and below can be implemented in a variety of ways and in a variety of contexts. As described in more detail below, several example implementations and contexts are provided with reference to the following figures. However, the following implementations and contexts are only some of many implementations and contexts. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a simplified block diagram of a distributed environment incorporating an exemplary embodiment.

[0020] Figure 2 is a simplified block diagram of a computing system implementing a master robot, according to some embodiments.

[0021] Figure 3 is a simplified block diagram of a computing system that implements a skill bot in accordance with certain embodiments.

[0022] Figure 4 is a simplified block diagram of a chatbot training and deployment system according to various embodiments.

[0023] Figure 5 Illustrated is a process flow for augmenting a training dataset with noise according to various embodiments.

[0024] Figure 6 Depicted is a simplified diagram of a distributed system for implementing various embodiments.

[0025] Figure 7 is a simplified block diagram of one or more components of a system environment according to various embodiments by which services provided by one or more components of an embodiment system may be provided as a cloud service.

[0026] Figure 8 An example computer system is illustrated that can be used to implement various embodiments. DETAILED DESCRIPTION

[0027] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that the various embodiments can be practiced without these specific details. The drawings and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0028] introduction

[0029] A digital assistant is an AI-driven interface that helps users complete various tasks in natural language conversations. For each digital assistant, customers can assemble one or more skills. Skills (also described herein as chatbots, robots, or skillbots) are individual robots that focus on specific types of tasks such as tracking inventory, submitting time cards, and creating expense reports. When an end user engages with a digital assistant, the digital assistant evaluates the end user input and routes the conversation to and from the appropriate chatbot. Messenger (instant messaging), SKYPE Various channels, such as messenger or Short Message Service (SMS), make digital assistants available to end users. Channels enable chats to flow back and forth between the end user and the digital assistant and its various chatbots over various messaging platforms. Channels can also support user agent escalation, event-initiated conversations, and testing.

[0030] Intents allow a chatbot to understand what the user wants the chatbot to do. Intents include a sequence of typical user requests and statements, which can also be called utterances (e.g., get an account balance, make a purchase, etc.). As used herein, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. Intents can be created by providing a name that describes a certain user action (e.g., ordering a pizza) and compiling a set of real-life user statements or utterances that are typically associated with triggering that action. Because the chatbot's cognition is derived from these intents, each intent can be created from a robust and varied dataset (one to two dozen utterances), allowing the chatbot to interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what the user wants when it receives messages such as "Forget this order!" or "Cancel delivery!" (messages that mean the same thing but are expressed in different ways). Intents and the utterances belonging to the intents together constitute the chatbot's training corpus. By training a model with the corpus, the customer can essentially turn the model into a reference tool for parsing end-user input into a single intent. Customers can improve the cognitive acuity of chatbots through multiple rounds of intent testing and intent training.

[0031] However, building a chatbot that can determine the intention of an end user based on user speech is a challenging task to some extent, partly due to the subtlety and ambiguity of natural language and the dimensionality of the input space (e.g., possible user speech) and the size of the output space (the number of intents). In this way, in order to improve the performance of the chatbot and the user experience with the chatbot, it may be necessary to train, monitor, debug, and retrain the chatbot. In traditional systems, a training system is provided for training and retraining the machine learning model of a digital assistant or chatbot in spoken language understanding (SLU) and natural language processing (NLP). Traditionally, the model for the chatbot system is trained in NLP using speech that is "made" for any intention. For example, the speech "Do you do price changes?" can be used to train the classifier model of the chatbot system to classify this type of speech as intent - "Do you offer a price match?" Using manufactured speech to train the model helps to initially train the chatbot system to provide a service, and then once the chatbot system is deployed and begins to obtain real speech from users, the chatbot system can be retrained.

[0032] Traditionally, training text classification models begins with pre-labeled data. One option for obtaining labeled data is to manually annotate all data by manually reading each text and categorizing it (for example, classifying utterances into intents). Another option is to outsource this labeling to freelancers or companies dedicated to data labeling. Either option is expensive in terms of time and money. To overcome this problem in similar technical spaces such as image analysis, data augmentation has been used to increase the size of image datasets for image classification tasks and accelerate the labeling process. While natural augmentation primitives exist for images, such as rotations, small translational shifts, and additive Gaussian noise, similar primitives are less well developed for text data. For example, in text-based data augmentation, the input data should be altered while maintaining the classification label. If the augmentation method significantly changes the meaning of an utterance (for example, a sentence or phrase), the original classification label may no longer be valid. Because long sentences or phrases contain more words than short ones, these types of text can absorb more noise while maintaining their original classification label. Therefore, traditional augmentation primitives developed for text focus on the presence of words in each sentence (e.g., synonym replacement, random insertion of synonyms, random swapping, random deletion, etc.) and the length of the sentence or phrase to maintain the original classification label of the sentence or phrase. However, these approaches can be difficult to implement for enterprise systems that use many chatbot systems, which are trained for many different tasks in multiple languages and receive a wide variety of utterances (many of which are very short utterances) for each task. In systems that use chatbots such as these, there is a need to automatically apply augmentation in a composition-agnostic manner to quickly and efficiently generate large corpora of training data in multiple languages for many different chatbots.

[0033] Therefore, different approaches are needed to address these problems. In various embodiments, a method is provided, comprising: receiving a training text dataset for an utterance or message; and augmenting the training text dataset with noise text to obtain an augmented training text dataset. The augmenting comprises: (i) obtaining noise text unrelated to the utterance or message from a word list, a text corpus, a publication, a dictionary, or any combination thereof, and (ii) merging the noise text into the training text dataset at a predefined augmentation ratio relative to the utterance or message. The computer-implemented method further comprises using the augmented training text dataset to train a predictive model to determine the likelihood that the utterance or message represents a task that the skill robot is configured to perform or to match the utterance or message to an intent associated with the skill robot.

[0034] Robotics and analytical systems

[0035] A bot (also known as a skill, chatbot, conversational bot, or conversational robot) is a computer program that can conduct a conversation with an end user. Bots can typically respond to natural language messages (e.g., questions or comments) via a messaging application that uses natural language messages. Enterprises can use one or more bot systems to communicate with end users via messaging applications. The messaging application (which may be referred to as a channel) can be a preferred messaging application that the end user already has installed and is familiar with. Therefore, in order to chat with the bot system, the end user does not need to download and install a new application. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), native or hybrid mobile and web application extensions / responsive mobile or web applications with chat functionality, or voice-based input (such as devices or applications with Siri, Microsoft Cortana, Google Voice, or other voice input for interaction).

[0036] In some examples, a robot system can be associated with a uniform resource identifier (URI). A URI can identify a robot system using a string of characters. A URI can be used as a webhook for one or more messaging application systems. A URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). A robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. An HTTP post call message can be directed to a URI from a messaging application system. In some embodiments, the message can be different from an HTTP post call message. For example, a robot system can receive a message from a short message service (SMS). Although the discussion herein may refer to communications received by a robot system as messages, it should be understood that a message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0037] End users can interact with the robot system through conversational interactions (sometimes called conversational user interfaces (UIs)), just as people interact with each other. In some cases, the interaction can include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking the end user how the robot can help. In some cases, the interaction can also be transactional, such as with a banking robot, such as transferring money from one account to another; informational, such as with an HR robot, such as checking a vacation balance; or interactive, such as with a retail robot, such as discussing returning a purchase or seeking technical support.

[0038] In some embodiments, a robotic system can intelligently handle end-user interactions without requiring interaction with an administrator or developer of the robotic system. For example, an end-user can send one or more messages to a robotic system in order to achieve a desired goal. Messages can include content such as text, emoticons, audio, images, video, or other methods of conveying messages. In some embodiments, the robotic system can convert the content into a standardized form (e.g., a Representational State Transfer (REST) call to an enterprise service with appropriate parameters) and generate a natural language response. The robotic system can also prompt the end-user to enter additional input parameters or request other additional information. In some embodiments, the robotic system can also initiate communication with the end-user rather than passively responding to end-user utterances. Described herein are various techniques for identifying explicit calls to a robotic system and determining the input of the called robotic system. In some embodiments, explicit call analysis is performed by a master robot based on detection of a call name in an utterance. In response to detecting the call name, the utterance can be refined for input to a skill robot associated with the call name.

[0039] A conversation with a robot can follow a specific conversation flow that includes multiple states. The flow can define what happens next based on input. In some embodiments, a state machine that includes user-defined states (e.g., end-user intent) and actions to be taken in or between states can be used to implement the robot system. The conversation can take different paths based on the end-user input, which may affect the decisions made by the robot regarding the flow. For example, in each state, based on the end-user input or utterance, the robot can determine the end-user's intent in order to determine the next appropriate action to take. As used herein and in the context of utterances, the term "intent" refers to the intention of the user providing the utterance. For example, a user may intend for the robot to engage in a conversation for ordering pizza, such that the user's intent can be expressed by the utterance "Order pizza". User intent can relate to a specific task that the user wants the chatbot to perform on the user's behalf. Therefore, utterances can be expressed as questions, commands, requests, etc. that reflect the user's intent. Intent can include a goal that the end-user wants to accomplish.

[0040] In the context of chatbot configuration, the term "intent" as used herein refers to configuration information used to map user utterances to specific tasks / actions or specific types of tasks / actions that the chatbot can perform. To distinguish the intent of an utterance (i.e., user intent) from the chatbot's intent, the latter is sometimes referred to herein as a "bot intent." A bot intent can include a set of one or more utterances associated with the intent. For example, an intent to order pizza can include various permutations of utterances expressing a desire to place an order for pizza. These associated utterances can be used to train the chatbot's intent classifier so that the intent classifier can subsequently determine whether an input utterance from the user matches the intent to order pizza. A bot intent can be associated with one or more dialog flows used to initiate a conversation with the user in a certain state. For example, the first message for an intent to order pizza could be the question "What kind of pizza would you like?" In addition to the associated utterances, a bot intent can further include named entities related to the intent. For example, an intent to order pizza can include variables or parameters for performing the task of ordering pizza, such as toppings 1, toppings 2, pizza type, pizza size, and number of pizzas. The values of the entities are typically acquired through conversation with the user.

[0041] Figure 1is a simplified block diagram of an environment 100 incorporating a chatbot system according to some embodiments. The environment 100 includes a Digital Assistant Builder Platform (DABP) 102 that enables users of the DABP 102 to create and deploy digital assistants or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, Figure 1 As shown, a user 104 representing a particular business can use DABP 102 to create and deploy a digital assistant 106 for the users of the particular business. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0042] For purposes of this disclosure, a "digital assistant" is an entity that assists a user of the digital assistant in completing various tasks through natural language conversation. A digital assistant may be implemented using software alone (e.g., a digital assistant is a digital entity implemented using a program, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, and the like. A digital assistant is also sometimes referred to as a chatbot system. Therefore, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0043] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of a conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and receive responses 112 in return from the digital assistant 106. The conversation can include one or more of the inputs 110 and the responses 112. Through these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested tasks and respond to the user with an appropriate response.

[0044] User input 110 is typically in the form of a natural language and is referred to as an utterance. User utterances can be in textual form, such as when a user types a sentence, a question, a text snippet, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user utterances can be in the form of audio input or speech, such as when a user speaks or says something as input to the digital assistant 106. The utterances are typically in the form of a language spoken by the user 108. For example, the utterances can be in English or some other language. When the utterances are in speech form, the speech input is converted into an utterance in textual form in that particular language, and the text utterances are then processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert the speech or audio input into textual utterances, which are then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion can be performed by the digital assistant 106 itself.

[0045] An utterance (which may be a text utterance or a voice utterance) may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the above types, and the like. The digital assistant 106 is configured to apply natural language understanding (NLU) technology to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing for understanding the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that these utterances are text utterances that have been directly provided by the user 108 of the digital assistant 106, or are the result of converting the input voice utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0046] For example, the input of user 108 can request to order a pizza by providing an utterance such as “I want to order a pizza”. After receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. Appropriate action can involve, for example, responding to the user with questions requesting user input regarding the type of pizza the user desires to order, the size of the pizza, any toppings on the pizza, etc. The responses provided by the digital assistant 106 can also be in natural language form, and are typically in the same language as the input utterance. As part of generating these responses, the digital assistant 106 can perform natural language generation (NLG). In order for the user to order a pizza via a conversation between the user and the digital assistant 106, the digital assistant can guide the user to provide all the necessary information for pizza ordering, and then have the pizza ordered at the end of the conversation. The digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0047] At a conceptual level, the digital assistant 106 performs various processing in response to an utterance received from the user. In some embodiments, the processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU), determining an action to be performed in response to the utterance, causing the action to be performed if appropriate, generating a response to be output to the user in response to the user utterance, and outputting the response to the user. NLU processing may include parsing the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a more understandable form (e.g., a logical form) or structure of the utterance. Generating the response may include using NLG techniques.

[0048] The NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related processing such as sentence grammatical analysis (e.g., tokenization, classification by inflectional form, identification of part-of-speech tags for sentences, identification of named entities in sentences, generation of dependency trees to represent sentence structure, division of sentences into clauses, analysis of separate clauses, resolution of references, execution of chunks, etc.). In some embodiments, the NLU processing or part thereof is performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 may use other resources to perform part of the NLU processing. For example, the syntax and structure of the input utterance sentence can be identified by processing the sentence using a grammatical analyzer, a part-of-speech tagger, and / or a named entity recognizer. In one embodiment, for the English language, a grammatical analyzer, a part-of-speech tagger, and a named entity recognizer provided by the Stanford Natural Language Processing (NLP) group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0049] Although the various examples provided in this disclosure show utterances in the English language, this is meant as an example only. In some embodiments, the digital assistant 106 is also capable of handling utterances in languages other than English. The digital assistant 106 can provide subsystems (e.g., components that implement NLU functionality) that are configured to perform processing for different languages. These subsystems can be implemented as pluggable units that can be called from the NLU core server using service calls. This makes NLU processing flexible and extensible for each language, including allowing different processing orders. Language packs can be provided for individual languages, wherein the language pack can register a list of subsystems that can be served from the NLU core server.

[0050] The digital assistant (e.g., a computer) may be made available through a variety of channels, such as, but not limited to, via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. Figure 1106) is available or accessible to its user 108. A single digital assistant can configure several channels for itself, so that a single digital assistant can run on and be accessed through different services at the same time.

[0051] A digital assistant or chatbot system typically includes or is associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill bots) that are configured to interact with a user and complete specific types of tasks (such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc.). For example, Figure 1 In the depicted embodiment, the digital assistant 106 or chatbot system includes skill bots 116 - 1 , 116 - 2 , etc. For purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skill bot” and “skill bots,” respectively.

[0052] Each skill associated with the digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, where the conversation can include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses can be in the form of text or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for selection.

[0053] There are various ways to associate skills or skill bots with or add them to a digital assistant. In some instances, a skill bot can be developed by an enterprise and then added to a digital assistant using DABP 102. In other instances, a skill bot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In still other instances, DABP 102 provides an online digital store (referred to as a "skill store") that offers a plurality of skills related to a variety of tasks. The skills offered through the skill store can also be exposed to various cloud services. To add a skill to a digital assistant generated using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and instruct that the selected skill be added to the digital assistant created using DABP 102. Skills from the skill store can be added to the digital assistant as is or in modified form (e.g., a user of DABP 102 can select and copy a particular skill bot provided by the skill store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0054] A digital assistant or chatbot system can be implemented using a variety of different architectures. For example, in some embodiments, a digital assistant created and deployed using DABP 102 can be implemented using a master robot / secondary (or child) robot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more secondary robots that are skill robots. For example, in Figure 1 In the depicted embodiment, the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are secondary robots to the master robot 114. In some embodiments, the digital assistant 106 itself is considered to act as the master robot.

[0055] A digital assistant implemented according to a master-secondary bot architecture enables users of the digital assistant to interact with multiple skills through a unified user interface (i.e., via a master bot). When a user engages with the digital assistant, the master bot receives the user input. The master bot then performs processing to determine the meaning of the user's input utterance. The master bot then determines whether the task requested in the user's utterance can be handled by the master bot itself. If not, the master bot selects an appropriate skill bot to handle the user's request and routes the conversation to the selected skill bot. This enables users to engage in a conversation with the digital assistant through a common, single interface while still having the ability to use several skill bots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the digital assistant's master bot can interface with skill bots with specific functions, such as a CRM bot for performing functions related to customer relationship management (CRM), an ERP bot for performing functions related to enterprise resource planning (ERP), an HCM bot for performing functions related to human capital management (HCM), and so on. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through the common master bot interface, while multiple skill bots are provided in the background to handle user requests.

[0056] In some embodiments, in a master robot / secondary robot infrastructure, the master robot is configured to understand a list of available skill robots. The master robot has access to metadata that identifies the various available skill robots, and for each skill robot, the skill robot's capabilities include tasks that can be performed by the skill robot. After receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot from multiple available skill robots that can best serve or handle the user request. The master robot then routes the utterance (or part of the utterance) to the specific skill robot for further handling. Thus, control flows from the master robot to the skill robots. The master robot can support multiple input channels and output channels.

[0057] Although Figure 1 The embodiment shown in FIG. 1 shows the digital assistant 106 including a master robot 114 and skill robots 116 - 1 , 116 - 2 , and 116 - 3 , but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations that use a combination of software and hardware.

[0058] DABP 102 provides the infrastructure, services, and features that enable users of DABP 102 to create digital assistants (including one or more skill bots associated with the digital assistant). In some instances, skill bots can be created by cloning existing skill bots, such as those provided by a skill store. As previously described, DABP 102 provides a skill store or skill catalog that provides multiple skill bots for performing various tasks. Users of DABP 102 can clone skill bots from the skill store. Cloned skill bots can then be modified or customized as needed. In other instances, users of DABP 102 create skill bots from scratch using the tools and services provided by DABP 102. As previously described, the skill store or skill catalog provided by DABP 102 provides multiple skill bots for performing various tasks.

[0059] In some embodiments, at a high level, creating or customizing a skill bot involves the following steps:

[0060] (1) Configuring settings for new skill bots

[0061] (2) Configure one or more intents for the skill robot

[0062] (3) Configure one or more entities for one or more intents

[0063] (4) Training skill robots

[0064] (5) Create a dialogue flow for the skill bot

[0065] (6) Add custom parts to skill robots as needed

[0066] (7) Test and deploy skill robots

[0067] Each of these steps is briefly described below.

[0068] (1) Configuring Settings for a New Skill Bot - Various settings can be configured for a skill bot. For example, a skill bot designer can specify one or more invocation names for the skill bot being created. These invocation names can then be used by users of the digital assistant to explicitly invoke the skill bot. For example, a user can enter an invocation name in a user utterance to explicitly invoke the corresponding skill bot.

[0069] (2) Configuring one or more intents and associated example utterances for the skill robot - The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classifications that the skill robot is trained to infer for the input utterance. After receiving the utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes appropriate actions to respond to the utterance based on the intent inferred for the utterance. In some instances, the intent of the skill robot represents the tasks that the skill robot can perform for the user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot may include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0070] For each intent defined for a skill bot, the skill bot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent the utterances that a user might enter into the skill bot for that intent. For example, for the CheckBalance intent, example utterances might include “What’s my savings account balance?”, “How much is in my checking account?”, “How much money do I have in my account?”, etc. Thus, various permutations of typical user utterances can be specified as example utterances for an intent.

[0071] These intents and their associated example utterances are used as training data for training the skill robot. A variety of different training techniques can be used. As a result of this training, a prediction model is generated that is configured to take an utterance as input and output the intent that the prediction model infers for the utterance. In some instances, the input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer the intent of the input utterance. The skill robot can then take one or more actions based on the inferred intent.

[0072] (3) Configuring entities for one or more intents of a skill bot - In some instances, additional context may be needed to enable a skill bot to respond appropriately to a user utterance. For example, there may be situations where user input utterances resolve to the same intent in a skill bot. For example, in the above example, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same intent of CheckBalance, but these utterances are different requests asking for different things. To clarify such requests, one or more entities are added to the intent. Taking the banking skill bot as an example, an entity called AccountType (which defines values called "checking" and "saving") can enable the skill bot to parse the user request and respond appropriately. In the above example, although these utterances resolve to the same intent, the values associated with the AccountType entity are different for the two utterances. This enables the skill bot to perform potentially different actions for the two utterances, even though the two utterances resolve to the same intent. One or more entities can be specified for certain intents configured for a skill bot. Thus, entities are used to add context to the intent itself. Entities help more fully describe intent and enable skill bots to complete user requests.

[0073] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (b) custom entities that can be specified by the skill bot designer. Built-in entities are general entities that can be used with various bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by the skill bot designer to implement various banking transactions by checking for keywords entered by the user (e.g., checking account, savings account, credit card, etc.).

[0074] (4) Training a Skill Bot - A skill bot is configured to receive user input in the form of utterances, parse or otherwise process the received input and identify or select an intent associated with the received user input. As indicated above, to do this, the skill bot must be trained. In some embodiments, the skill bot is trained based on the intents configured for the skill bot and example utterances associated with the intents (collectively referred to as training data) so that the skill bot can parse the user input utterance into one of the intents configured for it. In some embodiments, the skill bot uses a predictive model that is trained using the training data and allows the skill bot to discern what the user said (or in some cases, is attempting to say). DABP 102 provides a variety of different training techniques that can be used by skill bot designers to train skill bots, including various machine learning-based training techniques, rule-based training techniques and / or combinations thereof. In some embodiments, a portion (e.g., 80%) of the training data is used to train the skill bot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as a trained skill bot) can be used to process and respond to user utterances. In some cases, the user's utterance may be a question that requires only a single answer and no additional conversation is required. To handle this situation, a Q&A intent can be defined for the skill bot. This enables the skill bot to output a response to the user's request without having to update the dialog definition. Q&A intents are created in a similar manner to regular intents. The dialog flow for Q&A intents may be different from the dialog flow for regular intents.

[0075] (5) Create a dialogue flow for the skill bot - The dialogue flow specified for the skill bot describes how the skill bot reacts when parsing the different intents of the skill bot in response to the received user input. The dialogue flow defines the actions or behaviors that the skill bot will take, such as how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. The dialogue flow is like the process followed by the skill bot. Figure 1 Skill bot designers specify conversation flows using a language such as Markdown. In some embodiments, a version of YAML called OBotML can be used to specify a skill bot's conversation flow. The conversation flow definition for a skill bot serves as a model for the conversation itself, enabling skill bot designers to orchestrate interactions between the skill bot and the user it serves.

[0076] In some embodiments, the dialog flow definition for a skill bot includes the following three parts:

[0077] (a) Context

[0078] (b) Default Transformation Section

[0079] (c) Status section

[0080] Context section - Skill bot designers can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that enable skill bots to recognize and save user preferences.

[0081] Default Transitions Section - A skill bot's transitions can be defined in either the Dialog Flow State section or the Default Transitions section. Transitions defined in the Default Transitions section act as fallbacks and are triggered when no applicable transition is defined within a state or when the conditions required to trigger a state transition are not met. The Default Transitions section can be used to define routes that allow a skill bot to gracefully handle unexpected user actions.

[0082] State Part - A dialog flow and its associated operations are defined as a sequence of transient states that govern the logic within the dialog flow. Each state node within a dialog flow definition names a component that provides the functionality required at that point in the dialog. Thus, states are structured around components. States contain properties specific to a component and define transitions to other states that are triggered after the component executes.

[0083] Special case scenarios can be handled using the state component. For example, you might sometimes want to provide the user with the option to temporarily let the first skill they're engaged with do something in a second skill within the digital assistant. For example, if the user is busy in a conversation with a shopping skill (e.g., the user has already made some purchases), the user might want to jump to a banking skill (e.g., the user might want to make sure they have enough money for a purchase) and then return to the shopping skill to complete the user's order. To address this, an action in the first skill can be configured to initiate an interaction with a second, different skill within the same digital assistant and then return to the original flow.

[0084] (6) Adding Custom Components to a Skill Bot - As described above, the states specified in the skill bot's conversation flow name the components that provide the required functionality for the states. Components enable the skill bot to perform functions. In some embodiments, DABP 102 provides a set of preconfigured components for performing a variety of functions. The skill bot designer can select one or more of these preconfigured components and associate them with a state in the skill bot's conversation flow. The skill bot designer can also use the tools provided by DABP 102 to create custom components or new components and associate the custom components with one or more states in the skill bot's conversation flow.

[0085] (7) Testing and Deploying Skill Bots - DABP 102 provides several features that enable skill bot designers to test the skill bots they are developing. The skill bots can then be deployed and included in the digital assistant.

[0086] While the above description describes how to create a skill robot, similar techniques can also be used to create a digital assistant (or master robot). At the master robot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master robot) can handle without invoking the skill robot associated with the digital assistant. Examples of system intents defined for the master robot include: (1) Exit: applies when the user signals that they want to exit the current session or context in the digital assistant; (2) Help: applies when the user requests help or directions; and (3) UnresolvedIntent: applies to user input that does not quite match the exit intent and help intent. The digital assistant also stores information about one or more skill robots associated with the digital assistant. This information enables the master robot to select a specific skill robot for handling an utterance.

[0087] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and the associated conversation. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance is to be routed to a specific skill for handling, to be handled by the digital assistant or master robot itself according to built-in system intents, or to be handled into a different state in the current conversation flow.

[0088] In some embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill robot using its invocation name. If the invocation name is present in the user input, it can be treated as an explicit invocation of the skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill robot for further processing. In some embodiments, if no specific invocation or explicit invocation is present, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intents and skill robots associated with the digital assistant. The scores calculated for the skill robots or system intents indicate how likely it is that the user input represents the task that the skill robot is configured to perform or represents a system intent. Any system intent or skill robot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill robot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill robots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Any intent with a confidence score exceeding a threshold (e.g., 70%) is typically considered a candidate intent. If a specific skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, the main bot itself performs one or more actions based on the selected system intent.

[0089] Figure 2 2 is a simplified block diagram of a master robot (MB) system 200 according to certain embodiments. MB system 200 can be implemented solely in software, solely in hardware, or in a combination of hardware and software. MB system 200 includes a pre-processing subsystem 210, a multi-intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and data storage 250. Figure 2 The depicted MB system 200 is merely an example of a component arrangement in a master robot. One of ordinary skill in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have more than Figure 2 More or fewer systems or components than those shown may be used, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0090] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As described above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 can convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text.

[0091] Language detector 212 detects the language of utterance 202 based on the text of utterance 202. How utterance 202 is handled depends on the language, as each language has its own grammar and semantics. Differences between languages are taken into account when analyzing the syntax and structure of an utterance.

[0092] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for each linguistic unit (e.g., word) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., converting each word into a separate token) and classify words by inflectional form. A lemma is a primary form of a group of words as represented in a dictionary (e.g., "run" is a lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 may perform include chunking of compound expressions, e.g., combining "credit" and "card" into a single expression "credit_card". The language parser 214 may also identify relationships between words in the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a specific noun) is a direct object, which part of the utterance is a preposition, etc. The results of the processing performed by the language parser 214 form the extracted information 205 and are provided as input to the MIS 220 along with the utterance 202 itself.

[0093] As described above, utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in some embodiments, preprocessing may be performed, for example by preprocessing subsystem 210, to identify individual sentences from the multiple sentences for use in multiple intent analysis and explicit call analysis. Generally, MIS 220 and EIS 230 produce substantially the same results regardless of whether utterance 202 is processed at the level of individual sentences or as a single unit including multiple sentences.

[0094] MIS 220 determines whether utterance 202 indicates multiple intents. Although MIS 220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS 220 does not involve determining whether the intent of utterance 202 matches any intent that has been configured for the robot. Instead, the process of determining whether the intent of utterance 202 matches the robot's intent can be performed by intent classifier 242 of MB system 200 or (for example, Figure 3 The processing performed by MIS 220 assumes that there is a robot (e.g., a specific skill robot or the master robot itself) that can handle utterance 202. Therefore, the processing performed by MIS 220 does not need to know which robots are in the chatbot system (e.g., the identities of the skill robots registered with the master robot) or what intents have been configured for a particular robot.

[0095] To determine that an utterance 202 includes multiple intents, the MIS 220 applies one or more rules from a set of rules 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and may include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern may include a coordinating conjunction that connects two parts of a sentence (e.g., a conjunction), where the two parts correspond to different intents. If the utterance 202 matches the sentence pattern, it can be inferred that the utterance 202 represents multiple intents. It should be noted that an utterance with multiple intents does not necessarily have different intents (e.g., intents involving different robots or different intents within the same robot). Instead, an utterance can have different instances of the same intent, such as "Place a pizza order using payment account X, then place a pizza order using payment account Y."

[0096] As part of determining that the utterance 202 represents multiple intents, the MIS 220 also determines which parts of the utterance 202 are associated with each intent. The MIS 220 constructs a new utterance for separate processing for each intent represented in the utterance containing multiple intents to replace the original utterance, for example, Figure 2206 and utterance "C" 208. Thus, the original utterance 202 can be divided into two or more separate utterances, which are processed one at a time. The MIS 220 uses the extracted information 205 and / or based on an analysis of the utterance 202 itself to determine which of the two or more utterances should be processed first. For example, the MIS 220 may determine that the utterance 202 contains a marker word indicating that a particular intent should be processed first. The newly formed utterance corresponding to the particular intent (e.g., one of the utterances 206 or 208) will be sent first for further processing by the EIS 230. After the conversation triggered by the first utterance has ended (or has been temporarily paused), the next highest priority utterance (e.g., the other of the utterances 206 or 208) can then be sent to the EIS 230 for processing.

[0097] EIS230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) contains the invocation name of a skill robot. In some embodiments, each skill robot in the chatbot system is assigned a unique invocation name that distinguishes the skill robot from other skill robots in the chatbot system. A list of invocation names can be stored in data store 250 as part of skill robot information 254. When an utterance contains words that match the invocation name, the utterance is considered an explicit invocation. If the robot is not explicitly invoked, the utterance received by EIS230 is considered a non-explicit invocation utterance 234 and is input to the intent classifier of the master robot (e.g., intent classifier 242) to determine which robot to use to handle the utterance. In some instances, intent classifier 242 will determine that the master robot should handle the non-explicit invocation utterance. In other instances, intent classifier 242 will determine the skill robot to which to route the utterance for handling.

[0098] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that the master robot must perform. For example, when an explicit call is present, the master robot may not have to perform any intent classification analysis (e.g., using intent classifier 242), or may have to perform a simplified intent classification analysis to select a skill robot. Thus, explicit call analysis can enable the selection of a specific skill robot without resorting to intent classification analysis.

[0099] Furthermore, there may be cases where the functionality of multiple skill robots overlaps. This can occur, for example, if two skill robots handle intents that overlap or are very close to each other. In this case, it can be difficult for the master robot to identify which of the multiple skill robots to select based solely on intent classification analysis. In such scenarios, explicit invocation eliminates ambiguity regarding the specific skill robot to use.

[0100] In addition to determining whether an utterance is an explicit call, EIS 230 is also responsible for determining whether any portion of the utterance should be used as input to the explicitly called skill bot. Specifically, EIS 230 may determine whether a portion of the utterance is unrelated to the call. EIS 230 may perform this determination by analyzing the utterance and / or analyzing extracted information 205. Instead of sending the entire utterance received by EIS 230, EIS 230 may send the portion of the utterance unrelated to the call to the called skill bot. In some instances, the input to the called skill bot is simply formed by removing any portion of the utterance associated with the call. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using Pizza Bot" is related to the call to the pizza bot but not to any processing to be performed by the pizza bot. In some instances, EIS 230 may reformat the portion to be sent to the called bot, for example, to form a complete sentence. Thus, EIS 230 not only determines the presence of an explicit call but also determines what to send to the skill bot when an explicit call is present. In some instances, there may not be any text to be entered into the invoked bot. For example, if the utterance is "Pizza Bot," the EIS 230 may determine that the pizza bot is being invoked, but there is no text to be processed by the pizza bot. In such a scenario, the EIS 230 may indicate to the skill bot invoker 240 that there is no content to be sent.

[0101] The skill robot invoker 240 invokes a skill robot in various ways. For example, the skill robot invoker 240 may invoke the robot in response to receiving an indication 235 that a particular skill robot has been selected as a result of an explicit invocation. The indication 235 may be sent by the EIS 230 along with the input from the explicitly invoked skill robot. In this scenario, the skill robot invoker 240 hands control of the conversation to the explicitly invoked skill robot. The explicitly invoked skill robot will determine an appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0102] Another way that the skill robot invoker 240 can invoke a skill robot is implicitly by using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different classifications, one classification for each skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. The parameters generated as a result of this training (e.g., a set of parameter values for a machine learning model) can be stored as part of the skill robot information 254.

[0103] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training the machine learning model can involve inputting at least a subset of example utterances associated with various skill robots to generate, as output of the machine learning model, inferences about which robot is the correct robot for handling any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., via backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0104] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot that indicates the likelihood that the skill robot can handle an utterance (e.g., a non-explicit call utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each system-level intent that has been configured (e.g., help, exit). If a particular confidence score meets one or more conditions, the skill robot caller 240 will call the robot associated with that particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain margin. When the confidence scores of multiple skill robots all exceed the threshold confidence score value, applying such a condition will enable routing to a particular skill robot.

[0105] After identifying a robot based on the confidence score evaluation, the skill robot invoker 240 transfers processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is a skill robot. Further, the skill robot invoker 240 determines what to provide as input 247 to the identified robot. As described above, in the case of explicit invocation, the input 247 can be based on a portion of the utterance that is not relevant to the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of implicit invocation, the input 247 can be the entire utterance.

[0106] The data store 250 includes one or more computing devices that store data used by the various subsystems of the master robot system 200. The data store 250, as explained above, includes rules 252 and skill robot information 254. The rules 252 include, for example, rules for determining by the MIS 220 when an utterance represents multiple intents and how to segment utterances that represent multiple intents. The rules 252 further include rules for determining by the EIS 230 which parts of an utterance that explicitly calls a skill robot are to be sent to the skill robot. The skill robot information 254 includes the calling names of the skill robots in the chatbot system, for example, a list of the calling names of all skill robots registered with a particular master robot. The skill robot information 254 may also include information, such as parameters of a machine learning model, that the intent classifier 242 uses to determine a confidence score for each skill robot in the chatbot system.

[0107] Figure 3 is a simplified block diagram of a skill robot system 300 according to some embodiments. The skill robot system 300 is a computing system that can be implemented in software only, hardware only, or a combination of hardware and software. In some embodiments, such as Figure 1 In the depicted embodiment, the skill bot system 300 can be used to implement one or more skill bots within a digital assistant.

[0108] The skill robot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to Figure 2 The MIS 220 in FIG. 310 and provides similar functionality, including being operable to use rules 352 in the data store 350 to determine: (1) whether an utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by the MIS 310 for detecting multiple intents and for splitting utterances are the same as the rules applied by the MIS 220. The MIS 310 receives the utterance 302 and the extracted information 304. The extracted information 304 is similar to Figure 1The extracted information 205 in the skill robot system 300 can be generated using the language parser 214 or a language parser local to the skill robot system 300.

[0109] The intent classifier 320 can be combined with the above Figure 2 In some embodiments, the intent classifier 320 is trained in a similar manner to the intent classifier 242 discussed in detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained using at least a subset of example utterances associated with a particular skill bot as training utterances for the particular skill bot. The ground truth for each training utterance will be the particular bot intent associated with the training utterance.

[0110] The utterance 302 may be received directly from the user or provided by the master robot. When the utterance 302 is provided by the master robot, for example, as Figure 2 As a result of the processing by MIS 220 and EIS 230 in the embodiment depicted in , MIS 310 may be bypassed to avoid duplicating processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a conversation that occurs after being routed to a skill bot, MIS 310 may process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0111] The intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skill robot system 300. As explained above, a skill robot may be configured with one or more intents, each including at least one example utterance associated with the intent and used to train the classifier. Figure 2In an embodiment, the intent classifier 242 of the master robot system 200 is trained to determine confidence scores for individual skill robots and confidence scores for system intents. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and is therefore more fine-grained. The intent classifier 320 has access to intent information 354. For each intent associated with the skill robot system 300, the intent information 354 includes a list of utterances that represent the intent and explain the meaning of the intent and are typically associated with tasks that can be performed by the intent. The intent information 354 can further include parameters generated as a result of training on the utterance list.

[0112] The conversation manager 330 receives, as output from the intent classifier 320, an indication 322 of a particular intent that is identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some instances, the intent classifier 320 may not determine any match. For example, if the utterance relates to a system intent or an intent of a different skill bot, the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill bot system 300 may submit the utterance to the master bot for handling, e.g., for routing to a different skill bot. However, if the intent classifier 320 successfully identifies the intent within the skill bot, the conversation manager 330 will initiate a conversation with the user.

[0113] The session initiated by the session manager 330 is a session specific to the intent identified by the intent classifier 320. For example, the session manager 330 can be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine can include a default starting state (e.g., when the intent is invoked without any additional input) and one or more additional states, each of which is associated with an action to be performed by the skill bot (e.g., executing a purchase transaction) and / or a dialog to be presented to the user (e.g., a question, a response). Thus, the session manager 330 can determine an action / dialog 335 upon receiving an indication 322 that an intent was recognized, and can determine additional actions or dialogs in response to subsequent utterances received during the session.

[0114] The data storage 350 includes one or more computing devices that store data used by the various subsystems of the skill robot system 300. Figure 3 As depicted in , data store 350 may include rules 352 and intent information 354. In some embodiments, data store 350 may be integrated into the data store of a host robot or digital assistant, e.g. Figure 2Data storage 250 in.

[0115] Noisy data augmentation

[0116] It has been discovered that models for classifying utterances as intents provide poor results on text that contains irrelevant context (e.g., real utterances many times have context or irrelevant information that is irrelevant to the question asked by the user or the problem the user encounters). To overcome this problem, various embodiments are directed to techniques for augmenting training data with noise (e.g., adding irrelevant words around the original data) to make the model more resilient to irrelevant context. By augmenting the training data with random words around the original examples, the model becomes better at honing in on the most important parts of the examples to tie them to their classification. It was unexpectedly discovered that this also helps improve training performance on shorter test utterances that do not contain irrelevant context. Models trained on augmented text data can be implemented in chatbot systems, such as those related to Figure 1 、 Figure 2 and Figure 3 Advantageously, these models and chatbots perform better on utterances with extraneous context and regular utterances (without much extraneous context) because the models are better able to hone in on the important parts of the utterance. Furthermore, because the augmentations are automatically applied in a composition-agnostic manner, the customer or client does not need to be concerned with adding extraneous context to the training data.

[0117] Figure 4 The instructions are shown as being configured to train and utilize a classifier (e.g., about Figure 2 and Figure 3 242 or 320) of the chatbot system 400. Figure 4 As shown, the text classification performed by the chatbot system 400 in this example includes various stages: a prediction model training stage 410; a skill robot invocation stage 415, which is used to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform; and an intent prediction stage 420, which is used to classify an utterance into one or more intents. The prediction model training stage 410 establishes and trains one or more prediction models 425a-425n ('n' represents any natural number) for use by the other stages (the prediction models may be referred to herein individually as prediction models 425 or collectively as prediction models 425). For example, the prediction models 425 may include a model for determining the likelihood that an utterance represents a task that a particular skill robot is configured to perform, another model for predicting intent based on the utterances of a first type of skill robot, and another model for predicting intent based on the utterances of a second type of skill robot. Other types of prediction models may be implemented in other examples according to the present disclosure.

[0118] The prediction model 425 can be a machine learning ("ML") model, such as a convolutional neural network ("CNN"), for example, an initial neural network, a residual neural network ("Resnet"); or a recurrent neural network, such as a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model; other variants of a deep neural network ("DNN") (for example, a multi-label n-point DNN classifier or a multi-class DNN classifier for single intent classification. The prediction model 425 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, or a neural network. Model), shallow neural networks, or a combination of one or more of these techniques—for example, CNN-HMM or MCNN (Multi-Scale Convolutional Neural Network). The chatbot system 400 can employ the same type of prediction model or different types of prediction models to determine the likelihood of a task that a particular skill bot is configured to perform, predict intent based on utterances of a first type of skill bot, and predict intent based on utterances of a second type of skill bot. Other types of prediction models can be implemented in other examples according to the present disclosure.

[0119] To train the various predictive models 425, the training phase 410 consists of three main components: dataset preparation 430, feature engineering 435, and model training 440. Dataset preparation 430 includes the process of loading data assets 445, splitting the data assets 445 into training and validation sets 445a-n so that the system can train and test the predictive models 425, and performing basic preprocessing. Data assets 445 may include at least a subset of utterances from example utterances associated with various skill bots. As described above, utterances can be provided in various formats, including audio or text. Utterances can be sentence fragments, complete sentences, multiple sentences, and so on. For example, if utterances are provided as audio, dataset preparation 430 may convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text. In some instances, the example utterances are provided by the client or customer. In other instances, the example utterances are automatically generated from a library of previously collected utterances (e.g., identifying utterances from the library that are specific to the skill the chatbot is to learn). The data assets 445 of the prediction model 425 can include input text or audio (or input features of the text or audio frames) and labels 450 corresponding to the input text or audio (or input features) as a matrix or value table. For example, for each training utterance, an indication of the correct robot for the training utterance can be provided as the ground truth information of the label 450. The behavior of the prediction model 425 can then be adapted (e.g., by backpropagation) to minimize the difference between the generated inferences and the ground truth information. Alternatively, for a specific skill robot, the prediction model 425 can be trained using at least a subset of example utterances associated with the specific skill robot as training utterances. The ground truth information of the label 450 of each training utterance will be the specific robot intent associated with the training utterance.

[0120] In various embodiments, dataset preparation 430 includes performing noise augmentation 455 on data assets 445 to include noise (e.g., adding irrelevant words around the original data) to make prediction models 425 more resilient to irrelevant context. By augmenting data assets 445 with random words around the original examples, prediction models 425 become better at honing in on the most important parts of the examples, i.e., relating them to their classification. Augmentation 455 is performed using a noise augmentation technique to incorporate irrelevant text (noise text) into the original text of data assets 445. Noise augmentation techniques typically include five operations: (i) obtaining noise text that is irrelevant to the original text (in other words, the obtained noise text is not dependent on or based on the original text) from a word list, a text corpus (e.g., a large structured text collection), a publication, or a dictionary, and (ii) incorporating the noise text into each data asset 445 at a predefined augmentation ratio relative to the original text. For example, the noise text can be placed before the original text, after the original text, on both sides of the original text, or integrated with the original text at an expansion ratio of 1:0.5 to 1:5 (e.g., 1:1.2 or 1:2 - original text: noise text). The expansion ratio is selected to maintain the original classification label of the sentence or phrase.

[0121] One noise augmentation operation that can be used is to augment the original text with random text strings generated from a stop word list. Stop words are a set of common words in any language. For example, in English, "the," "is," and "and" can serve as stop words. In NLP and text mining applications, stop words are often removed to allow the algorithm to focus on important words. However, in this instance, stop words are introduced into the original text to create noise or irrelevant text, as shown in Table 1.

[0122] Table 1:

[0123] Stop word expansion example:

[0124]

[0125] In some instances, this noise augmentation operation is used for in-domain intents (i.e., within a set or group of intents specific to a particular vertical (such as ordering pizza)), and using stop words for in-domain intents means that it is less likely to cause false positives in these intents.

[0126] Another noise augmentation operation that can be used is to augment the original text with (verbatim) sentences from a corpus. The corpus may contain text in a single language (monolingual corpus) or text data in multiple languages (multilingual corpus). In some instances, the corpus is annotated with additional information, such as part-of-speech tags, where information about the part of speech (verb, noun, adjective, etc.) of each word is added to the corpus in the form of a tag. Sentences from the corpus are introduced verbatim (with exactly the same words used originally) into the original text to create noise or irrelevant text, as shown in Table 2.

[0127] Table 2:

[0128] Corpus (word by word) expansion example:

[0129]

[0130] This noise augmentation operation preserves the structure of the real sentence and provides more robust training because it is similar to what the model will receive as input during deployment.

[0131] Another noise augmentation operation that can be used is to augment the original text with (shuffled copies of) sentences from the corpus. Sentences from the corpus are shuffled (randomly or using an algorithm) and introduced into the original text to create noise or irrelevant text, as shown in Table 3.

[0132] Table 3:

[0133] Corpus (shuffle) expansion example:

[0134]

[0135] This noise augmentation operation preserves the distribution of words in real sentences, and the unstructured nature of the noise may mean that false positives are more difficult to occur.

[0136] Another noise augmentation operation that can be used is to augment the original text with random strings generated from a dictionary. Sentences from the dictionary are introduced into the original text without considering the frequency (i.e., without considering the number of repetitions or occurrences of words in the random strings) to create noise or irrelevant text, as shown in Table 4.

[0137] Table 4:

[0138] Example of dictionary expansion (regardless of frequency):

[0139]

[0140] In this noise augmentation operation, data is readily available and the unstructured nature of the noise can mean that false positives are more difficult to occur.

[0141] Another noise augmentation operation that can be used is to augment the original text with random strings generated from a dictionary. Sentences from the dictionary are introduced into the original text to create noise or irrelevant text while taking into account the frequency (i.e., taking into account the repetition or number of occurrences of words in the random strings), as shown in Table 5.

[0142] Table 5:

[0143] Example of dictionary expansion (taking frequency into account):

[0144]

[0145] In this noise augmentation operation, data is readily available, the distribution of words in the generated strings is more realistic, and the unstructured nature of the noise can mean that false positives are more difficult to occur.

[0146] With the introduction of noise augmentation technology, prediction model 425 performs better on utterances with irrelevant context and regular utterances (without much irrelevant context) because prediction model 425 is better able to hone in on important parts of the utterance. In addition, because noise augmentation is automatically applied in a composition-agnostic manner, the customer or client does not need to be concerned about adding irrelevant context to the training data. As shown in Table 6, the baseline multi-class model (without noise augmentation) either returns incorrect top-ranked intents or returns top-ranked intents with confidence scores below a support threshold (e.g., 0.7 or 70%), while the noise augmentation model is able to return the correct top-ranked intent and distinguish it from other intents (confidence scores>0.7 or 70%).

[0147] Table 6

[0148]

[0149]

[0150] As shown in Tables 7 and 8, on average, the confidence score of the top returned intent is higher in the noise augmentation model compared to the baseline model.

[0151] Table 7:

[0152] Table 8:

[0153]

[0154] In some instances, additional augmentation can be applied to the data assets 445 (using noise augmentation). For example, simple data augmentation (EDA) techniques can be used to improve the performance of text classification tasks. EDA includes four operations: synonym replacement, random insertion, random swap, and random deletion, which can prevent overfitting and help train more robust models. Note that, in contrast to noise augmentation, the EDA operation typically: (i) obtains words from the original text, and (ii) merges these words into each data asset 445 relative to the original text. For example, the synonym replacement operation includes randomly selecting n words that are not stop words from the original sentence (e.g., utterance) and replacing each of these words with one of its randomly selected synonyms. The random insertion operation includes—n times—finding a random synonym for a random word in the original sentence that is not a stop word and inserting the synonym into a random position in the sentence. The random swap operation includes—n times—randomly selecting two words in the sentence and swapping their positions. The random deletion operation includes randomly deleting each word in the sentence with probability p.

[0155] In various embodiments, feature engineering 435 includes transforming data assets 445 (augmented with noise) into feature vectors and / or creating new features to be created using data assets 445 (augmented with noise). Feature vectors can include count vectors as features, term frequency - inverse document frequency (TF-IDF) vectors as features (such as word level, n-gram level, or character level), word embeddings as features, text / NLP as features, topic models as features, or a combination thereof. A count vector is a matrix representation of data assets 445 where each row represents an utterance, each column represents a term from the utterance, and each cell represents a frequency count of a particular term in the utterance. The TF-IDF score represents the relative importance of a term in the utterance. Word embeddings are a form of representing words and utterances using a dense vector representation. The position of a word in the vector space is learned from the text and is based on the words that surround the word when it is used. Text / NLP-based features may include the number of words in an utterance, the number of characters in an utterance, the average word density, the number of punctuation marks, the number of capital letters, the number of title words, the frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof. Topic modeling is a technique for identifying groups of words (called topics) from a collection of utterances that contain the best information in the collection.

[0156] In various embodiments, model training 440 includes training a classifier using the feature vectors and / or new features created in feature engineering 435. In some instances, the training process includes an iterative operation to find a set of parameters for the prediction model 425 that minimizes a loss or error function for the prediction model 425. Each iteration can involve finding a set of parameters for the prediction model 425 such that the value of the loss or error function using that set of parameters is less than the value of the loss or error function using another set of parameters in the previous iteration. The loss or error function can be constructed to measure the difference between the output predicted using the prediction model 425 and the labels 450 contained in the data assets 445. Once the set of parameters is identified, the prediction model 425 has been trained and can be used for predictions as designed.

[0157] In addition to data assets 445, labels 450, feature vectors and / or new features, other techniques and information can be employed to refine the training process of the prediction model 425. For example, feature vectors and / or new features can be combined to help improve the accuracy of the classifier or model. Additionally or alternatively, hyperparameters can be adjusted or optimized, for example, multiple parameters such as tree length, leaves, network parameters, etc. can be fine-tuned to obtain a best fit model. Although the training mechanisms described herein are primarily focused on training the prediction model 425. These training mechanisms can also be used to fine-tune existing prediction models 425 trained from other data assets. For example, in some cases, the prediction model 425 may have been pre-trained using utterances specific to another skill robot. In these cases, the prediction model 425 can be retrained using the data assets 445 (using noise augmentation) as discussed herein.

[0158] The prediction model training phase 410 outputs trained prediction models 425, including a task prediction model 460 and an intent prediction model 465. The task prediction model 460 can be used in the skill bot invocation phase 415 to determine the likelihood that an utterance represents a task that a particular skill bot is configured to perform 470, and the intent prediction model 465 can be used in the intent prediction phase 420 to classify an utterance into one or more intents 475. In some instances, the skill bot invocation phase 415 and the intent prediction phase 420 can be performed independently by separate models in some examples. For example, the trained intent prediction model 465 can be used in the intent prediction phase 420 to predict the intent of a skill bot without first identifying the skill bot in the skill bot invocation phase 415. Similarly, the task prediction model 460 can be used in the skill bot invocation phase 415 to predict the task or skill bot to be used for an utterance without identifying the intent of the utterance in the intent prediction phase 420.

[0159] Alternatively, the skill bot invocation phase 415 and the intent prediction phase 420 can be performed sequentially, with one phase using the output of the other phase as input, or with one phase being invoked in a particular manner based on the output of the other phase for a particular skill bot. For example, for given text data 405, a skill bot invoker can invoke a skill bot via an implicit call using the skill bot invocation phase 415 and the task prediction model 460. The task prediction model 460 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill bot 470 is configured to perform. Then, for the identified or invoked skill bot and given text data 405, the intent prediction phase 420 and the intent prediction model 465 can be used to match received utterances (e.g., utterances within a given data asset 445) with intents 475 associated with the skill bot. As explained herein, a skill bot can be configured with one or more intents, each intent comprising at least one example utterance associated with the intent and used to train a classifier. In some embodiments, the skill robot invocation stage 415 and the task prediction model 460 used in the master robot system are trained to determine confidence scores for individual skill robots and confidence scores for system intents. Similarly, the intent prediction stage 420 and the intent prediction model 465 can be trained to determine confidence scores for each intent associated with the skill robot system. The classification performed by the skill robot invocation stage 415 and the task prediction model 460 is at the robot level, while the classification performed by the intent prediction stage 420 and the intent prediction model 465 is at the intent level and is therefore more fine-grained.

[0160] Techniques for noisy data augmentation and utterance classification

[0161] Figure 5 is a flow chart illustrating a process 500 of augmenting a training dataset with noise, according to certain embodiments. Figure 5 The depicted processes may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 5 The methods presented and described below are intended to be illustrative and not limiting. Figure 5 The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order or some steps may be performed in parallel. In some embodiments, such as Figures 1 to 4 In the embodiment depicted in Figure 5The processing depicted in can be performed by a preprocessing subsystem (e.g., preprocessing subsystem 210 or prediction model training stage 410) to generate a noise-augmented dataset for training by one or more prediction models (e.g., intent classifier 242 or 320 or prediction model 425).

[0162] At 505, a data processing system (e.g., Figure 4 The chatbot system 400 described above receives a training utterance set. In some instances, the training utterance set is used to train an intent classifier to identify one or more intents of one or more utterances. At 510, the training utterance set is augmented by a data processing system with noise text to obtain an augmented training utterance set. In various embodiments, the augmentation includes: (i) obtaining noise text that is unrelated to the original text in the utterances of the training utterance set from a word list, a text corpus, a publication, a dictionary, or any combination thereof, and (ii) merging the noise text into the utterances at a predefined augmentation ratio relative to the original text in the utterances of the training utterance set to generate augmented utterances. The noise text can be placed at a predefined augmentation ratio of 1:0.5 to 1:5 in the following manner: (i) before the original text in the utterance, (ii) after the original text in the utterance, (iii) on both sides of the original text in the utterance, (iv) integrated into the original text in the utterance, (v) or a combination of the foregoing. The noise text can be a random text string automatically generated from a stop word list. The noise text can be a text sentence automatically copied verbatim from a corpus. Noise text can be a random text string automatically generated from a corpus. Noise text can be a random text string automatically generated from a dictionary, where the frequency of the words or characters chosen for the random text string is not taken into account. Noise text can be a random text string automatically generated from a dictionary, where the frequency of the words or characters chosen for the random text string is taken into account.

[0163] At 515, a prediction model is trained using the augmented training text dataset to determine the likelihood that an utterance or message represents a task that a skill bot is configured to perform or to match an utterance or message to an intent associated with a skill bot. Thereafter, at 520, the trained prediction model can be deployed within a chatbot system (e.g., as part of a skill bot) to determine the likelihood that an utterance or message represents a task that a skill bot is configured to perform or to match an utterance or message to an intent associated with a skill bot. For example, an utterance can be received and the utterance can be analyzed to determine whether the utterance contains an invocation name for a skill bot. If an invocation name is not found, the utterance is considered a non-explicit invocation and the process continues using an intent classifier (e.g., the trained model). If a invocation name is determined to be present, the utterance is considered an explicit invocation and the process continues to determine which portions of the utterance are associated with the invocation name.

[0164] In the instance of calling the trained model, the entire utterance received is provided as input to the intent classifier. The intent classifier that receives the utterance can be the intent classifier of the master robot (e.g., Figure 2 Intent classifier 242 in ). The intent classifier can be a machine learning based or rule-based classifier that is trained using noise-augmented data to determine whether the intent of the utterance matches a system intent (e.g., exit, help) or a specific skill bot. As explained herein, the intent analysis performed by the master robot can be limited to matching with a specific skill bot without determining which intent within a specific skill bot is the best match for the utterance. Thus, the intent classifier that receives the utterance can identify the specific skill bot to invoke. Alternatively, if the utterance represents a specific system intent (e.g., the utterance contains the word "exit" or "help"), then the intent classifier that receives the utterance can identify the specific system intent to trigger a conversation between the master robot and the user based on the dialog flow configured for that specific system intent.

[0165] In instances where there is a call name, one or more explicit call rules are applied to determine which parts of the utterance are associated with the call name. This determination can be based on an analysis of the sentence structure of the utterance using POS tags, dependency information and / or other extracted information received with the utterance. For example, the part associated with the call name can be a noun phrase that includes the call name or a prepositional object corresponding to the call name. Any part associated with the call name as determined based on processing will be deleted. Other parts of the utterance that are not needed to convey the meaning of the utterance (e.g., prepositions) can also be deleted. Deleting parts of the utterance generates input for the skill robot associated with the call name. If any part of the received utterance still remains after the deletion, the remaining part forms a new utterance to be input to the skill robot, for example, as a text string. Otherwise, if the received utterance is completely deleted, the input can be an empty string.

[0166] Thereafter, the skill bot associated with the invocation name is invoked and the generated input is provided to the skill bot. Upon receiving the generated input, the invoked skill bot processes the input, for example, by performing intent analysis using the skill bot's intent classifier, trained with noise-augmented data, to identify a bot intent that matches the user intent represented in the input. Identification of a matching bot intent can cause the skill bot to perform a specific action or initiate a conversation with the user according to the dialog flow associated with the matching bot intent. For example, if the input is an empty string, the conversation can begin in a default state defined for the dialog flow, such as a welcome message. Alternatively, if the input is not an empty string, the conversation can begin in an intermediate state, for example, because the input contains the value of an entity or some other information that the skill bot no longer needs to query the user for because it has already received this information as part of the input. As another example, the skill bot may determine that it cannot process the input (for example, because the confidence score for each bot intent configured for the skill bot is below a certain threshold). In this case, the skill bot can return the input to the master bot for processing (for example, using the master bot's intent classifier for intent analysis), or the skill bot can prompt the user for clarification.

[0167] In various embodiments, deploying and using an intent classifier within a chatbot system includes receiving, by the chatbot system, an utterance generated by a user interacting with the chatbot system; using an intent classifier deployed within the chatbot system, classifying the utterance into an intent category corresponding to an intent; and outputting the intent based on the classification using the intent classifier. The intent classifier includes a plurality of model parameters identified using training data, the training data including: an augmented training utterance set for training the intent classifier to identify one or more intents of one or more utterances, wherein the augmented training utterance set is artificially generated to include augmented utterances from the training utterance set, wherein noise text is merged into the utterances from the training utterance set at a predefined augmentation ratio relative to the original text in the utterance to generate the augmented utterance. The plurality of model parameters are identified using the training data based on minimizing a loss function.

[0168] Descriptive System

[0169] Figure 6 Depicted is a simplified diagram of a distributed system 600. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608 coupled to a server 612 via one or more communication networks 610. The client computing devices 602, 604, 606, and 608 may be configured to execute one or more applications.

[0170] In various examples, server 612 can be adapted to run one or more services or software applications that implement one or more embodiments described in the present disclosure. In some examples, server 612 can also provide other services or software applications that can include non-virtual environments and virtual environments. In some examples, these services can be provided to users of client computing devices 602, 604, 606, and / or 608 as web-based services or cloud services (such as under a software-as-a-service (SaaS) model). Users operating client computing devices 602, 604, 606, and / or 608 can then utilize one or more client applications to interact with server 612 to utilize the services provided by these components.

[0171] exist Figure 6 In the depicted configuration, the server 612 may include one or more components 618, 620, and 622 that implement the functions performed by the server 612. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be understood that a variety of different system configurations are possible that may differ from the distributed system 600. Thus, Figure 6 The illustrated example is one example of a distributed system for implementing the example system and is not intended to be limiting.

[0172] The user may use client computing devices 602, 604, 606, and / or 608 to execute one or more applications, models, or chatbots that may generate one or more events or models that may then be implemented or serviced according to the teachings of the present disclosure. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although Figure 6 Only four client computing devices are depicted, but any number of client computing devices can be supported.

[0173] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome TM OS), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM 、 Palm Portable handheld devices may include cellular phones, smart phones (e.g., ), tablet computers (e.g. ), personal digital assistants (PDAs), etc. Wearable devices may include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Microsoft Gesture Input Devices Game consoles, Sony system, by Various game systems provided, and others), etc. The client device may be capable of executing various different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications) and may use various communication protocols.

[0174] The network(s) 610 may be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), By way of example only, the network(s) 610 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., according to the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, and / or any other wireless protocol) and / or any combination of these networks and / or other networks.

[0175] The server 612 may be composed of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, Servers 612 may include servers, mid-range servers, mainframe computers, rack servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. Server 612 may include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization, such as logical storage devices that may be virtualized to maintain one or more flexible pools of virtual storage devices for the servers. In various examples, server 612 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0176] The computing systems in the servers 612 may run one or more operating systems, including any of those discussed above and any commercially available server operating systems. The servers 612 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transport Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines Corporation) and others.

[0177] In some implementations, server 612 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. By way of example, data feeds and / or event updates may include, but are not limited to: feed, The server 612 may also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client computing devices 602 , 604 , 606 , and 608 .

[0178] The distributed system 600 may also include one or more data repositories 614, 616. In some examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 614, 616 may be used to store information (such as information about chatbot performance or generated models) for use by the chatbot used by the server 612 when performing various functions according to various embodiments. The data repositories 614, 616 may reside in various locations. For example, the data repository used by the server 612 may be local to the server 612 or may be remote from the server 612 and communicate with the server 612 via a network-based or dedicated connection. The data repositories 614, 616 may be of different types. In some examples, the data repository used by the server 612 may be a database, such as a relational database, such as an Oracle database. and databases provided by other suppliers. One or more of these databases can be adapted to respond to SQL format commands to implement storage, update and retrieval of data into and from the database.

[0179] In some examples, one or more of data repositories 614, 616 may also be used by applications to store application data. The data repositories used by applications may be of different types, such as, for example, key-value stores, object stores, or general-purpose storage stores backed by a file system.

[0180] In some examples, the functionality described in this disclosure can be provided as a service via a cloud environment. Figure 7 is a simplified block diagram of a cloud-based system environment in which various services can be provided as cloud services according to certain examples. Figure 7 In the depicted example, cloud infrastructure system 702 can provide one or more cloud services that can be requested by users using one or more client computing devices 704, 706, and 708. Cloud infrastructure system 702 can include one or more computers and / or servers, which can include those described above with respect to server 612. The computers in cloud infrastructure system 702 can be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0181] Network(s) 710 can facilitate data communication and exchange between clients 704, 706, and 708 and cloud infrastructure system 702. Network(s) 710 can include one or more networks. The networks can be of the same or different types. Network(s) 710 can support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0182] Figure 7 The depicted example is only one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in some other examples, the cloud infrastructure system 702 may have more than Figure 7 Those components depicted may have more or fewer components, may combine two or more components, or may have a different configuration or arrangement of components. For example, although Figure 7 Three client computing devices are depicted, but in alternative examples any number of client computing devices may be supported.

[0183] The term cloud service is generally used to refer to services that are made available to users on demand and via a communication network such as the Internet through a service provider's system (e.g., cloud infrastructure system 702). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can make use of the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase the infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle of Redwood Shores, California It provides several cloud services such as middleware services, database services, Java cloud services and other services.

[0184] In some examples, cloud infrastructure system 702 can provide one or more cloud services using different models, such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 702 can include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0185] The SaaS model enables applications or software to be delivered to customers as a service over a communication network such as the Internet, without the customer having to purchase the hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system 702. Oracle Examples of SaaS services provided include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and others.

[0186] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) as cloud services to customers to provide elastic computing and storage capabilities. Provides various IaaS services.

[0187] The PaaS model is typically used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to purchase, build, or maintain such resources. Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

[0188] Cloud services are typically provided on an on-demand, self-service basis, based on subscriptions, in an elastically scalable, reliable, highly available, and secure manner. For example, a customer can order one or more services provided by the cloud infrastructure system 702 via a subscription order. The cloud infrastructure system 702 then performs processing to provide the services requested in the customer's subscription order. For example, a user can use speech to request that the cloud infrastructure system take an action (e.g., an intent) as described above and / or provide services to a chatbot system as described herein. The cloud infrastructure system 702 can be configured to provide one or even multiple cloud services.

[0189] The cloud infrastructure system 702 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 702 can be owned by a third-party cloud service provider, and the cloud services are provided to any general public customer, where the customer can be an individual or a business. In some other examples, under a private cloud model, the cloud infrastructure system 702 can operate within an organization (e.g., within an enterprise organization) and the services are provided to customers within the organization. For example, the customers can be various departments of an enterprise such as the human resources department, the payroll department, or even individuals within the enterprise. In some other examples, under a community cloud model, the cloud infrastructure system 702 and the services provided can be shared by several organizations in a related community. Various other models, such as a hybrid of the models mentioned above, can also be used.

[0190] Client computing devices 704, 706, and 708 may be of different types (e.g., Figure 6 702 and 704) and can be capable of operating one or more client applications. A user can use a client device to interact with the cloud infrastructure system 702, such as requesting services provided by the cloud infrastructure system 702. For example, a user can use a client device to request information or actions from a chatbot as described in the present disclosure.

[0191] In some examples, the processing performed by the cloud infrastructure system 702 for providing services can involve model training and deployment. This analysis can involve using, analyzing, and manipulating data sets to train and deploy one or more models. The analysis can be performed by one or more processors, potentially processing data in parallel, performing simulations using the data, and the like. For example, big data analysis can be performed by the cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used for the analysis can include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0192] like Figure 7 As depicted in the example of , cloud infrastructure system 702 may include infrastructure resources 730 used to facilitate the provision of various cloud services provided by cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to service storage requests from applications may be part of cloud infrastructure system 702. In other examples, the storage virtual machine may be part of a different system.

[0193] In some examples, to facilitate efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 702 for different customers, the resources can be bundled into resource groups or resource modules (also referred to as "pods"). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods can be pre-provisioned for different types of cloud services. For example, a first group of pods can be provisioned for a database service, a second group of pods can be provisioned for a Java service (the second group of pods can include a different combination of resources than the pods in the first group), etc. For some services, the resources allocated for provisioning a service can be shared between services.

[0194] The cloud infrastructure system 702 itself may internally use services 732 that are shared by different components of the cloud infrastructure system 702 and that facilitate the provision of services by the cloud infrastructure system 702. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for implementing cloud support, email services, notification services, file transfer services, and the like.

[0195] The cloud infrastructure system 702 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. Figure 7 As depicted, the subsystems may include a user interface subsystem 712 that enables users or customers of the cloud infrastructure system 702 to interact with the cloud infrastructure system 702. The user interface subsystem 712 may include various interfaces, such as a web interface 714, an online store interface 716 (where cloud services provided by the cloud infrastructure system 702 are advertised and available for purchase by customers), and other interfaces 718. For example, a customer may use a client device to request (service request 734) one or more services provided by the cloud infrastructure system 702 using one or more of interfaces 714, 716, and 718. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 702, and place a subscription order for one or more services provided by the cloud infrastructure system 702 to which the customer wishes to subscribe. The service request may include information identifying the customer and the one or more services to which the customer wishes to subscribe. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 702. As part of the order, the customer may provide information identifying a chatbot system for which the services are to be provided and, optionally, one or more credentials for the chatbot system.

[0196] In some examples (such as Figure 7 In the example depicted, cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this processing, OMS 720 may be configured to: create an account for the customer (if one has not already been created); receive billing and / or billing information from the customer, which will be used to bill the customer for the requested services; verify the customer information; after verification, book the order for the customer; and orchestrate various workflows to prepare the order for fulfillment.

[0197] Once properly authenticated, the OMS 720 may then invoke the Order Provisioning Subsystem (OPS) 724, which is configured to provision resources for the order, including processing resources, memory resources, and networking resources. Provisioning may include allocating resources to the order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud services that the customer has ordered. For example, according to one workflow, the OPS 724 may be configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for the specific cloud service. The number of groups allocated to the order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users to be supported by the service, the duration of the service being requested, etc. The allocated groups may then be customized for the specific requesting customer to provide the requested service.

[0198] In some examples, the setup phase processing described above can be performed as part of the provisioning process by cloud infrastructure system 702. Cloud infrastructure system 702 can generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by cloud infrastructure system 702 itself or from storage virtual machines provided by systems other than cloud infrastructure system 702.

[0199] The cloud infrastructure system 702 can send a response or notification 744 to the requesting customer to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) can be sent to the customer to enable the customer to begin using and taking advantage of the benefits of the requested service. In some examples, for the customer requesting the service, the response can include a chatbot system ID generated by the cloud infrastructure system 702 and information identifying the chatbot system selected by the cloud infrastructure system 702 as the chatbot system corresponding to the chatbot system ID.

[0200] Cloud infrastructure system 702 can provide services to multiple customers. For each customer, cloud infrastructure system 702 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data associated with the order, and providing the requested services to the customer. Cloud infrastructure system 702 can also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics can be collected regarding the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information can be used to bill the customer. Billing can be performed, for example, on a monthly basis.

[0201] Cloud infrastructure system 702 can provide services to multiple customers concurrently. Cloud infrastructure system 702 can store information about these customers (possibly including proprietary information). In some examples, cloud infrastructure system 702 includes an identity management subsystem (IMS) 728 configured to manage customer information and provide separation of managed information so that information related to one customer cannot be accessed by another customer. IMS 728 can be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, and services for managing customer identities and roles and related functions.

[0202] Figure 8 An example of a computer system 800 is shown. In some examples, the computer system 800 can be used to implement any digital assistant or chatbot system in a distributed environment and the various servers and computer systems described above. Figure 8 As shown, computer system 800 includes various subsystems, including a processing subsystem 804 that communicates with a number of other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. The storage subsystem 818 may include non-transitory computer-readable storage media, including storage media 822 and system memory 810.

[0203] The bus subsystem 802 provides a mechanism for allowing the various components and subsystems of the computer system 800 to communicate with each other as expected. Although the bus subsystem 802 is schematically shown as a single bus, the alternative example of the bus subsystem can utilize multiple buses. The bus subsystem 802 can be any of several types of bus structures including a memory bus or a memory controller, a peripheral bus, a local bus using any of the various bus architectures, etc. For example, this architecture can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnect (PCI) bus (the PCI bus can be implemented as a mezzanine bus manufactured into the IEEE P1386.1 standard).

[0204] The processing subsystem 804 controls the operation of the computer system 800 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core processor or a multi-core processor. The processing resources of the computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 804 may include one or more dedicated coprocessors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 804 may be implemented using custom circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0205] In some examples, the processing units in the processing subsystem 804 can execute instructions stored in the system memory 810 or on the computer-readable storage medium 822. In various examples, the processing units can execute various programs or code instructions and can maintain multiple programs or processes that are executed simultaneously. At any given time, some or all of the program code to be executed can reside in the system memory 810 and / or on the computer-readable storage medium 822 (potentially including residing on one or more storage devices). Through suitable programming, the processing subsystem 804 can provide the various functions described above. In an instance where the computer system 800 executes one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0206] In some examples, a processing acceleration unit 806 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 804 , thereby speeding up the overall processing performed by the computer system 800 .

[0207] The I / O subsystem 808 may include devices and mechanisms for inputting information to the computer system 800 and / or for outputting information from or via the computer system 800. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information to the computer system 800. User interface input devices may include, for example, a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Windows, that enable a user to control and interact with the input device. Motion sensor, Microsoft 360 game controllers, devices that provide an interface for receiving input using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as devices that detect eye movements from the user (e.g., "blinking" when taking a picture and / or making a menu selection) and convert eye gestures to input devices (such as Google Assistant). ) In addition, the user interface input device may include an input device that enables the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0208] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing rods, game controllers and graphics boards, and audio / visual devices (such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser rangefinders, eye gaze tracking devices). In addition, user interface input devices can include, for example, medical imaging input devices, such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound equipment. User interface input devices can also include, for example, audio input devices, such as MIDI keyboards, digital musical instruments, etc.

[0209] In general, the term output device is used to include all possible types of devices and mechanisms for outputting information from the computer system 800 to a user or another computer. User interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices. A display subsystem may be a cathode ray tube (CRT), a flat panel device (such as one using a liquid crystal display (LCD) or plasma display), a projection device, a touch screen, or the like. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0210] The storage subsystem 818 provides a repository or data storage for information and data used by the computer system 800. The storage subsystem 818 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. The storage subsystem 818 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 804, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 804. The storage subsystem 818 can also provide authentication according to the teachings of the present disclosure.

[0211] The storage subsystem 818 may include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. Figure 8 As shown, the storage subsystem 818 includes system memory 810 and computer-readable storage media 822. The system memory 810 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, the basic input / output system (BIOS), which contains basic routines that help transfer information between components within the computer system 800 during startup, may typically be stored in ROM. RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 804. In some embodiments, the system memory 810 may include a variety of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0212] By way of example and not limitation, Figure 8 As depicted, system memory 810 may load executing applications 812 (which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc.), program data 814, and an operating system 816. By way of example, operating system 816 may include various versions of Microsoft Apple and / or Linux operating systems, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or such as iOS, Telephone, OS, OS, OS operating system and other mobile operating systems, and other operating systems.

[0213] The computer-readable storage medium 822 can store programming and data structures that provide some of the example functionality. The computer-readable medium 822 can provide storage for computer-readable instructions, data structures, program modules, and other data for the computer system 800. The software (programs, code modules, instructions) that provide the functionality described above when executed by the processing subsystem 804 can be stored in the storage subsystem 818. By way of example, the computer-readable storage medium 822 can include, for example, a hard drive, a magnetic disk drive, an optical disk drive (e.g., a CD ROM, DVD, Computer readable storage media 822 may include, but is not limited to, The computer-readable storage medium 822 may include a solid-state drive (SSD) based on flash memory, an enterprise-class flash memory drive, a non-volatile memory-based SSD such as solid-state ROM, a volatile memory-based SSD such as solid-state RAM, dynamic RAM, static RAM, etc., a DRAM-based SSD, a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD using a combination of DRAM and a flash memory-based SSD.

[0214] In some examples, the storage subsystem 818 may also include a computer-readable storage medium reader 820 that may be further connected to a computer-readable storage medium 822. The reader 820 may receive data from a storage device such as a disk, a flash memory drive, or the like and may be configured to read data from the storage device.

[0215] In some examples, the computer system 800 can support virtualization technology, including but not limited to the virtualization of processing and memory resources. For example, the computer system 800 can provide support for executing one or more virtual machines. In some examples, the computer system 800 can execute programs such as a hypervisor that promotes the configuration and management of virtual machines. Each virtual machine can be allocated with memory, computing (e.g., processor, core), I / O and networking resources. Each virtual machine is usually run independently of other virtual machines. A virtual machine usually runs its own operating system, which can be the same or different from the operating system executed by other virtual machines executed by the computer system 800. Therefore, multiple operating systems can potentially be run simultaneously by the computer system 800.

[0216] The communication subsystem 824 provides an interface to other computer systems and networks. The communication subsystem 824 serves as an interface for receiving data from other systems and transmitting data from the computer system 800 to other systems. For example, the communication subsystem 824 can enable the computer system 800 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and sending information to the client devices. For example, when the computer system 800 is used to implement Figure 1 When the robot system 120 is depicted, the communication subsystem can be used to communicate with the chat robot system selected for the application.

[0217] The communication subsystem 824 can support both wired communication protocols and / or wireless communication protocols. In some examples, the communication subsystem 824 can include a radio frequency (RF) transceiver component for accessing a wireless voice and / or data network (e.g., using cellular telephone technology, advanced data network technology such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family standards, or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, in addition to or in lieu of a wireless interface, the communication subsystem 824 can provide wired network connectivity (e.g., Ethernet).

[0218] The communication subsystem 824 can receive and transmit various forms of data. In some examples, the communication subsystem 824 can receive input communications in the form of structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., among other forms. For example, the communication subsystem 824 can be configured to receive (or send) data feeds 826 in real time from users of social media networks and / or other communication services, such as feed, Updates, web feeds (such as Rich Site Summary (RSS) feeds), and / or real-time updates from one or more third-party information sources.

[0219] In some examples, the communication subsystem 824 can be configured to receive data in the form of a continuous data stream, which can include an event stream 828 of real-time events (which may be inherently continuous or unbounded and may not have an explicit end) and / or event updates 830. Examples of applications that generate continuous data can include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0220] The communication subsystem 824 can also be configured to transmit data from the computer system 800 to other computer systems or networks. The data can be transmitted in various forms such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc. to one or more databases that can communicate with one or more streaming data sources coupled to the computer system 800.

[0221] Computer system 800 can be of various types, including a handheld portable device (e.g., Cellular phones, Computing tablets, PDAs), wearable devices (e.g., Google head-mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, Figure 8 The description of the depicted computer system 800 is intended to be a specific example only. Figure 8 Many other configurations of more or fewer components of the depicted systems are possible.Based on the disclosure and the teachings provided herein, it should be appreciated that there are other ways and / or methods to implement the various examples.

[0222] Although specific examples have been described, various modifications, changes, alternative constructions, and equivalents are possible. The examples are not limited to operation in certain specific data processing environments, but are free to operate in multiple data processing environments. In addition, although certain examples have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be restrictive. Although some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. The process may have additional steps not included in the figure. The various features and aspects of the examples described above can be used individually or in combination.

[0223] Furthermore, although certain examples have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are possible. Certain examples may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors.

[0224] Where a device, system, component or module is described as being configured to perform certain operations or functions, such configuration may be accomplished, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0225] Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the previous description of the examples will provide those skilled in the art with an enabling description for implementing the various examples. Various changes can be made to the functions and arrangements of the elements.

[0226] The present specification and drawings should, therefore, be viewed in an illustrative rather than a restrictive sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, while specific examples have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0227] In the foregoing description, various aspects of the present disclosure have been described with reference to specific examples of the present disclosure, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used individually or in combination. Further, without departing from the broader spirit and scope of the specification, the examples may be utilized in any number of environments and application environments other than those described herein. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0228] In the foregoing description, for the purpose of illustration, the method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order. It should also be understood that the method described above can be performed by hardware components or can be embodied in a sequence of machine-executable instructions, which can be used to make a machine (such as a general or special processor or logic circuit programmed with the instructions) perform the method. These machine-executable instructions can be stored on one or more machine-readable media, such as CD-ROM or other types of optical disks, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic cards or optical cards, flash memories or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the method can be performed by a combination of hardware and software.

[0229] Where a component is described as being configured to perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operations, or any combination thereof.

[0230] While illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be interpreted to encompass such variations, except to the extent limited by the prior art.

Claims

1. A method for training an intent classifier, the method comprising: Receiving a training utterance set at a data processing system, the training utterance set comprising a plurality of original utterances for training the intent classifier to identify one or more intents of one or more utterances from an intent set, each original utterance in the plurality of original utterances corresponding to a specific intent in the intent set; The data processing system augments the training utterance set with noisy text and generates a plurality of groups of augmented utterances to generate an augmented training utterance set, wherein each group of augmented utterances in the plurality of groups of augmented utterances corresponds to the specific intent of a corresponding original utterance in the plurality of original utterances, is generated for the corresponding original utterance and includes the corresponding original utterance, wherein the augmentation comprises: obtaining noise text unrelated to original text in the utterances of the training utterance set from a word list, a text corpus, a publication, a dictionary, or any combination thereof, wherein the noise text is a random text string or text sentence generated or copied from the word list, the text corpus, the publication, the dictionary, or any combination thereof, and Merging the noise text into the original utterances of the training utterance set at a predefined expansion ratio of original text to noise text relative to original text to generate an expanded utterance, wherein the noise text is arranged at the predefined expansion ratio of original text to noise text of 1:1.2 to 1:5 in at least one of the following ways: (i) merged in front of the original text of at least one original utterance in the original utterances, (ii) merged in behind the original text of at least one original utterance in the original utterance, and (iii) integrated into the original text of the original utterance, wherein the arrangement of the noise text does not change words in the original text of the original utterance; and The intent classifier is trained by the data processing system using the augmented set of training utterances, wherein the predefined augmentation ratio is selected to preserve original labels of the original utterances.

2. The method according to claim 1, wherein The noise text is a random text string automatically generated from a list of stop words.

3. The method according to claim 1, wherein The noise text is a text sentence automatically copied word for word or shuffled from the text corpus.

4. The method according to claim 1, wherein The noise text is a random text string automatically generated from the text corpus.

5. The method according to claim 1, wherein The noise text is a random text string automatically generated from the dictionary, wherein the frequency of words or characters selected for the random text string is not considered.

6. The method of claim 1, wherein: The noise text is a random text string automatically generated from the dictionary, taking into account the frequency of words or characters selected for the random text string.

7. A system for training an intent classifier, the system comprising: one or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform actions comprising: receiving a training utterance set, the training utterance set comprising a plurality of original utterances, the plurality of original utterances being used to train the intent classifier to identify one or more intents of one or more utterances from an intent set, each original utterance in the plurality of original utterances corresponding to a specific intent in the intent set; Augmenting the training utterance set with noise text and generating a plurality of groups of augmented utterances to generate an augmented training utterance set, wherein each group of augmented utterances in the plurality of groups of augmented utterances corresponds to the specific intent of a corresponding original utterance in the plurality of original utterances, is generated for the corresponding original utterance and includes the corresponding original utterance, wherein the augmentation comprises: obtaining noise text that is unrelated to original text in the utterances of the training utterance set from a word list, a text corpus, a publication, a dictionary, or any combination thereof, wherein the noise text is a random text string or text sentence, the noise text being a random text string or text sentence generated or copied from the word list, the text corpus, the publication, the dictionary, or any combination thereof, and merging the noise text into the original utterance at a predefined expansion ratio of original text to noise text relative to the original text in the original utterances of the training utterance set to generate an augmented utterance, wherein the noise text is arranged at the predefined expansion ratio of original text to noise text of 1:1.2 to 1:5 in at least one of the following ways: (i) merged in front of the original text of at least one original utterance in the original utterances, (ii) merged in behind the original text of at least one original utterance in the original utterances, and (iii) integrated into the original text of the original utterances, wherein the arrangement of the noise text does not change words in the original text of the original utterances; and training the intent classifier using the expanded training utterance set, The predefined expansion ratio is selected to maintain the original label of the original utterance.

8. The system of claim 7, wherein: The noise text is a random text string automatically generated from a list of stop words.

9. The system of claim 7, wherein: The noise text is a text sentence automatically copied word for word or shuffled from the text corpus.

10. The system of claim 7, wherein: The noise text is a random text string automatically generated from the text corpus.

11. The system of claim 7, wherein: The noise text is a random text string automatically generated from the dictionary, wherein the frequency of words or characters selected for the random text string is not considered.

12. The system of claim 7, wherein: The noise text is a random text string automatically generated from the dictionary, taking into account the frequency of words or characters selected for the random text string.

13. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform actions comprising: receiving a training utterance set, the training utterance set comprising a plurality of original utterances for training an intent classifier to identify one or more intents of one or more utterances from an intent set, each original utterance in the plurality of original utterances corresponding to a specific intent in the intent set; The training utterance set is augmented with noise text to generate a plurality of augmented utterances to generate an augmented training utterance set, wherein: Each group of expanded utterances in the plurality of groups of expanded utterances corresponds to the specific intent of a corresponding original utterance in the plurality of original utterances, is generated for the corresponding original utterance and includes the corresponding original utterance, wherein the expansion comprises: obtaining noise text unrelated to original text in the utterances of the training utterance set from a word list, a text corpus, a publication, a dictionary, or any combination thereof, wherein the noise text is a random text string or text sentence generated or copied from the word list, the text corpus, the publication, the dictionary, or any combination thereof, and Merging the noise text into the original utterance of the training utterance set at a predefined expansion ratio of original text to noise text relative to original text in the original utterance to generate an expanded utterance, wherein the noise text is arranged at the predefined expansion ratio of original text to noise text of 1:1.2 to 1:5 in at least one of the following ways: (i) merged in front of the original text of at least one original utterance in the original utterance, (ii) merged in behind the original text of at least one original utterance in the original utterance, and (iii) integrated into the original text of the original utterance, and wherein the setting of the noise text does not change words in the original text of the original utterance; and training the intent classifier using the expanded training utterance set, The predefined expansion ratio is selected to maintain the original label of the original utterance.

14. The computer program product of claim 13, wherein: The noise text is a random text string automatically generated from a list of stop words.

15. The computer program product of claim 13, wherein: The noise text is a text sentence automatically copied word for word or shuffled from the text corpus.

16. The computer program product of claim 13, wherein: The noise text is a random text string automatically generated from the text corpus.

17. The computer program product of claim 13, wherein: The noise text is a random text string automatically generated from the dictionary, wherein the frequency of the words or characters selected for the random text string is not taken into account; or the noise text is a random text string automatically generated from the dictionary, wherein the frequency of the words or characters selected for the random text string is taken into account.

18. A method for determining intent from an utterance, the method comprising: receiving, by a chatbot system, an utterance generated by a user interacting with the chatbot system; Classifying the utterance into an intent category corresponding to an intent in an intent set using an intent classifier deployed within the chatbot system, wherein the intent classifier includes a plurality of model parameters identified using training data, the training data including: An augmented training utterance set for training the intent classifier to identify one or more intents of one or more utterances from the intent set, wherein the augmented training utterance set including augmented utterances is artificially generated by incorporating noise text into a training utterance set including a plurality of original utterances to include a plurality of groups of augmented utterances, each of the plurality of original utterances corresponding to a specific intent in the intent set, wherein each group of augmented utterances in the plurality of groups of augmented utterances corresponds to the specific intent of a corresponding original utterance in the plurality of original utterances and is generated for and includes the corresponding original utterance, wherein the noise text is incorporated into the original utterances from the training utterance set at a predefined augmentation ratio of original text to noise text relative to the original text in the original utterances to generate the augmented utterance, wherein the noise text is a random text string or text sentence, the noise text being a random text string or text sentence generated or copied from a word list, a text corpus, a publication, a dictionary, or any combination thereof, wherein the noise text is arranged with the predefined expansion ratio of original text to noise text of 1:1.2 to 1:5 in at least one of the following ways: (i) merged in front of the original text of at least one original utterance in the original utterance, (ii) merged in back of the original text of at least one original utterance in the original utterance, and (iii) integrated into the original text of the original utterance, wherein the arrangement of the noise text does not change words in the original text of the original utterance, wherein the predefined expansion ratio is selected to maintain the original labels of the original utterance; and wherein the plurality of model parameters are identified using the training data based on minimizing a loss function; and The intent is output based on the classification using the intent classifier.

Images