Entity perception data enhancement techniques

Abstract

In some embodiments, a two-stage enhancement technique is applied to a first set of training data of the intent prediction model by first applying one or more data enhancement techniques and then applying an additional enhancement technique to post-process the first-stage results; the first set of training data and the post-processed enhanced training data are combined to train an intent prediction model. In another embodiment, an entity awareness ("EA") technique and a two-stage enhancement technique are applied together to produce a second set of training data; the first set of training data and the second set of training data are combined to train an intent prediction model. In another embodiment, one or more negative entity awareness data enhancement techniques are applied to the first set of training data to produce a second set of training data; the first set of training data and the second set of training data are combined to train an intent prediction model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Nonprovisional Application No. 18 / 163,230, filed on February 1, 2023, entitled “TECHNIQUES FOR TWO-STAGE ENTITY—AWARE DATA AUGMENTATION,” U.S. Nonprovisional Application No. 18 / 163,231, filed on February 1, 2023, entitled “TECHNIQUES FOR POSITIVE ENTITY AWARE AUGMENTATION USING TWO-STAGE AUGMENTATION,” and U.S. Nonprovisional Application No. 18 / 163,235, filed on February 1, 2023, entitled “TECHNIQUES FOR NEGATIVE ENTITY AWARE AUGMENTATION,” all of which claim priority under 35 U.S.C. § 119(e) to U.S. Nonprovisional Application No. 18 / 163,235, filed on February 1, 2023, entitled “TECHNIQUES FOR NEGATIVE ENTITY AWARE AUGMENTATION,” filed on June 22, 2022. The disclosure of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to machine learning techniques, and more particularly to techniques for data augmentation using a combination of positive augmentation with post-processing and negative augmentation to improve the robustness of intent prediction models to different entity values. Background Art

[0004] Artificial intelligence / machine learning (AI / ML) has many applications. For example, many users around the world are on instant messaging or chat platforms in order to get instant responses. Organizations often use these instant messaging or chat platforms to have real-time conversations with customers (or end users). However, it can be very costly for organizations to hire service personnel to communicate with customers or end users in real time. Chatbots or robots have begun to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with robots through messaging applications that end users have already installed and used. Intelligent robots, which are often powered by artificial intelligence (AI), can communicate context more intelligently in real-time conversations, and therefore can allow more natural conversations between robots and end users to improve the conversation experience. Unlike the end user learning a fixed set of keywords or commands that the robot knows how to respond to, the intelligent robot can understand the end user's intent based on the user's natural language utterances and respond accordingly. Summary of the invention

[0005] A machine learning technique is disclosed for performing data augmentation using a combination of positive augmentation with post-processing and negative augmentation to improve the robustness of an intent prediction model to different entity values.

[0006] In various embodiments, a computer-implemented method includes: applying a combination of entity-aware data augmentation techniques to a first set of training data to produce a second set of training data, wherein the entity-aware data augmentation techniques include a positive augmentation technique, a two-stage augmentation technique for post-processing the augmented data, and a negative augmentation technique; and training an intent prediction model using the combined first set of training data and second set of training data.

[0007] In various embodiments, a computer-implemented method includes: accessing a first set of training data for an intent prediction model, the first set of training data including utterances and intent labels; extracting named entities from the first set of training data using a trained named entity recognition (NER) model; applying one or more data augmentation techniques to the first set of training data to produce a second set of training data; applying additional augmentation techniques to augment the second set of training data to create post-processed augmented training data, wherein the additional augmentation techniques include replacing at least one or more entity values ​​of named entities within the second set of training data with random values ​​of the same entity type; combining the first set of training data and the post-processed augmented training data to generate extended training data; training the intent prediction model using the extended training data; and deploying the trained intent prediction model.

[0008] In yet another embodiment, the one or more data augmentation techniques include randomly replacing a portion of the first set of training data with a synonym, and randomly deleting one or more named entities within the first set of training data.

[0009] In yet another embodiment, the one or more data augmentation techniques include randomly inserting synonyms in random positions in the first set of training data, or randomly swapping positions of two named entities within the first set of training data.

[0010] In yet another embodiment, the one or more data augmentation techniques include randomly adding in-domain context around entity values ​​within the first set of training data.

[0011] In yet another embodiment, the entity type comprises a category to which the named entity belongs; and the entity value of the at least one or more entity values ​​comprises one or more words representing individual named entities within the entity type.

[0012] In yet another embodiment, the selection of applying one or more data augmentation techniques is a hyperparameter.

[0013] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0014] In various embodiments, a computer program product is provided, the computer program product being tangibly embodied in a non-transitory machine-readable medium and comprising instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0015] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts are only a small part of the numerous implementations and contexts. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a simplified block diagram of a distributed environment incorporating an exemplary embodiment.

[0017] Figure 2 is a simplified block diagram of a computing system that implements a master robot in accordance with certain embodiments.

[0018] Figure 3 is a simplified block diagram of a computing system that implements a skill robot in accordance with certain embodiments.

[0019] Figure 4 is a simplified block diagram of a chatbot training and deployment system according to various embodiments.

[0020] Figure 5is a simplified flow chart illustrating the overall tuning process of a predictive model augmented with entity-aware data.

[0021] Figure 6 Depicted is a simplified diagram of a distributed system for implementing various embodiments.

[0022] Figure 7 is a simplified block diagram of one or more components of a system environment according to various embodiments, by which services provided by one or more components of an embodiment system may be provisioned as cloud services.

[0023] Figure 8 An example computer system is illustrated that may be used to implement various embodiments. DETAILED DESCRIPTION

[0024] In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it is apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not intended to be limiting. The word "exemplary" as used herein means "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or superior to other embodiments or designs.

[0025] introduce

[0026] AI / ML has many applications. As an example application, a digital assistant is an artificial intelligence-driven interface that helps users complete various tasks in a natural language conversational manner. For each digital assistant, customers can assemble one or more skills. Skills (also referred to herein as chatbots, robots, or skillbots) are individual robots that focus on specific types of tasks (such as tracking inventory, submitting time cards, and creating expense reports). When an end user interacts with a digital assistant, the digital assistant evaluates the end user input and routes the conversation to and from the appropriate chatbot. The digital assistant can be provided to the end user through various channels, such as Messenger, SKYPE Messenger or Short Message Service (SMS). Channels carry chats back and forth from end users on various messaging platforms to digital assistants and their various bots. These channels can also support user agent escalations, event-initiated chats, and testing.

[0027] Intent allows the chatbot to understand what the user wants the chatbot to do. Intent refers to the user's intention conveyed to the chatbot via user requests and statements, which are also referred to as discourse (e.g., getting an account balance, making a purchase, etc.). As used herein, discourse or messages can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. Intent can be created by providing a name that describes a certain user action (e.g., ordering pizza) and compiling a set of real-life user statements or discourses that are usually associated with triggering the action. Since the chatbot's cognition is derived from these intentions, each intent can be created from a robust (one to twenty discourses) and variable data set so that the chatbot can interpret ambiguous user input. A rich set of discourses enables the chatbot to understand what the user wants to do when it receives messages such as "Forget this order!" or "Cancel delivery!" - messages with the same meaning but different expressions. In general, these intentions and the discourses belonging to these intentions constitute the chatbot's training corpus. By using the corpus training algorithm, the customer can turn the algorithm into a model that is used as a reference tool for parsing (one or more) end-user inputs into a single intent. Customers can improve the cognitive acumen of their chatbots through multiple rounds of intent testing and intent training.

[0028] There are several challenges in training machine learning (ML) models. One issue is robustness to entity values ​​(or entity sensitivity), because when entity values ​​change, the model produces different outputs with different confidence levels. As a result, for input utterances that differ only in their entity values, the resulting model predictions can identify different categories and sometimes different prediction confidence levels. For example, when the input utterance "Give Peter a raise" is changed to "Give James a raise", the model may output a different category or a different confidence level. Another issue is that ML models can use entity values ​​as a shortcut for predictions, such that there is an intent-entity association between the entity value and its intent output category. For example, if the input utterance contains a name and an amount, then the ML model may learn and predict an intent about sending money, which may cause the model to incorrectly predict an intent about sending money for the following utterance: "Did Katy Smith deduct $20 from my account?". As another example, if the input utterance contains a name and a date, then the model may learn and predict an intent about creating meeting minutes, which may cause the model to incorrectly predict an intent about creating meeting minutes for the following utterance: "Is Katy Smith's birthday December 19?".

[0029] Thus, different approaches are needed to address the challenge of model prediction sensitivity to entity value changes. The entity-aware data augmentation techniques described in the present disclosure provide advancements and improvements to training ML models by utilizing a combination of various positive, negative, and multi-stage augmentations of entity values ​​in training data to focus more on the overall context of input utterances rather than individual entity value changes. In various embodiments, a method is provided, the method comprising: obtaining a first set of training data for an intent prediction model, wherein the first set of training data comprises utterances and intent labels; extracting named entities from the first set of training data using a trained NER model; depending on the adjustment requirements for hyperparameters, applying one or more data augmentation techniques to the first set of training data to produce a second set of training data; applying additional augmentation techniques to augment the second set of training data to create post-processed augmented training data, wherein the additional augmentation techniques include replacing at least one or more entity values ​​of named entities within the second set of training data with random values ​​of the same entity type; combining the first set of training data and the post-processed augmented training data to generate extended training data to train the intent prediction model; and deploying the trained intent prediction model to a chatbot system.

[0030] Robotics and analytics systems

[0031] A robot (also referred to as a skill, a chatbot, a chatterbot or a conversational robot) is a computer program that can perform a conversation with an end user. The robot can typically respond to a natural language message (e.g., a question or comment) through a messaging application using a natural language message. An enterprise can use one or more robots to communicate with an end user through a messaging application. Messaging applications may include, for example, an upper layer (over-the-top, OTT) messaging channel (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack or SMS), a virtual private assistant (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), a mobile and web application extension that extends a local or hybrid / responsive mobile application or web application with chat capabilities, or a voice-based input (such as a device or application with an interface for interacting using Siri, Cortana, Google Voice or other speech input).

[0032] In some examples, a robot can be associated with a uniform resource identifier (URI). The URI can use a string to identify the robot. The URI can be used as a webhook for one or more messaging applications. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The robot can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application. The HTTP post call message can be directed to the URI from the messaging application. In some examples, the message may be different from the HTTP post call message. For example, the robot can receive a message from a short message service (SMS). Although the discussion herein refers to the robot receiving communications as messages, it should be understood that the message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0033] End users interact with robots through conversational interactions (sometimes called conversational user interfaces (UIs)), just as end users interact with other people. In some cases, a conversational interaction can include an end user saying "hello" to a robot, and the robot responding with "hi" and asking the end user how it can help. End users also interact with robots through other types of interactions, such as transactional interactions (e.g., with a banking robot that is at least trained to transfer funds from one account to another), informational interactions (e.g., with a human resources robot that is at least trained to check a user's remaining vacation time), and / or retail interactions (e.g., with a retail robot that is at least trained to discuss returning a purchase or seeking technical support).

[0034] In some examples, the robot can intelligently handle end-user interactions without intervention by the robot's administrator or developer. For example, the end user can send one or more messages to the robot to achieve a desired goal. The message can include certain content, such as text, emoticons, audio, images, videos, or other methods of conveying messages. In some examples, the robot can automatically convert the content into a standardized form and generate a natural language response. The robot can also automatically prompt the end user to enter additional input parameters or request other additional information. In some examples, the robot can also initiate communication with the end user instead of passively responding to the end user's words.

[0035] A conversation with a robot can follow a specific conversation flow including multiple states. The flow can define what will happen next based on the input. In some examples, a state machine including user-defined states (e.g., end-user intent) and actions to be taken in a state or from one state to another can be used to implement the robot. The conversation can take different paths based on the end-user input, which may affect the robot's decision on the process. For example, in each state, based on the end-user's input or speech, the robot can determine the end-user's intent in order to determine the appropriate next action to be taken. As used herein and in the context of speech, the term "intent" refers to the intention of the user providing the speech. For example, a user may intend to have a conversation with the robot to order a pizza, where the user's intent will be represented by the speech "order pizza". User intent can be for a specific task that the user wants the robot to perform on behalf of the user. Therefore, the speech reflecting the user's intent can be expressed as a question, command, request, etc.

[0036] In the context of robot configuration, the term "intent" is also used in this article to refer to configuration information that maps a user's speech to a specific task / action or task / action category that the robot can perform. In order to distinguish the intent of the speech (i.e., the user intent) from the robot's intent, the latter is sometimes referred to as "robot intent" in this article. Robot intent can include a set of one or more utterances associated with the intent. For example, the intent of ordering pizza can have various permutations of utterances expressing the desire to order pizza. These associated utterances can be used to train the robot's intent classifier so that the intent classifier can then determine whether the input utterance from the user matches the intent of ordering pizza. Robot intent can be associated with one or more dialogue flows used to start a conversation with the user and be in a certain state. For example, the first message of the intent of ordering pizza may be the question "What kind of pizza do you want?" In addition to the associated utterances, the robot intent can also contain named entities related to the intent. For example, the intent of ordering pizza can include variables or parameters (e.g., ingredients 1, ingredients 2, pizza type, pizza size, pizza quantity, etc.) for performing the task of ordering pizza. The value of the entity is usually obtained by talking to the user.

[0037] Figure 1 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to some embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102 that enables a user 104 of the DABP 102 to create and deploy a digital assistant or chatbot system. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, Figure 1As shown in , a user 104 representing a particular business can use DABP 102 to create and deploy a digital assistant 106 for users of that particular business. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform can be used by multiple businesses to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0038] For purposes of this disclosure, a "digital assistant" is a tool that helps the user of the digital assistant complete various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital tool implemented using a program, code, or instruction that can be executed by one or more processors), hardware, or a combination of hardware and software. A digital assistant can be implemented or implemented in a variety of physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Thus, for purposes of this disclosure, the terms "digital assistant" and "chatbot system" are interchangeable.

[0039] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks via a natural language-based conversation between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and get responses 112 from the digital assistant 106. The conversation can include one or more of the inputs 110 and the responses 112. Via these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond with an appropriate response to the user.

[0040] User input 110 is generally in the form of natural language and is referred to as utterance. User utterance 110 can be in text form, such as when a user types a sentence, a question, a text fragment, or even a single word and provides it as input to the digital assistant 106. In some examples, user utterance 110 can be in the form of audio input or speech, such as when a user speaks or speaks something provided as input to the digital assistant 106. The utterance is usually the language spoken by the user. For example, the utterance can be English, or some other language. When the utterance is in the form of speech, the speech input is converted into a text form utterance of the particular language, and then the text utterance is processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert speech or audio input into text utterances, which are then processed by the digital assistant 106. In some examples, the conversion of speech to text can be completed by the digital assistant 106 itself.

[0041] An utterance (which may be a textual utterance or a verbal utterance) may be a fragment, a sentence, a plurality of sentences, one or more words, one or more questions, a combination of the types mentioned above, and the like. The digital assistant 106 is configured to apply natural language understanding (NLU) technology to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that the utterance is a textual utterance provided directly by the user of the digital assistant 106 or is the result of a conversion of an input verbal utterance to a textual form. However, this is not intended to be limiting or restrictive in any way.

[0042] For example, user input can request to order pizza by providing an utterance such as "I want to order pizza". Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate actions. Appropriate actions may involve, for example, responding to the user with questions requesting the user to input the type of pizza the user wants to order, the size of the pizza, any ingredients of the pizza, etc. The response 106 provided by the digital assistant may also be in natural language form and is typically the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). For users who order pizza, via a conversation between the user and the digital assistant 106, the digital assistant may guide the user to provide all the information required for pizza ordering, and then order the pizza at the end of the conversation. The digital assistant 106 may end the conversation by outputting information indicating that the pizza has been ordered to the user.

[0043] At a conceptual level, the digital assistant 106 performs various processing in response to utterances received from the user. In some examples, this processing involves a sequence or pipeline of processing steps, including, for example, understanding the meaning of the input utterance, determining an action to be performed in response to the utterance, causing the action to be performed where appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing can include parsing the received input utterance to understand the structure and meaning of the utterance, refining and reconstructing the utterance to construct a more understandable form (e.g., logical form) or structure for the utterance. Generating a response can include using NLG techniques.

[0044] The NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related tasks, such as sentence parsing (e.g., tokenization, lemmatization, identifying part-of-speech tags of sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing single clauses, resolving anaphora, performing chunking, etc.). In some examples, the NLU processing is performed by the digital assistant 106 itself. In some other examples, the digital assistant 106 may use other resources to perform part of the NLU processing. For example, the syntax and structure of the sentence can be identified by using a parser, a part-of-speech tagger, and / or NER processing input utterance sentences. In one embodiment, for English, sentence structure and syntax are analyzed using a parser, a part-of-speech tagger, and a named entity recognizer such as provided by the Stanford NLP group. These are provided as part of the Stanford CoreNLP toolkit.

[0045] Although the various examples provided in this disclosure show utterances in the English language, this is meant only as an example. In some examples, the digital assistant 106 is also capable of handling utterances in languages ​​other than English. The digital assistant 106 can provide subsystems (e.g., components that implement NLU functionality) configured to perform processing on different languages. These subsystems can be implemented as pluggable units that can be called using service calls from the NLU core server. This makes NLU processing flexible and scalable for each language, including allowing different processing orders. Language packs can be provided for various languages, where the language pack can register a list of subsystems that can provide services from the NLU core server.

[0046] Digital assistants (such as Figure 1 The digital assistant 106 depicted in FIG. 1 may be available or accessible to its user 108 through a variety of different channels, such as, but not limited to, via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. A single digital assistant may have several channels configured for it so that it can run and be accessed on different services simultaneously.

[0047] A digital assistant or chatbot system generally includes or is associated with one or more skills. In some embodiments, these skills are separate chatbots (referred to as skill bots) that are configured to interact with a user and complete specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing gadgets, etc. For example, for Figure 1 In the embodiment depicted in , the digital assistant or chatbot system 106 includes skills 116 - 1, 116 - 2, 116 - 3, etc. For the purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skill bot” and “skillbots”, respectively.

[0048] Each skill associated with the digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, where the conversation can include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses can be in the form of text or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for selection by the user.

[0049] Skills or skill robots can be associated or added to digital assistants in a variety of ways. In some cases, skill robots can be developed by an enterprise and then added to a digital assistant using DABP 102. In other cases, skill robots can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In still other cases, DABP 102 provides an online digital store (referred to as a "skill store") that provides a variety of skills for a range of tasks. Skills provided by the skill store can also expose various cloud services. In order to add skills to a digital assistant being generated using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skills, and indicate that the selected skills will be added to the digital assistant created using DABP 102. Skills from the skill store can be added to the digital assistant as is or in a modified form (e.g., a user of DABP 102 can select and clone a specific skill robot provided by a skill store, customize or modify the selected skill robot, and then add the modified skill robot to the digital assistant created using DABP 102).

[0050] A digital assistant or chatbot system may be implemented using a variety of different architectures. For example, in some embodiments, a digital assistant created and deployed using DABP 102 may be implemented using a master robot / child (or slave) robot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more child robots that are skill robots. For example, in Figure 1 In the illustrated embodiment, the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are child robots of the master robot 114. In some examples, the digital assistant 106 itself is considered to act as a master robot.

[0051] A digital assistant implemented according to the master-sub robot architecture enables the user of the digital assistant to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user interacts with the digital assistant, the user input is received by the master robot. The master robot then performs processing to determine the meaning of the utterance input by the user. The master robot then determines whether the task requested by the user in the utterance can be handled by the master robot itself, otherwise the master robot selects an appropriate skill robot to handle the user request and routes the conversation to the selected skill robot. This enables the user to talk to the digital assistant through a shared single interface, and still provides the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot of the digital assistant can be docked with a skill robot with specific functionality (such as a customer relationship management (CRM) robot for performing functions related to customer relationship management, an enterprise resource planning (ERP) robot for performing functions related to enterprise resource planning, a human capital management (HCM) robot for performing functions related to human capital management, etc.). In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a shared master robot interface, and multiple skill robots are provided behind the scenes to handle user requests.

[0052] In some examples, in the master robot / sub-robot infrastructure, the master robot is configured to know the available list of skill robots. The master robot has access to metadata that identifies each available skill robot, and for each skill robot, the capabilities of the skill robot include the tasks that the skill robot can perform. After receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot that can best serve or handle the user request from multiple available skill robots. The master robot then routes the utterance (or a portion of the utterance) to the specific skill robot for further handling. Thus, control flows from the master robot to the skill robot. The master robot can support multiple input and output channels. In some examples, routing can be performed with the aid of processing performed by one or more available skill robots. For example, as discussed below, a skill robot can be trained to infer the intent of an utterance and determine whether the inferred intent matches the intent that the skill robot is configured with. Therefore, routing performed by the master robot can involve the skill robot conveying to the master robot an indication of whether the skill robot has been configured with an intent suitable for handling the utterance.

[0053] Although Figure 1 The embodiment in FIG. 1 shows a digital assistant 106 including a master robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide digital assistant functionality. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an implementation using a combination of software and hardware.

[0054] DABP 102 provides an infrastructure and various services and features that enable users of DABP 102 to create a digital assistant including one or more skill robots associated with the digital assistant. In some cases, a skill robot can be created by cloning an existing skill robot, for example, cloning a skill robot provided by a skill store. As indicated above, DABP 102 provides a skill store or skill catalog that supplies multiple skill robots for performing various tasks. Users of DABP 102 can clone skill robots from the skill store. The cloned skill robots can be modified or customized as needed. In some other cases, users of DABP 102 create skill robots from scratch using the tools and services provided by DABP 102. As indicated above, the skill store or skill catalog provided by DABP 102 can supply multiple skill robots for performing various tasks.

[0055] In some examples, at a high level, creating or customizing a skill bot involves the following steps:

[0056] (1) Configuring the settings of a new skill robot

[0057] (2) Configure one or more intents for the skill robot

[0058] (3) Configure one or more entities for one or more intents

[0059] (4) Training Skill Robots

[0060] (5) Create a dialogue flow for the skill bot

[0061] (6) Add custom components to skill robots as needed

[0062] (7) Testing and deploying skill robots

[0063] Each of these steps is briefly described below.

[0064] (1) Configuring settings for a new skill robot - Various settings can be configured for a skill robot. For example, a skill robot designer can specify one or more invocation names for the skill robot being created. These invocation names can then be used by a user of the digital assistant to explicitly invoke the skill robot. For example, a user can enter an invocation name in the user's utterance to explicitly invoke the corresponding skill robot.

[0065] (2) Configure one or more intents and associated example utterances for the skill robot - The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classes of utterances that the skill robot is trained to infer the input. After receiving the utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then responds to the utterance by taking appropriate actions based on the intent inferred for the utterance. In some cases, the intent of the skill robot represents the tasks that the skill robot can perform for the user of the digital assistant. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot may include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0066] For each intent defined for a skill bot, the skill bot designer may also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent utterances that a user might enter into the skill bot for that intent. For example, for the CheckBalance intent, example utterances may include "What is my savings account balance?", "How much money is in my checking account?", "How much money is in my account?", etc. Thus, various permutations of typical user utterances may be specified as example utterances for an intent.

[0067] Intents and their associated example utterances are used as training data to train a skill robot. A variety of different training techniques can be used. As a result of this training, a prediction model is generated that is configured to take an utterance as input and output an intent inferred by the prediction model for the utterance. In some cases, the input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer the intent of the input utterance. The skill robot can then take one or more actions based on the inferred intent.

[0068] (3) Configuring entities for one or more intents of a skill bot - In some cases, additional context may be required to enable a skill bot to respond correctly to a user request. For example, there may be cases where user input utterances resolve to the same intent in a skill bot. For example, in the example above, the utterances "What is my savings account balance?" and "How much money is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests asking different things. To clarify such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "savings" enables the skill bot to resolve user requests and respond appropriately. In the example above, although the utterances are resolved to the same intent, the values ​​associated with the AccountType entity are different for the two utterances. This enables the skill bot to perform potentially different actions for the two utterances, even though they resolve to the same intent. One or more entities can be specified for certain intents configured for a skill bot. Thus, entities are used to add context to the intent itself. Entities help to more completely describe the intent and enable the skill bot to complete the user request.

[0069] In some examples, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by a skill robot designer. Built-in entities are general entities that can be used with various robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by a skill robot designer that enables various banking transactions to be performed by checking keywords entered by the user (such as checking, savings, and credit card, etc.).

[0070] (4) Training a Skill Robot - A skill robot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select intents associated with the received user input. As indicated above, the skill robot must be trained for this. In some embodiments, the skill robot is trained based on the intents configured for the skill robot and example utterances associated with those intents (collectively referred to as training data) so that the skill robot can parse the user input utterance into one of the intents configured for the robot. In some examples, the skill robot uses a predictive model that is trained using the training data and allows the skill robot to discern what the user said (or in some cases, attempted to say). DABP 102 provides a variety of different training techniques that can be used by skill robot designers to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some examples, a portion (e.g., 80%) of the training data is used to train the skill robot model, while another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as a trained skill robot) can be used to handle and respond to user utterances. In some cases, the user's utterance may be a question that requires only a single answer and does not require further conversation. To handle such situations, a Q&A (question and answer) intent can be defined for the skill robot. This enables the skill robot to output a reply to the user's request without updating the dialog definition. The Q&A intent is created in a similar manner to the regular intent. The dialog flow for the Q&A intent may be different from the dialog flow for the regular intent.

[0071] (5) Creating a dialogue flow for a skill robot - The dialogue flow specified for a skill robot describes how the skill robot reacts when different intents for the skill robot are parsed in response to received user input. The dialogue flow defines the operations or actions that the skill robot will take (e.g., how the skill robot responds to user utterances, how the skill robot prompts the user for input, how the skill robot returns data). The dialogue flow is like a flowchart that the skill robot follows. Skill robot designers use a language such as markdown to specify the dialogue flow. In some embodiments, a version of YAML called OBotML can be used to specify the dialogue flow of a skill robot. The dialogue flow definition for a skill robot acts as a model for the conversation itself, that is, a model that allows skill robot designers to choreograph the interaction between the skill robot and the users served by the skill robot.

[0072] In some examples, the dialog flow definition for a skill bot contains three parts:

[0073] (a) Context

[0074] (b) Default transition section

[0075] (c) Status section

[0076] Context section - Skill bot designers can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, user variables that enable skill bots to recognize and retain user preferences, etc.

[0077] Default Transitions Section - You can define transitions for a skill bot in the Dialog Flow States section or in the Default Transitions section. Transitions defined in the Default Transitions section act as fallbacks and are triggered when no applicable transitions are defined within a state or when the conditions required to trigger a state transition cannot be met. The Default Transitions section can be used to define routes that allow a skill bot to gracefully handle unexpected user actions.

[0078] States section - A dialog flow and its associated actions are defined as a series of transition states that govern the logic within the dialog flow. Each state node within a dialog flow definition names a component that provides the functionality required at that point in the dialog. States are therefore built around components. States contain component-specific characteristics and define transitions to other states that are triggered after the component executes.

[0079] You can use the state section to handle special case scenarios. For example, sometimes you may want to give the user the option to temporarily leave the first skill they are interacting with in order to do something in a second skill within the digital assistant. For example, if the user is talking to a shopping skill (e.g., the user has made some purchase choices), the user may want to jump to the banking skill (e.g., the user may want to make sure he / she has enough money for the purchase), and then return to the shopping skill to complete the user's order. To address this, the action in the first skill can be configured to initiate an interaction with a different second skill in the same digital assistant and then return to the original flow.

[0080] (6) Adding custom components to a skill bot - As described above, a state specified in a skill bot's dialog flow names a component that provides the functionality required to correspond to that state. Components enable a skill bot to perform a function. In some embodiments, DABP 102 provides a set of preconfigured components for performing a wide range of functions. A skill bot designer can select one or more of these preconfigured components and associate them with a state in a skill bot's dialog flow. A skill bot designer can also use the tools provided by DABP 102 to create custom or new components and associate the custom components with one or more states in a skill bot's dialog flow.

[0081] (7) Testing and deploying skill bots - DABP 102 provides several features that enable skill bot designers to test the skill bots they are developing. The skill bots can then be deployed and included in digital assistants.

[0082] Although the above description describes how to create a skill robot, similar techniques can also be used to create a digital assistant (or master robot). At the master robot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master robot) can handle without invoking the skill robot associated with the digital assistant. Examples of system intents defined for the master robot include: (1) Exit: Applicable when the user signals a desire to exit the current conversation or context in the digital assistant; (2) Help: Applicable when the user requests help or guidance; and (3) Unresolved intent: Applicable to user input that does not match well with the exit and help intents. The digital assistant also stores information about one or more skill robots associated with the digital assistant. This information enables the master robot to select a specific skill robot to handle the utterance.

[0083] At the master robot or digital assistant level, when a user enters a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant determines this using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the utterance entered by the user is to be routed to a specific skill for handling, handled by the digital assistant or master robot itself based on built-in system intents, or handled as a different state in the current conversation flow.

[0084] In some embodiments, as part of this processing, the digital assistant determines whether the utterance of the user input explicitly identifies a skill robot using its invocation name. If there is an invocation name in the user input, it is considered an explicit invocation of the skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the skill robot that is explicitly invoked for further processing. If there is no specific or explicit invocation, then in some embodiments, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intent and skill robot associated with the digital assistant. The score calculated for the skill robot or system intent represents the likelihood that the user input represents the task that the skill robot is configured to perform or represents the system intent. Any system intent or skill robot with an associated calculated confidence score that exceeds a threshold (e.g., a confidence threshold routing parameter) will be selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill robot from the identified candidates to further process the utterance of the user input. In some embodiments, after one or more skill robots are identified as candidates, the intents associated with these candidate skills are evaluated (according to the intent model for each skill) and the confidence score for each intent is determined. In general, any intent with a confidence score exceeding a threshold (e.g., 70%) is considered a candidate intent. If a specific skill robot is selected, the user utterance is routed to that skill robot for further processing. If a system intent is selected, one or more actions are performed by the main robot itself according to the selected system intent.

[0085] Figure 2 2 is a simplified block diagram of a master robot (MB) system 200 according to certain embodiments. The MB system 200 can be implemented with software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multi-intention subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data repository 250. Figure 2The MB system 200 depicted in FIG. 1 is merely an example of an arrangement of components in a master robot. A person skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have more than Figure 2 More or fewer systems or components than shown in the drawings may be included, two or more subsystems may be combined, or there may be a different configuration or arrangement of the subsystems.

[0086] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided in audio form, the pre-processing subsystem 210 can convert the audio into text using a speech-to-text converter (not shown), which inserts punctuation into the resulting text, such as commas, semicolons, periods, etc.

[0087] Language detector 212 detects the language of utterance 202 based on the text of utterance 202. The way utterance 202 is processed depends on the language, because each language has its own grammar and semantics. When analyzing the grammar and structure of an utterance, the differences between languages ​​are taken into account.

[0088] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for each language unit (e.g., words) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 can also tokenize the language units of the utterance 202 (e.g., convert each word into a separate token) and lemmatize the words. Lemmatization is the main form of a group of words as represented in a dictionary (e.g., "run" is the lemmatization of run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking of compound expressions, for example, combining "credit" and "card" into a single expression "credit card". The language parser 214 can also identify relationships between words in the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a specific noun) is a direct object, which part of the utterance is a preposition, and so on. The processing performed by the language parser 214 forms the extracted information 205 and is provided as input to the MIS 220 along with the utterance 202 itself.

[0089] As indicated above, the utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, the utterance 202 may be treated as a single unit even though it includes multiple sentences. However, in some embodiments, preprocessing may be performed, for example by the preprocessing subsystem 210, to identify a single sentence among multiple sentences for multiple intent analysis and explicit call analysis. In general, the results produced by the MIS 220 and the EIS 230 are substantially the same regardless of whether the utterance 202 is processed at the level of individual sentences or as a single unit containing multiple sentences.

[0090] MIS 220 determines whether utterance 202 indicates multiple intents. Although MIS 220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS 220 does not involve determining whether the intent of utterance 202 matches any intent that has been configured for the robot. Instead, the processing of determining whether the intent of utterance 202 matches the robot's intent can be performed by intent classifier 242 of MB system 200 or an intent classifier of a skill robot (e.g., Figure 3 ). The processing performed by MIS 220 assumes that there is a robot (e.g., a particular skill robot or the master robot itself) that can handle utterance 202. Therefore, the processing performed by MIS 220 does not require knowledge of which robots are in the chatbot system (e.g., the identities of the skill robots registered with the master robot) or knowledge of which intents have been configured for a particular robot.

[0091] To determine that the utterance 202 contains multiple intents, the MIS 220 applies one or more rules from a set of rules 252 in the data repository 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and may include a sentence pattern indicating the presence of multiple intents. For example, the sentence pattern may contain a coordinating conjunction that connects two parts of the sentence (e.g., a conjunction), each of which corresponds to a separate intent. If the utterance 202 matches the sentence pattern, it can be inferred that the utterance 202 represents multiple intents. It should be noted that utterances with multiple intents do not necessarily have different intents (e.g., intents for different robots or intents for different intents within the same robot). Alternatively, the utterance may have separate instances of the same intent (e.g., "order pizza using payment account X, then order pizza using payment account Y").

[0092] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines which parts of utterance 202 are associated with each intent. For each intent represented in an utterance containing multiple intents, MIS 220 constructs a new utterance to replace the original utterance for separate processing, for example, utterance "B" 206 and utterance "C" 208, as shown in FIG. Figure 2. Thus, the original utterance 202 may be divided into two or more separate utterances, one utterance at a time. MIS 220 uses the extracted information 205 and / or by analyzing the utterance 202 itself to determine which of the two or more utterances should be handled first. For example, MIS 220 may determine that utterance 202 contains a marker word indicating that a particular intent should be handled first. The newly formed utterance corresponding to the particular intent (e.g., one of utterance 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance ends (or is temporarily suspended), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) may then be sent to EIS 230 for processing.

[0093] EIS230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) contains a call name for a skill robot. In some embodiments, each skill robot in the chatbot system is assigned a unique call name that distinguishes the skill robot from other skill robots in the chatbot system. A list of call names can be maintained as part of the skill robot information 254 in the data repository 250. When a utterance contains words that match the call name, the utterance is considered an explicit call. If there is no explicit call to the robot, the utterance received by EIS230 is considered a non-explicit call utterance 234 and is input to the intent classifier of the main robot (e.g., intent classifier 242) to determine which robot to use to handle the utterance. In some cases, intent classifier 242 will determine that the main robot should handle the utterance of the non-explicit call. In other cases, intent classifier 242 will determine to route the utterance to the skill robot for processing.

[0094] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that the master robot must perform. For example, when there is an explicit call, the master robot may not have to perform any intent classification analysis (e.g., using intent classifier 242), or may have to perform a simplified intent classification analysis to select a skill robot. Thus, the explicit call analysis can enable the selection of a particular skill robot without resorting to intent classification analysis.

[0095] Additionally, there may be situations where there is overlap in functionality between multiple skill robots. This may happen, for example, if the intents handled by two skill robots overlap or are very close to each other. In this case, it may be difficult for the master robot to identify which of the multiple skill robots to select based solely on intent classification analysis. In such scenarios, explicit calls can disambiguate the specific skill robot to use.

[0096] In addition to determining that the utterance is an explicit call, the EIS 230 is also responsible for determining whether any part of the utterance should be used as input to the skill robot of the explicit call. In particular, the EIS 230 can determine whether a part of the utterance is irrelevant to the call. The EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. The EIS 230 can send the part of the utterance that is irrelevant to the call to the called skill robot instead of sending the entire utterance received by the EIS 230. In some cases, the input of the called skill robot is formed only by removing any part of the utterance that is relevant to the call. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using PizzaBot" is related to the call of the pizza robot, but is irrelevant to any processing to be performed by the pizza robot. In some cases, the EIS 230 can reformat the part to be sent to the called robot, for example, to form a complete sentence. Therefore, the EIS 230 not only determines that there is an explicit call, but also determines what to send to the skill robot when there is an explicit call. In some cases, there may not be any text to be input to the called robot. For example, if the utterance is "PizzaBot", then EIS 230 can determine that a pizza robot is being called, but there is no text to be processed by the pizza robot. In this scenario, EIS 230 can indicate to the skill robot invoker 240 that there is no content to be sent.

[0097] The skill robot invoker 240 invokes a skill robot in various ways. For example, the skill robot invoker 240 may invoke a robot in response to receiving an indication 235 indicating that a particular skill robot has been selected as a result of an explicit invocation. The indication 235 may be sent by the EIS 230 along with the input for the explicitly invoked skill robot. In this scenario, the skill robot invoker 240 transfers control of the conversation to the explicitly invoked skill robot. The explicitly invoked skill robot will determine the appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0098] Another way that the skill robot invoker 240 can invoke a skill robot is implicitly by using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different categories, one category for each skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. Parameters generated as a result of this training (e.g., a set of values ​​for parameters of a machine learning model) can be stored as part of the skill robot information 254.

[0099] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training of the machine learning model may involve inputting at least a subset of utterances from example utterances associated with various skill robots to generate, as output of the machine learning model, inferences about which robot is the correct robot to handle any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance may be provided as ground truth information. The behavior of the machine learning model may then be adapted (e.g., by backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0100] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot, which indicates the likelihood that the skill robot can handle an utterance (e.g., a non-explicit call utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each system-level intent (e.g., help, exit) that has been configured. If a particular confidence score satisfies one or more conditions, the skill robot caller 240 will call the robot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is the identification of the system intent or the identification of a particular skill robot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must also exceed the next highest confidence score by a certain winning margin. When the confidence scores of multiple skill robots each exceed the threshold confidence score value, applying such a condition will enable routing to a particular skill robot.

[0101] After identifying the robot based on the evaluation of the confidence score, the skill robot invoker 240 hands over processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is a skill robot. In addition, the skill robot invoker 240 will determine what to provide as input 247 to the identified robot. As indicated above, in the case of an explicit invocation, the input 247 can be based on a portion of the utterance that is not relevant to the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invocation, the input 247 can be the entire utterance.

[0102] The data repository 250 includes one or more computing devices that store data used by various subsystems of the master robot system 200. As explained above, the data repository 250 includes rules 252 and skill robot information 254. The rules 252 include, for example, rules for determining when an utterance represents multiple intents by the MIS 220 and how to split utterances that represent multiple intents. The rules 252 also include rules for determining which parts of an utterance that explicitly calls a skill robot are sent to the skill robot by the EIS 230. The skill robot information 254 includes the calling names of the skill robots in the chatbot system, for example, a list of the calling names of all skill robots registered with a particular master robot. The skill robot information 254 may also include information used by the intent classifier 242 to determine the confidence score of each skill robot in the chatbot system, for example, parameters of a machine learning model.

[0103] Figure 3 is a simplified block diagram of a skill robot system 300 according to some embodiments. The skill robot system 300 is a computing system that can be implemented using only software, only hardware, or a combination of hardware and software. In some embodiments, such as Figure 1 In the embodiment depicted in FIG, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0104] The skill robot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to Figure 2 302 and extracted information 304. The extracted information 304 is similar to the MIS 310. Figure 1The extracted information 205 in the skill robot system 300 may be generated using the language parser 214 or a language parser local to the skill robot system 300.

[0105] The intent classifier 320 can be combined with the above Figure 2 In some embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a specific skill robot using at least a subset of example utterances associated with the specific skill robot as training utterances. The ground truth for each training utterance will be the specific robot intent associated with the training utterance.

[0106] The utterance 302 may be received directly from the user or provided by the master robot. When the utterance 302 is provided by the master robot, for example, Figure 2 In the embodiment depicted in , when supplied as a result of processing by MIS 220 and EIS 230, MIS 310 may be bypassed to avoid duplicating processing already performed by MIS 220. However, if utterance 302 is received directly from a user, such as during a conversation that occurs after being routed to a skill robot, then MIS 310 may process utterance 302 to determine whether utterance 302 represents multiple intents. If so, then MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, then MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0107] The intent classifier 320 is configured to match a received utterance (e.g., utterance 306 or 308) with an intent associated with the skill robot system 300. As explained above, a skill robot may be configured with one or more intents, each of which includes at least one example utterance associated with the intent and used to train the classifier. Figure 2In an embodiment, the intent classifier 242 of the master robot system 200 is trained to determine confidence scores for individual skill robots and confidence scores for system intents. Similarly, the intent classifier 320 can be trained to determine confidence scores for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and is therefore more fine-grained. The intent classifier 320 can access intent information 354. The intent information 354 includes a list of utterances for each intent associated with the skill robot system 300, which represent and explain the meaning of the intent and are typically associated with a task that can be performed by the intent. The intent information 354 can also include parameters generated by training the list of utterances.

[0108] The conversation manager 330 receives, as an output of the intent classifier 320, an indication 322 of a particular intent that was identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some cases, the intent classifier 320 is unable to determine any match. For example, if the utterance is for a system intent or an intent for a different skill robot, then the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill robot system 300 may submit the utterance to the master robot for disposal, e.g., routing to a different skill robot. However, if the intent classifier 320 successfully identifies the intent within the skill robot, then the conversation manager 330 will initiate a conversation with the user.

[0109] The conversation initiated by the conversation manager 330 is a conversation specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 can be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine can include a default starting state (e.g., when the intent is invoked without any additional input) and one or more additional states, each of which is associated with an action to be performed by the skill robot (e.g., perform a purchase transaction) and / or a dialog to be presented to the user (e.g., a question, a response). Thus, the conversation manager 330 can determine an action / dialog 335 when receiving an indication 322 of an identified intent, and can determine additional actions or dialogs in response to subsequent utterances received during the conversation.

[0110] The data repository 350 includes one or more computing devices that store data used by the various subsystems of the skill robot system 300. Figure 3 As depicted in , data repository 350 includes rules 352 and intent information 354. In some embodiments, data repository 350 can be integrated into the data repository of a main robot or digital assistant, for example, Figure 2In the data repository 250.

[0111] Figure 4 1 shows a block diagram illustrating aspects of a chatbot system 400 configured to train one or more models based on text data. Figure 4 As shown in , the entity recognition performed by the chatbot system 400 in this example includes various stages: a prediction model training stage 410 for building and training models; an evaluation stage 415 for evaluating the performance of the trained models; and an implementation stage 420 for implementing one or more chatbots. The prediction model training stage 410 builds and trains one or more prediction models 425a-425n ("n" represents any natural number) to be used by other stages (which may be individually referred to as prediction models 425 or collectively referred to as prediction models 425 herein). For example, the prediction model 425 may include a model for encoding an utterance, another model for determining the likelihood that an utterance represents a task that a particular skill robot is configured to perform, another model for predicting the intent of a first type of skill robot from an utterance, and another model for predicting the intent of a second type of skill robot from an utterance. According to the present disclosure, other types of prediction models may be implemented in other examples.

[0112] The prediction model 425 can be a machine learning ("ML") model, such as a convolutional neural network ("CNN"), for example, an initial neural network, a residual neural network ("Resnet"), or a recurrent neural network, for example, a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, other variants of a deep neural network ("DNN") (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single intent classification). The prediction model 425 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model (such as a random forest model), a boosting model, a shallow neural network, or a combination of one or more such techniques - for example, a CNN-HMM or an MCNN (multi-scale convolutional neural network). The chatbot system 400 can employ the same type of predictive model or different types of predictive models to identify one or more entities in an utterance, determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform, predict the intent of a first type of skill robot from an utterance, and predict the intent of a second type of skill robot from an utterance. Still other types of predictive models can be implemented in other examples in accordance with the present disclosure.

[0113] To train the various predictive models 425, the training phase 410 consists of three main components: dataset preparation 430, data augmentation 435, and model training 440. The dataset preparation 430 includes the process of loading data assets 445, splitting the data assets 445 into training and validation sets 445a-n so that the system can train and test the predictive models 425, and performing basic preprocessing. The splitting of the data assets 445 into training and validation sets 445a-n can be performed randomly (e.g., 90 / 10% or 70 / 30%), or the split can be performed according to more complex validation techniques (such as K-fold cross validation, leave one out cross validation, leave one out cross validation, nested cross validation, etc.) to minimize sampling bias and overfitting.

[0114] The training data 445a may include at least a subset of utterances from example utterances associated with one or more skill robots. As indicated above, utterances may be provided in various ways, including audio or text. The utterances may be sentence fragments, complete sentences, multiple sentences, etc. For example, if the utterances are provided in audio form, then the data preparation 430 may convert the audio into text using a speech-to-text converter (not shown), which inserts punctuation marks into the resulting text, such as commas, semicolons, periods, etc. In some cases, the example utterances are provided by a client or customer. In other cases, the example utterances are automatically generated from a previous utterance library (e.g., identifying utterances specific to the skills that the chatbot is designated to learn from the library). The training data 445a of the prediction model 425 may include input text or audio (or input features of text or audio frames) and labels 450 corresponding to the input text or audio (or input features), for example, as a value matrix or table. For example, for each training utterance, an indication of the correct entity, intent, and classification thereof inferred by the prediction model 425 may be provided as basic fact information in the label 450. The behavior of the prediction model 425 may then be adapted (eg, via back-propagation) to minimize the discrepancy between the generated inferences and the ground truth information for various entities.

[0115] Data augmentation 435 includes the process of using training data 445a as anchor utterances to generate positive utterances and negative utterances with multiple sets of contexts and entity values ​​in any number of languages ​​and formats so that the prediction model 425 provides improved utterance representations. By using training data 445a as anchor utterances to generate positive utterances and negative utterances with multiple sets of contexts and entity values ​​in any number of languages ​​and formats, the prediction model 425 is able to better provide representations of utterances. The generation includes multiple operations, which generally include: (i) using the utterances of the training data as anchor utterances and generating positive utterances using translation, perturbation, and / or augmentation operations; and (ii) using the anchor utterances and generating negative utterances using augmentation and / or sampling operations. The anchor utterances, positive utterances, and negative utterances are used together in an unsupervised contrastive loss framework to extract knowledge from a pre-trained sentence encoding model to learn a new sentence encoding model that performs well in intent classification tasks.

[0116] The training process of model 425 includes selecting hyperparameters of model 425 and performing an iterative operation of inputting utterances from a subset of augmented training data 445a into model 425 to find a set of model parameters (e.g., weights and / or biases) that maximize or minimize an objective function (e.g., minimize the loss function of model 425). Hyperparameters are settings that can be adjusted or optimized to control the behavior of model 425. Most models explicitly define hyperparameters that control different aspects of the model (such as memory or execution cost). However, additional hyperparameters can be defined to adapt the model to a specific scenario. For example, hyperparameters can include the number of hidden units or layers of the model, the learning rate of the model, the convolution kernel width, or the number of parameters of the model. Each training iteration may involve finding a set of model parameters for model 425 (configured with a set of defined hyperparameters) so that the value of the objective function using the set of model parameters is less than the value of the objective function using a different set of model parameters in the previous iteration. The objective function can be constructed to measure the difference between the output inferred using model 425 and the ground truth annotated to the subset of augmented training data 445a using labels 150.

[0117] Once the set of model parameters is identified, the model 425 is trained and can be tested or validated using a subset of the test data 445b (test or validation data set). The testing or validation process includes an iterative operation of inputting utterances from a subset of the test data 445b into the model 425 using validation techniques such as K-fold cross validation, leave-one-out cross validation, leave-one-out cross validation, nested cross validation, etc., to adjust the hyperparameters and ultimately find a set of optimal hyperparameters. Once a set of optimal hyperparameters is obtained, a reserved test set from a subset of the test data 445a can be input into the model 425 to obtain an output (one or more identified entities in this example), and the output is evaluated with the basic fact entity using correlation techniques such as the Bland-Altman method and the Spearman ranking correlation coefficient. In addition, performance metrics in the analysis results 452, such as error rate, accuracy, precision, recall rate, receiver operating characteristic curve (ROC), etc., can be calculated in the entity recognition evaluation stage 415. The metrics in the analysis results 452 may be used in the entity recognition evaluation phase 415 to analyze the performance of the model 425 for recognizing entities.

[0118] The model training phase 410 outputs one or more trained models, such as an entity recognition model 455. The entity recognition model 455 can be deployed and used in the prediction model training phase 410 and / or the implementation phase 420 to implement one or more chatbots. For example, the entity recognition model 455 can be used to generate named entities 465 in each set of training data, and the named entities together with the labels 450 (e.g., intent labels) can be used to train other prediction models 425, such as intent prediction models. Alternatively, one or more chatbots can be configured with the trained entity recognition model 455 to receive utterances, such as text data 460, from one or more chatbot users, and extract entities 465 of various utterances received from the one or more chatbot users. The entities 465 can be extracted information obtained from the text data 460 (e.g., respectively in Figure 2 and Figure 3 The extracted information 205; 304) described in , and can be used for downstream processing such as intent classification.

[0119] Entity-aware data enhancement technology

[0120] For the purposes of this application, an entity type may refer to a type of named entity or a category to which a named entity belongs, such as a person, organization, and place. An entity value may refer to a single named entity (a word or phrase) within an entity type. For example, as discussed above with respect to the input utterances "Give Peter a raise" and "Give James a raise", the words Peter and James are entity values, and they both belong to the entity type "person". The sources of entity types may include system entities generated by cloud service providers and custom entities provided by customers.

[0121] There are different methods to perform entity-aware augmentation. It has been found that random entity value augmentation has better improvements in reducing entity sensitivity than augmentation using named entities in the training set. The novel random entity value augmentation technique disclosed in the present disclosure not only applies different entity values ​​on the input utterance and entity normalization, but also generates new sentences in which named entities are replaced by entity values ​​randomly sampled through an augmentation package (e.g., faker python package) with various augmentation rates to achieve optimal performance. Additional balancing (e.g., stratified batch balancing and source balancing) can also be performed on the intent training data. Finally, random entity values ​​can also be applied based on entity type (e.g., person, organization, number, date, etc.). For example, random entity values ​​involving certain entity types (e.g., person) can achieve better performance. All of these techniques help improve the performance of random entity value augmentation. The following table illustrates the improvement comparison between these two different methods:

[0122]

[0123]

[0124] In the above table, the number of improvements for the random named entity augmentation method ranges from 232 to 315, while the number of improvements for the entity augmentation method using named entities in the training set ranges from 108 to 197. The numbers in the Improvement column refer to the total number of utterances correctly predicted by the intent prediction model using a particular method, which exceeds the total number of utterances correctly predicted by the baseline model (i.e., without entity-aware augmentation). The numbers in the Regression column refer to the total number of utterances that were correctly predicted in the baseline model but became incorrect in the new model trained using the new method. The above numbers are cumulative across test datasets. The proposed augmentation techniques described in this disclosure provide good accuracy and have low inference time because they do not require much work during deployment (e.g., no NER model is required at inference time). The training time may be longer, but the inference time should be the same.

[0125] In certain embodiments, data augmentation can expand a data set by generating synthetic samples using the underlying data distribution. The entity-aware data augmentation techniques described in the present disclosure include positive and negative entity-aware data augmentation. Positive data augmentation creates in-domain samples, while negative data augmentation creates out-of-domain (or out-of-distribution (OOD)) samples. Positive entity-aware augmentation techniques address the stability of the model for intent prediction to changes in entity values. Positive entity-aware data augmentation has two modes of operation: diversification mode and enhanced diversification (AD) mode.

[0126] The diversified mode post-processes a subset of positive enhancements by performing an entity value replacement operation on positive enhancement utterances from enhancement techniques such as simple data augmentation ("EDA") and noise augmentation or a combination thereof, and does not introduce a separate additional training data bucket. In some embodiments, the replacement operation may include replacing at least one or more entity values ​​of a named entity with a random value of the same entity type. In a non-limiting example of the diversified mode, for the EDA enhanced utterance "grantPeter a promotion", the word "Peter" can also be replaced with "Luke" to generate the utterance "grant Luke a promotion". In another example, for the noise enhanced utterance "Melbourne is a city; give Peter araise on August 20th", the words "Peter" and "August 20th" can also be replaced with "Sara" and "nextFriday" to generate the utterance "Melbourne is a city; give Sara a raise on next Friday". In other words, the diversified mode can be considered as a two-stage enhancement technique, where the first stage includes forward enhancement using techniques such as EDA and noise, and the second stage includes additional replacement operations (i.e., post-processing) on ​​the enhanced data. EDA and noise techniques are described in more detail below.

[0127] In some embodiments, the EDA generates positive utterances from input utterances in the training data using one of the following operations: synonym replacement (i.e., randomly selecting n words that are not stop words from the sentence and replacing each of these words with one of its synonyms); random insertion (i.e., finding a random synonym for a random word in the sentence and inserting the synonym into a random position in the sentence); random swap (i.e., randomly selecting two words in the utterance and swapping their positions); and random deletion (i.e., randomly removing each word in the sentence with probability p). For example, for the input utterance "give Peter araise", the word "give" can be replaced with the synonym "grant" to generate the utterance "grant Peter a raise". Similarly, for the input utterance "give Peter a raise on August 20th", the word "August 20th" can be removed to generate the utterance "give Peter a raise on".

[0128] Noise augmentation generates positive utterances from input utterances in the training data by adding one or more words to the input utterances (i.e., adding noisy in-domain context). For example, for the input utterance "give Peter a raise on August 20th", the words "Melbourne is a city" can be added before "give Peter a raise on August 20th" to generate the positive utterance "Melbourne is a city; give Peter a raise on August 20th".

[0129] On the other hand, the enhanced diversity (AD) mode includes two parts, forward EA enhancement and diversified mode enhancement. In some embodiments, the entity awareness (EA) enhancement part performs a random entity value replacement operation on all detected named entities in the input utterance by randomly sampling entity values ​​from a repository in the domain, and introduces the generated utterance as additional EA enhanced training data (i.e., the mode introduces a separate additional training data bucket). As discussed above, the diversified mode enhancement part is also performed on existing enhanced utterances from EDA, noise, or a combination thereof. Therefore, the AD mode can generate multiple enhanced utterance training sets-one from EA enhancement, another or other from post-processing EDA and / or noise enhancement. In some embodiments, the EA enhancement part and the first stage of the diversified mode (e.g., EDA, noise, or a combination) can be executed in parallel, and then the second stage of the diversified mode is executed (i.e., post-processing the EDA and / or noise enhancement data).

[0130] In addition, in some embodiments, EA enhancement may include an enhancement rate configuration, thereby allowing multiple operations of the EA technique to be performed. For example, when the enhancement rate of the AD mode is set to 2, for the input utterance "grant Peter a promotion", the configuration rate 2 can perform EA enhancement to generate two utterances: "grant James a raise" and "grant Sara araise", as well as the utterance "grant Luke a promotion" of the diversified mode (discussed above about the diversified mode that replaces Peter with Luke only). The AD mode is a comprehensive application of the EA data enhancement method. However, the choice of which mode to use can be a hyperparameter, or controlled by or dependent on a hyperparameter that can be adjusted. For example, when the training data is small, additional diversified mode enhancement in addition to EA enhancement may be helpful. In some cases, hyperparameter adjustment may only require diversified mode. Since the EA enhancement technology introduces more entity values ​​to the training model, the training model can learn to treat entity values ​​equally to solve the entity value sensitivity problem. In other words, after the training model is trained using EA, the two input utterances "give Peter a raise on August 20th" and "give John a raise on September 4th" should produce the same output prediction.

[0131] Negative augmentation techniques address the second challenge, where machine learning models use entity values ​​as shortcuts for prediction. Therefore, the techniques described in this article introduce entity values ​​in the training data as OOD data, so the training model can learn that these out-of-domain entity values ​​are irrelevant to the in-domain intent. Negative entity-aware data augmentation can include KAT plus context-free entities, and KAT plus entities in random context as OOD. Negative data augmentation without entity awareness can include KAT, SWAT, and OOD, and can improve the performance of the model in out-of-domain testing and sensitivity to stop words.

[0132] Keyword Augmentation Technology (“KAT”) is a negative augmentation technique and can be entity-aware or entity-unaware. Entity-unaware KAT augmentation samples utterances containing keywords that are present in the training data in the domain but in a different context from the corpus and adds them as OOD. The goal of KAT is to minimize the pull of keywords that are present in the training data in a different OOD context. For example, for the training utterance “What is a dynamic entity”, KAT might identify the words “dynamic” and “entity” as keywords, thereby augmenting the training set with the following negative utterances: “but the information on the dynamics of population was often quite misleading”, and “does society really exist as an entity over and above the agglomeration of men?”

[0133] KAT plus context-free augmented entities generates entity values ​​that appear as negative out-of-domain examples in the training data by removing the in-domain context information around the named entities, so the trained model may not notice specific entity values ​​because they appear as out-of-domain examples in addition to in-domain examples. For example, for the input utterance "give Peter araise on August 20th", KAT plus context-free augmented entities generates negative examples by removing the context ("give" and "raise") around the entity value, and the resulting utterance becomes "Peter August 20th".

[0134] KAT plus entities in random context as OOD enhancement generates entity values ​​in random context that appear in the training data as out-of-domain examples. In some embodiments, the random context is sampled from an out-of-domain corpus. The out-of-domain corpus has tokens such as person names (i.e., person entity types), and then random contexts are generated by replacing entity values ​​based on entity types (e.g., replacing tokens of Person type with entity values ​​"Peter"). Since entity values ​​are not associated with categories in the domain, this enhancement can help break the association with intent categories in the domain. For example, for the input utterance "create$20expense for my uber ride to the airport", the entity value "$20" can be placed in a random context to generate the utterance "the minimum wage in somecities is$20".

[0135] In some embodiments, the utterance examples generated by KAT enhancement should be away from the utterances of the in-domain examples by ensuring that the distance between the embedding vectors of the in-domain examples and the out-of-domain examples is above a threshold so that the training model does not observe positive and negative data from the same intent to avoid confusing the training model. The embedding vector is created by putting the utterance example into the language model. For example, given an input utterance for which a KAT version is generated, two utterances (i.e., an in-domain example and an out-of-domain example) are passed through the language model to generate an embedding vector. The distance between the two embedding vectors should be above a threshold. The threshold can be a hyperparameter that can be adjusted to find the optimal distance.

[0136] SWAT generates discourse by identifying all stop words and non-stop words in input discourse in training data, retaining stop words in input discourse and replacing all non-stop words in input discourse with random stop words. Stop words generally refer to those words that do not contribute to the meaning of phrases, sentences and / or discourse in phrases, sentences and / or discourse. However, in natural language processing (" NLP ") applications, stop words generally refer to those words that are filtered out from discourse in phrases, sentences and / or discourse to make named entity recognition and / or intention classification model pay attention to entity value and other context information in discourse. Stop word list can be provided by user or customer and / or can be obtained by publicly available and well-known stop word list of given language. For example, for input discourse " do you offer a one day trial membership ", by removing non-stop words " offer ", " day ", " trial " and " membership ", and inserting stop words " to ", " say ", " a " and " likely ", generate discourse " do you to a one say a likely ". In this way, the stop word pattern in input discourse is retained. By replacing all non-stop words with stop words, the meaning and intent category of the stop word enhanced utterance is different from that of the input utterance.

[0137] OOD enhancement is a negative enhancement technique, but is not entity-aware (i.e., entity value replacement is not performed on the OOD bucket), and it identifies one or more utterances that are not in the domain (i.e., related to a specific domain or a specific task) and treats these one or more utterances as negative utterances. For example, an utterance may correspond to one or more different domains. In some cases, an utterance in one domain may be treated as a negative utterance. In some cases, an utterance in two or more domains may be treated as a negative utterance. The foregoing examples are not intended to be limiting. A user or customer may select one or more domains. Alternatively or additionally, one or more domains may be automatically selected depending on a specific task or domain of a chatbot or intent classifier. Thus, utterances that are not in the domain (i.e., not in one or more desired domains) are treated as negative utterances. For example, for the selected domain food ordering, anchor utterances in the training corpus that do not belong to the food ordering domain may be treated as negative utterances. The utterances in the training corpus may be annotated with one or more domains. Thus, these labels may be referenced when determining one or more domains of an utterance. Alternatively or additionally, domain recognition models and / or other models known in the art may be used to determine one or more domains.

[0138] Figure 5 is a simplified flow chart illustrating the process of overall tuning of a predictive model using entity-aware data augmentation. Figure 5The processing depicted in the can be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 5 The process 500 presented in and described below is intended to be illustrative and not limiting. Figure 5 Various processing steps are depicted as occurring in a particular order or sequence, but this is not limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1-4 In the embodiment depicted in FIG. Figure 5 The processing depicted in can be performed by a pre-processing subsystem (e.g., pre-processing subsystem 210 or prediction model training stage 410).

[0139] At step 502, a data processing system (eg, Figure 4 The chatbot system 400 described in the foregoing description obtains a set of data containing utterances for training an intent prediction model. At step 504, the set of data is split into a set of training data and a set of test and / or validation data (evaluation data sets as targets for one or more prediction models). The training data includes utterances and intent labels. At step 506, named entities within the set of training data are extracted based on entity types supported by the NER model to prepare for data enhancement. Depending on the type and mode of data enhancement to be applied considering the hyperparameter requirements of the model, the process may use one or more or a combination of several enhancement techniques and proceed according to the different processes to be discussed below.

[0140] For positive enhancement only, any one or more of steps EA 510, EDA 520, and noise 522 may be applied to generate enhanced training data (or utterances). At step 550, a prediction model (e.g., an intent prediction model) is trained for the set of extended training data in block 502 and the enhanced training data generated by these steps (i.e., one or more of 510, 520, and 522) to learn parameters for recognizing and classifying entities within utterances. These steps may be performed in parallel or in some desired order.

[0141] For the forward enhancement diversified mode only, the process goes to step EDA 520 or step noise 522 or both, and then the diversified mode enhancement step 540 to generate enhanced training data (or utterances). At step 550, the prediction model is trained for the set of extended training data in box 502 and the enhanced training data through these steps (i.e., 520 & 540 and / or 522 & 540) to learn parameters for classifying the intent expressed in the utterance. As discussed above with respect to the diversified mode, the result of step 540 is the post-processing result of EDA enhancement or noise enhancement, not new training data.

[0142] For the forward enhanced AD mode, step EA 510 and the above-mentioned diversification mode steps (i.e., 520 & 540 and / or 522 & 540) are applied to generate enhanced training data (or utterances). In some embodiments, step EA 510 and the first stage of the diversification mode (520 and / or 522) can be performed in parallel and simultaneously, and then step 540 is performed. At step 550, a prediction model is trained for the set of extended training data in box 502 and the enhanced training data enhanced by EA and diversification mode (e.g., post-processed EDA and / or post-processed noise) to learn parameters for classifying the intent expressed in the utterance.

[0143] For negative enhancement, any one or more of step SWAT 528 and KAT-related enhancement 530 may be applied to generate enhanced training data (or utterances). Step 530 may include KAT, KAT plus context-free entities, KAT plus entities in random context as OOD, and OOD enhancement. One or more of these enhancement techniques may be applied in parallel or in a specific order to generate enhanced training data (or utterances). In step 550, a prediction model is trained for the set of expanded training data in box 502 and the enhanced training data through step 528 and / or step 530.

[0144] The steps of positive enhancement and negative enhancement, such as 510 - 530 , may be performed in parallel, but step 540 as a post-processing step should be performed after the EDA step 520 and the noise step 522 have been performed.

[0145] At 560, the one or more predictive models are tested and / or validated on one or more sets of testing and / or validation data for intent classification. The testing and / or validation is performed and analyzed to obtain a revised evaluation of the one or more predictive models and, if any, to obtain an understanding of additional information (e.g., additional nouns or text formats) that needs to be added to the one or more sets of training data to improve the performance of the one or more predictive models in classifying intents expressed in utterances in various languages ​​and / or text formats.

[0146] Thereafter, based on the evaluation of the training and testing of the one or more predictive models, steps 510-560 may be iteratively repeated until the evaluation demonstrates that the one or more predictive models have optimal performance for classifying intents expressed in utterances in various languages ​​and / or text formats. Once the one or more predictive models are trained and / or achieve optimal performance as determined by various tools. At step 570, the trained predictive models may be deployed within a chatbot system (e.g., as part of a skill bot) to classify intents expressed in utterances in various languages ​​and / or text formats. For example, an utterance may be received and the utterance may be analyzed by one or more predictive models to determine whether the utterance belongs to one of the intents.

[0147] Descriptive System

[0148] Figure 6 Depicted is a simplified diagram of a distributed system 600. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608 coupled to a server 612 via one or more communication networks 610. The client computing devices 602, 604, 606, and 608 may be configured to execute one or more applications.

[0149] In various examples, server 612 may be adapted to run one or more services or software applications that enable one or more embodiments described in the present disclosure. In some examples, server 612 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided to users of client computing devices 602, 604, 606, and / or 608 as web-based services or cloud services, such as under a software as a service (SaaS) model. Users operating client computing devices 602, 604, 606, and / or 608 may then interact with server 612 using one or more client applications to utilize the services provided by these components.

[0150] exist Figure 6 In the depicted configuration, the server 612 may include one or more components 618, 620, and 622 that implement the functions performed by the server 612. These components may include software components, hardware components, or a combination thereof that may be executed by one or more processors. It should be appreciated that a variety of different system configurations are possible that may differ from the distributed system 600. Thus, Figure 6 The example shown in is one example of a distributed system for implementing the example system and is not intended to be limiting.

[0151] A user may use client computing devices 602, 604, 606, and / or 608 to execute one or more applications, models, or chatbots, which may generate one or more events or models, which may then be implemented or serviced in accordance with the teachings of the present disclosure. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although Figure 6 Only four client computing devices are depicted, but any number of client computing devices may be supported.

[0152] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors and other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, Linux or Linux-like operating systems such as Google Chrome TM OS), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM , Palm Portable handheld devices may include cellular phones, smart phones (e.g., ), tablets (e.g. ), personal digital assistants (PDAs), etc. Wearable devices can include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Gesture Input Devices Microsoft Game consoles, Sony System, by Various game systems provided by and others), etc. The client device may be able to execute a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use various communication protocols.

[0153] The network(s) 610 may be any type of network familiar to those skilled in the art, and may support data communications using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internetwork Packet Exchange), The network(s) 610 may be, by way of example only, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., in any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suites, and / or any other wireless protocol operating under the network) and / or any combination of these and / or other networks.

[0154] The server 612 may be composed of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, The server 612 may be a server, a mid-range server, a mainframe computer, a rack-mounted server, etc.), a server farm, a server cluster, or any other suitable arrangement and / or combination. The server 612 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain a virtual storage device of the server. In various examples, the server 612 may be suitable for running one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0155] The computing system in server 612 may run one or more operating systems, including any of the operating systems discussed above as well as any commercially available server operating system. Server 612 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines) and other commercially available database servers.

[0156] In some implementations, server 612 may include one or more applications to analyze and integrate data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. As examples, data feeds and / or event updates may include, but are not limited to, data received from one or more third-party information sources and continuous data streams. feed, Updates or real-time updates, which may include real-time events related to sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 612 may also include one or more applications that display data feeds and / or real-time events via one or more display devices of client computing devices 602, 604, 606, and 608.

[0157] The distributed system 600 may also include one or more data repositories 614, 616. In some examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 614, 616 may be used to store information, such as information related to chatbot performance or generated models, for use by the chatbot used by the server 612 when performing various functions according to various embodiments. The data repositories 614, 616 may reside in various locations. For example, the data repository used by the server 612 may be local to the server 612 or may be remote from the server 612 and communicate with the server 612 via a network-based connection or a dedicated connection. The data repositories 614, 616 may be of different types. In some examples, the data repository used by the server 612 may be a database, such as a relational database, such as an Oracle database. Databases provided by companies and other suppliers. One or more of these databases may be adapted to respond to SQL formatted commands to store, update, and retrieve data in the database.

[0158] In some examples, the application may also use one or more of the data repositories 614, 616 to store application data. The data repositories used by the application may be of different types, such as, for example, a key-value storage repository, an object storage repository, or a general storage repository backed by a file system.

[0159] In some examples, the functionality described in this disclosure can be provided as a service via a cloud environment. Figure 7 is a simplified block diagram of a cloud-based system environment in which various services can be provided as cloud services according to certain examples. Figure 7 , cloud infrastructure system 702 may provide one or more cloud services that may be requested by users using one or more client computing devices 704, 706, and 708. Cloud infrastructure system 702 may include one or more computers and / or servers, which may include those described above for server 612. The computers in cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0160] The network(s) 710 may facilitate communication and data exchange between the clients 704, 706, and 708 and the cloud infrastructure system 702. The network(s) 710 may include one or more networks. The networks may be of the same or different types. The network(s) 710 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0161] Figure 7 The example depicted in FIG. 7 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in some other examples, the cloud infrastructure system 702 may have more Figure 7 The components shown may be more or fewer components, may be combined with two or more components, or may have a different configuration or arrangement of components. Figure 7 Three client computing devices are depicted, but any number of client computing devices may be supported in alternative examples.

[0162] The term "cloud service" is generally used to refer to services that are made available to users on demand by a service provider's system (e.g., cloud infrastructure system 702) and via a communications network such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own internal servers and systems. The cloud service provider's system is managed by the cloud service provider. Customers can therefore take advantage of cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can order and use the applications on demand via the Internet without having to purchase infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle Corp. of Redwood Shores, California, offers cloud services. The company provides several cloud services such as middleware services, database services, Java cloud services, etc.

[0163] In some examples, cloud infrastructure system 702 can provide one or more cloud services using different models such as software as a service (SaaS) model, platform as a service (PaaS) model, infrastructure as a service (IaaS) model, and other models including hybrid service models. Cloud infrastructure system 702 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0164] The SaaS model enables applications or software to be delivered to customers as a service over a communication network such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system 702. Examples of SaaS services provided by the Company include, but are not limited to, services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and others.

[0165] The IaaS model is typically used to provide infrastructure resources (e.g., servers, storage devices, hardware, and networking resources) to customers as cloud services to provide elastic computing and storage capabilities. The company provides a variety of IaaS services.

[0166] The PaaS model is typically used to provide platform and environment resources as a service, which enables customers to develop, run, and manage applications and services without having to purchase, build, or maintain such resources. Examples of PaaS services provided by the Company include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

[0167] Cloud services are typically provided on an on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure basis. For example, a customer may order one or more services provided by the cloud infrastructure system 702 via a subscription order. The cloud infrastructure system 702 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use speech to request that the cloud infrastructure system take an action (e.g., an intent), as described above, and / or provide services for the chatbot system described herein. The cloud infrastructure system 702 may be configured to provide one or even multiple cloud services.

[0168] The cloud infrastructure system 702 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 702 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In certain other examples, under a private cloud model, the cloud infrastructure system 702 can be operated within an organization (e.g., within an enterprise organization) and provide services to customers within the organization. For example, customers can be various departments of an enterprise, such as a human resources department, a payroll department, etc., or even individuals within an enterprise. In certain other examples, under a community cloud model, the cloud infrastructure system 702 and the services provided can be shared by several organizations in a related community. Various other models can also be used, such as a mix of the models mentioned above.

[0169] Client computing devices 704, 706, and 708 may be of different types (such as Figure 6 702) and may be capable of operating one or more client applications. A user may use a client device to interact with the cloud infrastructure system 702, such as to request services provided by the cloud infrastructure system 702. For example, a user may use a client device to request information or actions from a chatbot, as described in the present disclosure.

[0170] In some examples, the processing performed by the cloud infrastructure system 702 to provide services may involve model training and deployment. Such analysis may involve using, analyzing, and manipulating data sets to train and deploy one or more models. Such analysis may be performed by one or more processors, possibly processing data in parallel, using data to perform simulations, and the like. For example, big data analysis may be performed by the cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used for such analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0171] As in Figure 7 As depicted in the example of FIG. 7 , cloud infrastructure system 702 may include infrastructure resources 730 for facilitating the provision of various cloud services provided by cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that may be used to service storage requested from an application may be part of cloud infrastructure system 702. In other examples, the storage virtual machine may be part of a different system.

[0172] In some examples, in order to facilitate efficient provisioning of these resources to support various cloud services provided by cloud infrastructure system 702 for different customers, resources can be bundled into groups of resources or resource services (also referred to as "pods"). Each resource service or cluster can include a pre-integrated and optimized combination of one or more types of resources. In some examples, different clusters can be pre-provisioned for different types of cloud services. For example, a first group of clusters can be provisioned for database services, and a second group of clusters can be provisioned for Java services, where the second group of clusters can include a different combination of resources than the pods in the first group of clusters. For some services, resources allocated to provision the service can be shared between services.

[0173] The cloud infrastructure system 702 itself may internally use services 732 that are shared by different components of the cloud infrastructure system 702 and facilitate the service provisioning of the cloud infrastructure system 702. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, and the like.

[0174] The cloud infrastructure system 702 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. Figure 7 As shown, the subsystem may include a user interface subsystem 712 that enables users or customers of the cloud infrastructure system 702 to interact with the cloud infrastructure system 702. The user interface subsystem 712 may include a variety of different interfaces, such as a web interface 714, an online store interface 716 (where cloud services provided by the cloud infrastructure system 702 are advertised and available for purchase by consumers), and other interfaces 718. For example, a customer may use a client device to request (service request 734) one or more services provided by the cloud infrastructure system 702 using one or more of the interfaces 714, 716, and 718. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 702, and place an order for one or more services provided by the cloud infrastructure system 702 that the customer wishes to subscribe to. The service request may include information identifying the customer and the one or more services that the customer wishes to subscribe to. For example, a customer may place an order for services provided by the cloud infrastructure system 702. As part of the order, the customer may provide information identifying a chatbot system for which services are to be provided and optionally one or more credentials for the chatbot system.

[0175] In some examples, such as Figure 7In the example depicted in , cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this processing, OMS 720 may be configured to: create an account for a customer (if not already created); receive billing and / or accounting information from the customer, which will be used to bill the customer for providing the requested service to the customer; verify the customer information; upon verification, book the order for the customer; and orchestrate various workflows to prepare the order for supply.

[0176] Once properly authenticated, the OMS 720 may call an order provisioning subsystem (OPS) 724, which is configured to provision resources for the order, including processing resources, memory resources, and networking resources. Provisioning may include allocating resources to the order, and configuring resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service that the customer has ordered. For example, according to a workflow, the OPS 724 may be configured to determine the specific cloud service requested, and identify multiple clusters that may have been pre-configured for the specific cloud service. The number of clusters allocated for the order may depend on the size / quantity / level / scope of the requested service. For example, the number of clusters to be allocated may be determined based on the number of users supported by the service, the duration of the service being requested, and the like. The allocated clusters may then be customized for the specific requesting customer to provide the requested service.

[0177] In some examples, the setup phase process described above can be performed as part of the provisioning process by cloud infrastructure system 702. Cloud infrastructure system 702 can generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by cloud infrastructure system 702 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 702.

[0178] The cloud infrastructure system 702 may send a response or notification 744 to the requesting client to indicate when the requested service is now ready for use. In some cases, information (e.g., a link) may be sent to the client to enable the client to begin using and taking advantage of the benefits of the requested service. In some examples, for the client requesting the service, the response may include a chatbot system ID generated by the cloud infrastructure system 702 and information identifying the chatbot system selected by the cloud infrastructure system 702 for the chatbot system corresponding to the chatbot system ID.

[0179] The cloud infrastructure system 702 can provide services to multiple customers. For each customer, the cloud infrastructure system 702 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the order, and providing the requested services to the customer. The cloud infrastructure system 702 can also collect usage statistics about the customer's use of the subscribed services. For example, statistics can be collected for the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information can be used to bill the customer. Billing can be performed, for example, on a monthly cycle.

[0180] The cloud infrastructure system 702 may provide services to multiple customers in parallel. The cloud infrastructure system 702 may store information of these customers, including potentially proprietary information. In some examples, the cloud infrastructure system 702 includes an identity management subsystem (IMS) 728 that is configured to manage customer information and provide separation of managed information so that information associated with one customer is not accessible to another customer. The IMS 728 may be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles and related capabilities, and the like.

[0181] Figure 8 An example of a computer system 800 is shown. In some examples, the computer system 800 can be used to implement any digital assistant or chatbot system in a distributed environment and within the various servers and computer systems described above. Figure 8 As shown, computer system 800 includes various subsystems, including a processing subsystem 804 that communicates with multiple other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. The storage subsystem 818 may include non-transitory computer-readable storage media, including storage media 822 and system memory 810.

[0182] The bus subsystem 802 provides a mechanism for enabling the various components and subsystems of the computer system 800 to communicate with each other as desired. Although the bus subsystem 802 is schematically shown as a single bus, alternative examples of the bus subsystem can utilize multiple buses. The bus subsystem 802 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard, etc.

[0183] The processing subsystem 804 controls the operation of the computer system 800 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core or multi-core processor. The processing resources of the computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 804 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing subsystem 804 may be implemented using custom circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0184] In some examples, the processing units in the processing subsystem 804 can execute instructions stored in the system memory 810 or on the computer-readable storage medium 822. In various examples, the processing units can execute various programs or code instructions, and can maintain multiple concurrently executed programs or processes. At any given time, some or all of the program code to be executed may reside in the system memory 810 and / or on the computer-readable storage medium 822, including possibly on one or more storage devices. Through appropriate programming, the processing subsystem 804 can provide the various functions described above. In the case where the computer system 800 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0185] In some examples, a processing acceleration unit 806 may optionally be provided for performing customized processing or for offloading some of the processing performed by the processing subsystem 804 , thereby speeding up the overall processing performed by the computer system 800 .

[0186] The I / O subsystem 808 may include devices and mechanisms for inputting information to the computer system 800 and / or for outputting information from or via the computer system 800. In general, the use of the term "input device" is intended to include all possible types of devices and mechanisms for inputting information to the computer system 800. User interface input devices may include, for example, keyboards, pointing devices such as a mouse or trackball, touch pads or touch screens incorporated into a display, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include software such as Microsoft Windows that enables a user to control and interact with input devices. Motion sensing and / or gesture recognition devices, Microsoft 360 game controllers, devices that provide an interface for receiving input using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as detecting eye activity from a user (e.g., a "wink" when taking a picture and / or making a menu selection) and translating eye gestures to input devices (e.g., Google ) In addition, the user interface input device may include a user interface that enables the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0187] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mouse, joystick or indicator rod, game board and graphic tablet, and audio / video equipment, such as loudspeaker, digital camera, digital video camera, portable media player, webcam, image scanner, fingerprint scanner, bar code reader 3D scanner, 3D printer, laser rangefinder, and eye tracking device. In addition, user interface input devices can include, for example, medical imaging input devices, such as computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound examination equipment. User interface input devices can also include, for example, audio input devices, such as MIDI keyboard, digital musical instrument, etc.

[0188] In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 800 to a user or other computers. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as one utilizing a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0189] The storage subsystem 818 provides a repository or data repository for storing information and data used by the computer system 800. The storage subsystem 818 provides a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide some of the example functionality. The storage subsystem 818 can store software (e.g., programs, code services, instructions) that, when executed by the processing subsystem 804, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 804. The storage subsystem 818 can also provide authentication according to the teachings of the present disclosure.

[0190] The storage subsystem 818 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. Figure 8 As shown, the storage subsystem 818 includes a system memory 810 and a computer-readable storage medium 822. The system memory 810 may include multiple memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) may typically be stored in ROM, which contains basic routines that help transfer information between elements within the computer system 800, such as during startup. RAM typically contains data and / or program services currently operated and executed by the processing subsystem 804. In some embodiments, the system memory 810 may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0191] By way of example and not limitation, Figure 8As shown, system memory 810 may load application programs 812 being executed, which may include various applications (such as a web browser, a middle-tier application, a relational database management system (RDBMS)), program data 814, and an operating system 816. As an example, operating system 816 may include various versions of Microsoft Apple and / or Linux operating system, various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems such as iOS, Phone, OS, OS, OS operating system, and other operating systems.

[0192] Computer-readable storage media 822 may store programming and data structures that provide some example functionality. Computer-readable media 822 may provide storage for computer-readable instructions, data structures, program services, and other data for computer system 800. When executed by processing subsystem 804, the software (programs, code services, instructions) that provide the above functionality may be stored in storage subsystem 818. As an example, computer-readable storage media 822 may include non-volatile memory, such as a hard drive, a disk drive, a hard drive such as a CD ROM, a DVD, (Blu-ray) discs or other optical media. Computer readable storage media 822 may include, but are not limited to, The computer-readable storage medium 822 may include a solid-state drive (SSD) based on a nonvolatile memory (such as a flash memory-based SSD, an enterprise flash drive, a solid-state ROM, etc.), a volatile memory-based SSD (such as a solid-state RAM, dynamic RAM, static RAM, DRAM-based SSD, a magnetoresistive RAM (MRAM) SSD), and a hybrid SSD using a combination of a DRAM-based and a flash memory-based SSD.

[0193] In some examples, storage subsystem 818 may also include a computer-readable storage medium reader 820, which may also be connected to computer-readable storage medium 822. Reader 820 may receive and be configured to read data from a memory device such as a disk, a flash drive, or the like.

[0194] In some examples, computer system 800 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, computer system 800 can provide support for executing one or more virtual machines. In some examples, computer system 800 can execute programs such as hypervisors that promote the configuration and management of virtual machines. Memory, computing (e.g., processors, cores), I / O and networking resources can be allocated to each virtual machine. Each virtual machine is usually independent of other virtual machines. Virtual machines usually run their own operating system, which can be the same or different from the operating system executed by other virtual machines executed by computer system 800. Thus, computer system 800 can potentially run multiple operating systems simultaneously.

[0195] The communication subsystem 824 provides an interface to other computer systems and networks. The communication subsystem 824 is used as an interface for receiving data from the computer system 800 and transmitting data to other systems. For example, the communication subsystem 824 can enable the computer system 800 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and sending information to the client devices. For example, when the computer system 800 is used to implement Figure 1 When the robot system 120 is depicted, the communication subsystem can be used to communicate with the chat robot system selected for the application.

[0196] The communication subsystem 824 can support both wired and / or wireless communication protocols. In some examples, the communication subsystem 824 may include a radio frequency (RF) transceiver component, a global positioning system (GPS) receiver component, and / or other components for accessing a wireless voice and / or data network (e.g., using cellular phone technology, advanced data network technology (such as 3G, 4G or EDGE (Enhanced Data Rates for Global Evolution), Wi-Fi (IEEE 802.XX family of standards), or other mobile communication technologies, or any combination thereof). In some examples, in addition to or in lieu of a wireless interface, the communication subsystem 824 may provide a wired network connection (e.g., Ethernet).

[0197] The communication subsystem 824 can receive and transmit data in various forms. In some examples, the communication subsystem 824 can receive incoming communications in the form of structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., among other forms. For example, the communication subsystem 824 can be configured to receive real-time data from users of social media networks and / or other platforms such as feed, The data feed 826 is received (or sent) by other communication services such as web feeds, updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0198] In some examples, the communication subsystem 824 can be configured to receive data that may be continuous or unbounded in nature without a clear end in the form of a continuous data stream, where the continuous data stream can include an event stream 828 of real-time events and / or event updates 830. Examples of applications that generate continuous data can include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0199] The communication subsystem 824 may also be configured to transmit data from the computer system 800 to other computer systems or networks. The data may be transmitted in a variety of different forms, such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc. to one or more databases that may communicate with one or more streaming data sources coupled to the computer system 800.

[0200] Computer system 800 may be one of various types, including a handheld portable device (e.g., Cellular phone, computing tablets, PDAs), wearable devices (e.g., Google head mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, Figure 8 The description of the computer system 800 depicted in FIG. 8 is intended to be a specific example only. Figure 8 Many other configurations of more or fewer components of the systems depicted are possible. Based on the disclosure and teachings provided herein, it should be appreciated that there are other ways and / or methods to implement the various examples.

[0201] Although specific examples have been described, various modifications, changes, alternative constructions, and equivalents are possible. Examples are not limited to operations within certain specific data processing environments, but can be freely operated within multiple data processing environments. In addition, although certain examples have been described using a series of specific transactions and steps, it is clear to those skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as sequential processing, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. Processing may have other steps not included in the figure. The various features and aspects of the above examples can be used alone or in combination.

[0202] In addition, although certain examples have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are also possible. Certain examples may be implemented using only hardware or only software or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0203] Where a device, system, component or service is described as being configured to perform certain operations or functions, such configuration may be achieved, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations, such as by executing computer instructions or code, or by a processor or core programmed to execute code or instructions stored on a non-transitory storage medium, or any combination thereof to perform the operations. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0204] Specific details are given in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description only provides illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. On the contrary, the previous description of the examples will provide an enabling description for implementing various examples for those skilled in the art. Various changes can be made to the functions and arrangements of the elements.

[0205] Thus, the description and drawings should be regarded as illustrative rather than restrictive. However, it will be clear that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, although specific examples have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0206] In the foregoing description, aspects of the present disclosure are described with reference to their specific examples, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects disclosed above can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, examples can be used in any number of environments and applications beyond the environments and applications described herein. Thus, this specification and the accompanying drawings are considered to be illustrative rather than restrictive.

[0207] In the foregoing description, for the purpose of illustration, methods are described in a particular order. It should be appreciated that, in alternative examples, these methods may be performed in an order different from the described order. It should also be appreciated that the above methods may be performed by hardware components or may be implemented in a sequence of machine executable instructions, which may be used to make a machine, such as a general or special processor or a logic circuit executed by instruction programming. These machine executable instructions may be stored on one or more machine-readable media, such as CD-ROM or other types of optical disks, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, these methods may be performed by a combination of hardware and software.

[0208] Where a component is described as being configured to perform certain operations, such configuration may be accomplished by, for example, designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0209] While illustrative examples of the present application have been described herein in detail, it should be understood that the inventive concept may be otherwise variously embodied and employed, and that the appended claims are intended to be interpreted to include such variations, except as limited by the prior art.

[0210] The embodiments may be implemented by using a computer program product comprising a computer program / instructions which, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure.

Claims

1. A method comprising: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using a trained named entity recognition (NER) model; applying one or more data augmentation techniques to the first set of training data to generate a second set of training data; applying an additional augmentation technique to augment the second set of training data to create post-processed augmented training data, wherein the additional enhancement technique includes replacing at least one or more entity values ​​of the named entities in the second set of training data with random values ​​of the same entity type; combining the first set of training data and the post-processed enhanced training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

2. The method of claim 1, wherein the one or more data enhancement techniques include: A portion of the first set of training data is randomly replaced with a synonym, or one or more named entities in the first set of training data are randomly deleted.

3. The method of claim 1, wherein the one or more data enhancement techniques include: Randomly insert synonyms in random positions in the first set of training data, or randomly swap positions of two named entities within the first set of training data.

4. The method of claim 1, wherein the one or more data augmentation techniques include randomly adding in-domain context around entity values ​​within the first set of training data. The method of claim 1 , wherein the entity type comprises a category to which the named entity belongs.

6. The method of claim 5, wherein an entity value of the at least one or more entity values ​​comprises one or more terms representing individual named entities within the entity type.

7. The method of claim 1, wherein the selection of applying the one or more data augmentation techniques is a hyperparameter.

8. A non-transitory computer-readable medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform operations comprising: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using a trained named entity recognition (NER) model; applying one or more data augmentation techniques to the first set of training data to generate a second set of training data; applying an additional augmentation technique to augment the second set of training data to create post-processed augmented training data, wherein the additional enhancement technique includes replacing at least one or more entity values ​​of the named entities in the second set of training data with random values ​​of the same entity type; combining the first set of training data and the post-processed enhanced training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

9. The non-transitory computer readable medium of claim 8, wherein the one or more data enhancement techniques include: A portion of the first set of training data is randomly replaced with a synonym, or one or more named entities in the first set of training data are randomly deleted.

10. The non-transitory computer readable medium of claim 8, wherein the one or more data enhancement techniques include: Randomly insert synonyms in random positions in the first set of training data, or randomly swap positions of two named entities within the first set of training data.

11. The non-transitory computer-readable medium of claim 8, wherein the one or more data augmentation techniques include randomly adding in-domain context around entity values ​​within the first set of training data.

12. The non-transitory computer-readable medium of claim 8, wherein the entity type comprises a category to which the named entity belongs.

13. The non-transitory computer-readable medium of claim 12, wherein an entity value of the at least one or more entity values ​​comprises one or more terms representing individual named entities within the entity type.

14. The non-transitory computer-readable medium of claim 8, wherein the selection to apply the one or more data augmentation techniques is a hyperparameter.

15. A system comprising: one or more processors; as well as one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using a trained named entity recognition (NER) model; applying one or more data augmentation techniques to the first set of training data to generate a second set of training data; applying an additional augmentation technique to augment the second set of training data to create post-processed augmented training data, wherein the additional enhancement technique includes replacing at least one or more entity values ​​of the named entities in the second set of training data with random values ​​of the same entity type; combining the first set of training data and the post-processed enhanced training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

16. The system of claim 15, wherein the one or more data enhancement techniques include: Randomly replace a portion of the first set of training data with a synonym, randomly delete one or more named entities in the first set of training data, randomly insert synonyms in random positions in the first set of training data, or randomly swap positions of two named entities in the first set of training data.

17. The system of claim 15, wherein the one or more data augmentation techniques include randomly adding in-domain context around entity values ​​within the first set of training data.

18. The system of claim 15, wherein the entity type comprises a category to which the named entity belongs.

19. The system of claim 18, wherein an entity value of the at least one or more entity values ​​comprises one or more terms representing individual named entities within the entity type.

20. The system of claim 15, wherein the selection to apply the one or more data augmentation techniques is a hyperparameter.

21. A method comprising: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using the trained NER model; Applying one or more forward data augmentation techniques to the first set of training data to generate a second set of training data, depending on the hyperparameter adjustment requirements; The forward data enhancement techniques include entity awareness ("EA") techniques and two-stage enhancement techniques; combining the first set of training data and the second set of training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

22. The method of claim 21, wherein the EA technique comprises replacing at least one or more entity values ​​of named entities within the first set of training data with random values ​​sampled from a repository in the domain.

23. The method of claim 22, further comprising being configured to perform an enhancement rate of a plurality of operations of the EA technique.

24. The method of claim 21, wherein the two-stage enhancement technique comprises a first stage enhancement and a second stage enhancement, and the first stage enhancement is performed simultaneously with the EA technique.

25. The method of claim 24, wherein the first stage enhancement of the two-stage enhancement technique comprises: Randomly replace a portion of the first set of training data with synonyms; or Randomly delete one or more named entities in the first set of training data.

26. The method of claim 24, wherein the first stage enhancement of the two-stage enhancement technique comprises: randomly inserting synonyms at random positions in the first set of training data; or Randomly swap the positions of two named entities in the first set of training data.

27. The method of claim 24, wherein the first stage augmentation of the two-stage augmentation technique comprises randomly adding in-domain context around entity values ​​within the first set of training data.

28. The method of claim 24, wherein the second stage enhancement of the two-stage enhancement technique comprises replacing at least one or more entity values ​​of named entities within the result of the first stage enhancement with random values ​​of the same entity type.

29. A non-transitory computer readable medium storing computer executable instructions, which, when executed by one or more processors, cause the one or more processors to perform operations comprising: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using the trained NER model; Applying one or more forward data augmentation techniques to the first set of training data to generate a second set of training data, depending on the hyperparameter adjustment requirements; The forward data enhancement techniques include entity awareness ("EA") techniques and two-stage enhancement techniques; combining the first set of training data and the second set of training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

30. The non-transitory computer-readable medium of claim 29, wherein the EA technique comprises replacing at least one or more entity values ​​of named entities within the first set of training data with random values ​​sampled from an in-domain repository.

31. The non-transitory computer readable medium of claim 30, further comprising an enhancement rate configured to perform a plurality of operations of the EA technique.

32. The non-transitory computer-readable medium of claim 29, wherein the two-stage enhancement technique comprises a first stage enhancement and a second stage enhancement, and the first stage enhancement is performed simultaneously with the EA technique.

33. The non-transitory computer readable medium of claim 32, wherein the first stage enhancement of the two-stage enhancement technique comprises: Randomly replace a portion of the first set of training data with synonyms; Randomly deleting one or more named entities in the first set of training data; randomly inserting synonyms at random positions in the first set of training data; Randomly swapping the positions of two named entities in the first set of training data; or Randomly add in-domain context around entity values ​​in the first set of training data.

34. The non-transitory computer-readable medium of claim 32, wherein the second stage enhancement of the two-stage enhancement technique comprises replacing at least one or more entity values ​​of named entities within the result of the first stage enhancement with random values ​​of the same entity type.

35. A system comprising: one or more processors; as well as one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using the trained NER model; Applying one or more forward data augmentation techniques to the first set of training data to generate a second set of training data, depending on the hyperparameter adjustment requirements; The forward data enhancement techniques include entity awareness ("EA") techniques and two-stage enhancement techniques; combining the first set of training data and the second set of training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

36. The system of claim 35, wherein the EA technique comprises replacing at least one or more entity values ​​of named entities within the first set of training data with random values ​​sampled from a repository in the domain.

37. The system of claim 36, further comprising an enhancement rate configured to perform a plurality of operations of the EA technique.

38. The system of claim 35, wherein the two-stage enhancement technique comprises a first stage enhancement and a second stage enhancement, and the first stage enhancement is performed simultaneously with the EA technique.

39. The system of claim 38, wherein the first stage enhancement of the two-stage enhancement technique comprises: Randomly replace a portion of the first set of training data with synonyms; Randomly deleting one or more named entities in the first set of training data; randomly inserting synonyms at random positions in the first set of training data; Randomly swapping the positions of two named entities in the first set of training data; or Randomly add in-domain context around entity values ​​in the first set of training data.

40. The system of claim 38, wherein the second stage enhancement of the two-stage enhancement technique comprises replacing at least one or more entity values ​​of named entities within the result of the first stage enhancement with random values ​​of the same entity type.

41. A method comprising: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using the trained NER model; Applying one or more negative entity-aware data augmentation techniques to the first set of training data to generate a second set of training data, depending on the hyperparameter adjustment requirements; wherein the one or more negative entity-aware data augmentation techniques include keyword augmentation technique ("KAT") plus context-free entity technique and KAT plus random in-context entities as OOD technique; combining the first set of training data and the second set of training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

42. The method of claim 41, further comprising: A stop word augmentation technique ("SWAT") is applied to generate the second set of training data by replacing non-stop words in the first set of training data with random stop words while retaining existing stop words.

43. The method of claim 41, wherein the KAT plus context-free entity technology comprises: In-domain contextual information around the named entities in the first set of training data is removed, thereby generating modified data as out-of-domain training data.

44. The method of claim 43, wherein the vector distance between the out-of-domain training data and any in-domain training data is above a threshold.

45. The method of claim 41, wherein the KAT plus random contextual entities as OOD technology comprises: The context surrounding the entity value in the first set of training data is replaced with random out-of-domain context.

46. ​​The method of claim 41, wherein the entity-aware data augmentation technique trains the intent prediction model to focus on the overall context rather than changes to entity values ​​of an utterance.

47. The method of claim 46, wherein the entity value comprises one or more terms representing individual named entities within a named entity category.

48. A non-transitory computer readable medium storing computer executable instructions, which, when executed by one or more processors, cause the one or more processors to perform operations comprising: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using the trained NER model; Applying one or more negative entity-aware data augmentation techniques to the first set of training data to generate a second set of training data, depending on the hyperparameter adjustment requirements; wherein the one or more negative entity-aware data augmentation techniques include keyword augmentation technique ("KAT") plus context-free entity technique and KAT plus random in-context entities as OOD technique; combining the first set of training data and the second set of training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

49. The non-transitory computer readable medium of claim 48, further comprising: A stop word augmentation technique ("SWAT") is applied to generate the second set of training data by replacing non-stop words in the first set of training data with random stop words while retaining existing stop words.

50. The non-transitory computer readable medium of claim 48, wherein the KAT plus context-free entity technology comprises: In-domain contextual information around the named entities in the first set of training data is removed, thereby generating modified data as out-of-domain training data.

51. The non-transitory computer-readable medium of claim 50, wherein a vector distance between the out-of-domain training data and any in-domain training data is above a threshold.

52. The non-transitory computer readable medium of claim 48, wherein the KAT plus random contextual entities as an OOD technique comprises: The context surrounding the entity value in the first set of training data is replaced with random out-of-domain context.

53. The non-transitory computer-readable medium of claim 48, wherein the entity-aware data augmentation technique trains the intent prediction model to focus on overall context rather than changes to entity values ​​of an utterance.

54. The non-transitory computer-readable medium of claim 53, wherein the entity value comprises one or more terms representing individual named entities within a named entity category.

55. A system comprising: one or more processors; as well as one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to: Accessing a first set of training data for an intent prediction model, the first set of training data comprising utterances and intent labels; extracting named entities from the first set of training data using the trained NER model; Applying one or more negative entity-aware data augmentation techniques to the first set of training data to generate a second set of training data, depending on the hyperparameter adjustment requirements; wherein the one or more negative entity-aware data augmentation techniques include keyword augmentation technique ("KAT") plus context-free entity technique and KAT plus random in-context entities as OOD technique; combining the first set of training data and the second set of training data to generate extended training data; Using the extended training data to train the intent prediction model; as well as Deploy the trained intent prediction model.

56. The system of claim 55, further comprising: A stop word augmentation technique ("SWAT") is applied to generate the second set of training data by replacing non-stop words in the first set of training data with random stop words while retaining existing stop words.

57. The system of claim 55, wherein the KAT plus context-free entity technology comprises: In-domain contextual information around the named entities in the first set of training data is removed, thereby generating modified data as out-of-domain training data.

58. The system of claim 57, wherein a vector distance between the out-of-domain training data and any in-domain training data is above a threshold.

59. The system of claim 55, wherein the KAT plus random contextual entities as OOD technology comprises: The context surrounding the entity value in the first set of training data is replaced with random out-of-domain context.

60. The system of claim 55, wherein the entity-aware data augmentation technique trains the intent prediction model to focus on overall context rather than changes to entity values ​​of an utterance.

Images