Adaptive training data augmentation to facilitate training of named entity recognition models

Abstract

Techniques for adaptive training data augmentation to facilitate training of a Named Entity Recognition (NER) model are disclosed herein. Adaptive augmentation techniques that take into account the distribution of different entity types within training data are disclosed herein. An adaptive augmentation technique generates an adaptive number of augmentation examples (e.g., utterances) based on the distribution of entities to ensure that a sufficient number of minority classified entity examples are generated during augmentation of training data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Non-Provisional Patent Application No. 18 / 450,678, filed on August 16, 2023, which claims the benefit of and priority to U.S. Provisional Application No. 63 / 373,148, filed on August 22, 2022, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] The present disclosure relates generally to machine learning techniques for augmenting training data, and more particularly to machine learning techniques for adaptive training data augmentation to facilitate the training of named entity recognition models. Background Art

[0004] Artificial intelligence has many applications. To illustrate, many users around the world use instant messaging or chat platforms to get instant responses. Organizations often use these instant messaging or chat platforms to conduct real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots or robots have been developed to simulate conversations with end users, especially over the Internet. End users can communicate with such robots through messaging apps. Intelligent robots (usually robots driven by artificial intelligence (AI)) can communicate intelligently and contextually in real-time conversations with end users, which allows the conversation to be more natural and improves the conversational experience. Instead of relying on a fixed set of keywords or commands, intelligent robots can receive utterances expressed by end users in natural language, understand their intent, and respond accordingly.

[0005] However, artificial intelligence-based solutions (such as chatbots) can be difficult to build because these automated solutions require specific knowledge in certain fields and the application of certain technologies that may only be within the capabilities of professional developers. As part of building such a chatbot, developers can first understand the needs of the business and end users. The developer can then analyze and make decisions related to, for example, the following: selecting a data set to be used for analysis; preparing an input data set for analysis (e.g., cleaning data, extracting, formatting and / or transforming data before analysis, performing data feature engineering, etc.); identifying appropriate (multiple) machine learning (ML) techniques or (multiple) ML models for performing analysis; and improving techniques or models to improve results / effects based on feedback. The task of identifying an appropriate model can include developing multiple models (possibly in parallel), iteratively testing and experimenting with these models, and then identifying a specific model (or models) for use. Further, solutions based on supervised learning typically involve a training phase, followed by an application (i.e., reasoning) phase, and an iterative cycle between the training phase and the application phase. Developers can be responsible for carefully implementing and monitoring these phases to achieve the best solution. For example, to train ML techniques or models, accurate training data is needed to enable the algorithm to understand and learn certain patterns or features (e.g., for chatbots - intent extraction and careful syntactic analysis are needed, not just raw language processing), which the ML techniques or models will use to predict desired outcomes (e.g., infer intent from utterances). To ensure that the ML techniques or models learn these patterns and features correctly, developers may be responsible for selecting, enriching, and optimizing training datasets for the ML techniques or models. Summary of the invention

[0006] Disclosed herein are machine learning techniques for adaptive training data augmentation to facilitate the training of named entity recognition (NER) models.

[0007] In various embodiments, a computer-implemented method includes: accessing training data including a plurality of utterances, wherein each utterance of the plurality of utterances includes at least one named entity corresponding to a named entity category of a plurality of named entity categories; calculating an expansion rate for each of the plurality of named entity categories based on a distribution of each of the plurality of named entity categories in the training data and a function selected from a plurality of functions in a hyperparameter tuning process of a NER model; generating a plurality of additional utterances based on the expansion rate calculated for each of the plurality of named entity categories; augmenting the training data by combining the plurality of additional utterances with the plurality of utterances to form augmented training data; and training the NER model with the augmented training data.

[0008] In some embodiments, the hyperparameter tuning process includes: dividing the training data into training data sets, each of the training data sets including a corresponding set of utterances from the multiple utterances; for each of the training data sets: selecting a candidate hyperparameter from a candidate hyperparameter set, generating an expanded utterance set for the corresponding training data set from the corresponding utterance set based on the candidate hyperparameter and based on a function associated with the candidate hyperparameter from the multiple functions, training the NER model with the expanded utterance set, and calculating a performance indicator for the NER model trained with the expanded utterance set; and determining a hyperparameter set for the NER model based on the performance indicator calculated for the NER model trained with the corresponding expanded utterance sets.

[0009] In some embodiments, a number of additional utterances generated for a first named entity category in the plurality of named entity categories is different than a number of additional utterances generated for a second named entity category in the plurality of named entity categories.

[0010] In some embodiments, determining the expansion rate of each of the multiple named entity categories includes calculating the number of occurrences of each of the multiple named entity categories in the training data and normalizing the number of occurrences of each of the multiple named entity categories in the training data.

[0011] In some embodiments, determining the expansion rate of each named entity category in the multiple named entity categories includes calculating a normalized distribution value for each named entity category in the multiple named entity categories, wherein, for each named entity category in the multiple named entity categories, the normalized distribution value is calculated by dividing the number of occurrences calculated for the corresponding named entity category by the sum of the number of occurrences of all named entity categories in the multiple named entity categories in the training data.

[0012] In some embodiments, the multiple functions include at least one of a first function and a second function, wherein the first function calculates the expansion rate of a given named entity category based on an exponential function that receives a normalized distribution value and a β value of the given named entity category as input, and the second function calculates the expansion rate of the given named entity category based on a logarithmic function with base 2 that receives the number of occurrences of the given named entity category in the training data as input.

[0013] In some embodiments, the method further includes providing the trained NER model to the system, wherein providing the trained NER model includes detecting and classifying named entities in utterances received by the system from the user.

[0014] Some embodiments include a system comprising one or more processors and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0015] Some embodiments include one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a system to perform some or all of the operations and / or methods disclosed herein.

[0016] The technology described above and below can be implemented in a variety of ways and in a variety of contexts. As described in more detail below, several example implementations and contexts are provided with reference to the following figures. However, the following implementations and contexts are only some of many implementations and contexts. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a simplified block diagram of a distributed environment incorporating an exemplary embodiment.

[0018] Figure 2 is a simplified block diagram of a computing system implementing a master robot according to some embodiments.

[0019] Figure 3 is a simplified block diagram of a computing system that implements a skill robot in accordance with certain embodiments.

[0020] Figure 4A is a simplified block diagram of a model training and deployment system according to certain embodiments.

[0021] Figure 4B is a simplified block diagram of a system incorporating a named entity recognition model according to certain embodiments.

[0022] Figure 5A is a flow chart illustrating an example process for adaptively augmenting training data in accordance with certain embodiments.

[0023] Figure 5B is a flow diagram illustrating an example process for classifying the intent of an utterance in accordance with certain embodiments.

[0024] Figure 6 Depicted is a simplified diagram of a distributed system for implementing various embodiments.

[0025] Figure 7 is a simplified block diagram of one or more components of a system environment according to various embodiments by which services provided by one or more components of an embodiment system may be provided as a cloud service.

[0026] Figure 8 An example computer system is illustrated that can be used to implement the various embodiments. DETAILED DESCRIPTION

[0027] In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments or designs.

[0028] introduction

[0029] Artificial intelligence has broad applicability. For example, a digital assistant is an AI-driven interface that helps users complete various tasks based on natural language conversations. For each digital assistant, customers can assemble one or more skills. Skills (also described herein as chatbots, robots, or skillbots) are individual robots that focus on specific types of tasks such as tracking inventory, submitting time cards, and creating expense reports. When an end user engages with a digital assistant, the digital assistant evaluates the end user input and routes the conversation to and from the appropriate chatbot. This can be accomplished through, for example, Messenger (Instant Messenger), SKYPE Various channels such as messenger or short message service (SMS) make the digital assistant available to the end user. Channels enable chat to be passed back and forth between the end user and the digital assistant and its bots through various messaging platforms. Channels can also support user agent escalation, event-initiated conversations, and testing.

[0030] Intent allows artificial intelligence-based technologies such as chatbots to understand what users want the chatbot to do. Intent refers to the intentions that users convey to the chatbot via user requests and statements, which are also referred to as discourse (e.g., get an account balance, make a purchase, etc.). As used herein, discourse or messages can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. Intent can be created by providing a name that describes a certain user action (e.g., ordering pizza) and compiling a set of real-life user statements or discourses that are usually associated with the triggering action. Because the cognition of the chatbot is derived from these intentions, each intention can be created and varied from a robust data set (one to twenty or so discourses) so that the chatbot can interpret ambiguous user input. A rich set of discourses enables the chatbot to understand what the user wants when it receives messages such as "Forget this order! (Ignore this order!)" or "Cancel delivery! (Cancel delivery!)" (messages that mean the same thing but are expressed in different ways). Intent classifiers are included in the chatbot system to automatically classify the intent of discourse.

[0031] Discourses can include named entities, which further assist the chatbot in understanding the meaning of discourse. For example, if a user types "show me yesterday's financial news", the named entities "yesterday" and "financial" will help the chatbot understand the user's request (i.e., check the news). Entities can be classified according to what they represent. For example, "yesterday" can be classified as a "dateTime" entity, and "financial" can be classified as a "newsType" entity. Named entity recognition (NER) is a tool used by intent classifiers and chatbot systems to automatically detect, extract, and classify entities. In short, discourse (including named entities and the intents belonging to them) can be used as a training corpus for a chatbot. By training the algorithm with a corpus, the customer turns the algorithm into a model used as a reference tool for parsing (multiple) end-user inputs into a single intent. Customers can improve the cognitive acuity of chatbots through multiple rounds of intent testing and intent training.

[0032] The use of artificial intelligence in the context of chatbots illustrates some of the challenges of applying artificial intelligence techniques. For example, building a chatbot that can determine the intent of an end user based on their utterances is a challenging task, at least due to the subtlety and ambiguity of natural language and the dimensionality of the input / output space (e.g., the number of possible user utterances, intents, etc.). Illustrative examples of this difficulty come from the characteristics of natural language, such as the use of euphemisms, synonyms, or ungrammatical speech to express intent. For example, an utterance may express the intent to order a pizza without mentioning the words pizza, ordering, or delivery. These characteristics of natural language create uncertainty and lead chatbots to use confidence as a parameter for predicting user intent. As such, in order to improve the performance of the chatbot and the user experience with the chatbot, it may be necessary to train, monitor, debug, and retrain the chatbot. In traditional spoken language understanding (SLU) and natural language processing (NLP) systems, training mechanisms are provided for training and retraining the machine learning algorithms of the digital assistant or chatbot contained therein. Traditionally, these algorithms are trained using utterances that are "made" for any intent. For example, the utterance "Do you do price changes?" can be used to train the classification algorithm of the chatbot system to classify this type of utterance as the intent "Do you offer a price match?" Training the algorithm with manufactured utterances helps to initially train the chatbot system to provide a service and retrain the chatbot system when the chatbot system is deployed and receives utterances from users.

[0033] Entity extraction usually has two stages: the named entity recognition stage and the entity resolution stage. The specific problem addressed here involves the named entity recognition stage. Entities are usually things like dates, times, locations, names, brands, etc. Entities can also be system entities because they are generic and domain-independent (e.g., PERSON, NUMBER, CURRENCY, DATE_TIME). In the named entity recognition stage, a NER model is usually trained to predict the classification labels of tokens in an utterance. An utterance is a natural language text, and the classification labels represent named entities. Traditional NER models are trained using publicly available datasets and privately available datasets. However, these datasets are often not diverse enough to train NER models to detect entities with all kinds of context and entity variations. In addition, these datasets are usually unbalanced in their representation of different entity classifications, so some entity classifications are overrepresented in the dataset, while some entity classifications are underrepresented. For example, a dataset may include a total of 100 examples, in which the entity class CURRENCY appears in 26 of the examples, the entity class TIME_DATE appears in 40 of the examples, and the entity class PERSON appears in 73 of the examples. Thus, the entity class PERSON is the majority class in the dataset (i.e., over-represented), while the entity classes CURRENCY and TIME_DATE are the minority classes in the dataset (i.e., under-represented). NER models are typically biased toward learning majority class entities, and thus often misclassify minority class entities.

[0034] One option to diversify and balance the representation of entity classifications in these data sets is to generate additional labeled utterances and add them to the data set. The additional labeled utterances can be written by end users, written by freelancers and / or companies specializing in data labeling, and / or crowdsourced using crowdsourcing workers. However, these methods may be difficult to implement for enterprise systems that employ many artificial intelligence-based systems (such as multiple chatbot systems), which are trained for many different tasks in multiple languages ​​and receive a variety of utterances for each task. In systems that employ artificial intelligence-based systems such as this, it is necessary to automatically obtain the diversity of entities in a synthetically agnostic manner so as to quickly and efficiently generate a large number of training data corpora in multiple languages ​​for many different chatbots.

[0035] Another option to diversify these datasets is to augment the utterances in these datasets and combine the augmented utterances to the datasets. Traditional utterance augmentation techniques for NER typically augment the utterances in the dataset by replacing entities in the utterance with different entities (e.g., other entities obtained from the dataset itself or from a dictionary (such as a gazetteer)). For example, for the utterance "What is the cost of parking at IAD airport" in the example dataset, in a traditional utterance augmentation scheme, the entity "IAD airport" can be replaced by extracting entities from the same entity classification from the gazetteer (e.g., "SFO airport") and replacing the entity "IAD airport" with the entity "SFO airport" to produce an augmented utterance "What is the cost of parking at SFO airport". The augmented utterances can be combined with the dataset to expand the dataset. However, adding a static number of augmented utterances under this approach may further tilt the dataset toward the majority-classified entities represented in the dataset, thereby adversely affecting the performance of the NER model on the minority-classified entities represented in the dataset.

[0036] Therefore, a different approach is needed to address these and other challenges. The developed method utilizes adaptive augmentation techniques that take into account the distribution of different entity types within the training data and ultimately improves the performance of NER models trained on the adaptively augmented training data. The adaptive augmentation techniques generate an adaptive number of augmented examples (e.g., utterances) based on the distribution of entities to ensure that a sufficient number of minority classified entity examples are generated during the augmentation of the training data. This is different from traditional static augmentation methods, which tend to generate augmented utterances with a fixed number of utterances without considering the overall representation of entity types within the training data. For example, if there are different numbers of utterances including entity type A compared to entity type B, the adaptive augmentation technique calculates the adaptive rate of generating augmented examples with entity type A and entity type B separately. The result is that each entity distribution across the training data is normalized so that during training, the NER model does not become biased towards learning majority classified entities.

[0037] In various embodiments, a computer-implemented method includes: accessing training data including a plurality of utterances, wherein each utterance of the plurality of utterances includes at least one named entity corresponding to a named entity category of a plurality of named entity categories; calculating an expansion rate for each of the plurality of named entity categories based on a distribution of each of the plurality of named entity categories in the training data and a function selected from a plurality of functions in a hyperparameter tuning process of a NER model; generating a plurality of additional utterances based on the expansion rate calculated for each of the plurality of named entity categories; augmenting the training data by combining the plurality of additional utterances with the plurality of utterances to form augmented training data; and training the NER model with the augmented training data.

[0038] As used herein, when an action is "based on" something, it means that the action is based, at least in part, on at least a portion of that something. As used herein, the terms "similarly," "substantially," "approximately," and "about" are defined as being largely, but not necessarily entirely, what is specified (and entirely including what is specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "similarly," "substantially," "approximately," or "about" may be replaced with "within [percentage] of what is specified," where the percentage includes 0.1%, 1%, 5%, and 10%.

[0039] Robotic system

[0040] A robot (also known as a skill, chatbot, conversational robot, or conversational robot) is a computer program that can perform a conversation with an end user. A robot can typically respond to natural language messages (e.g., questions or comments) through a messaging application that uses natural language messages. An enterprise can use one or more robots to communicate with end users through a messaging application. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web application extensions / responsive mobile applications or web applications with chat capabilities, or voice-based input (such as devices or applications with Siri, Cortana, Google Voice, or other voice input for interaction).

[0041] In some examples, a robot can be associated with a uniform resource identifier (URI). A URI can use a string of characters to identify a robot. A URI can be used as a webhook for one or more messaging application systems. A URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). A robot can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. An HTTP post call message can involve a URI from a messaging application system. In some examples, a message can be different from an HTTP post call message. For example, a robot can receive a message from a short message service (SMS). Although the discussion herein refers to communications received by a robot as messages, it should be understood that a message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0042] End users interact with robots through conversational interactions (sometimes called conversational user interfaces (UIs)) just as they interact with other people. In some cases, a conversational interaction can include an end user saying "Hello" to a robot and the robot responding with "Hi" and asking the end user how the robot can help. End users also interact with robots through other types of interactions, such as transactional interactions (e.g., with a banking robot that is at least trained to transfer funds from one account to another), informational interactions (e.g., with a human resources robot that is at least trained to check a user's remaining vacation time), and / or retail interactions (e.g., with a retail robot that is at least trained to discuss returning a purchase or seeking technical support).

[0043] In some examples, the robot can intelligently handle end-user interactions without intervention by the robot's administrator or developer. For example, the end user can send one or more messages to the robot in order to achieve a desired goal. The message can include some content, such as text, emoticons, audio, images, videos, or other methods of conveying messages. In some examples, the robot can automatically convert the content into a standardized form and generate a natural language response. The robot can also automatically prompt the end user to enter additional input parameters or request other additional information. In some examples, the robot can also initiate communication with the end user instead of passively responding to the end user's speech.

[0044] A conversation with a robot can follow a specific conversation flow including multiple states. The flow can define what happens next based on input. In some examples, a state machine including user-defined states (e.g., end-user intent) and actions to be taken in or between states can be used to implement the robot. The conversation can take different paths based on the end-user input, which may affect the decision made by the robot for the flow. For example, in each state, based on the end-user input or speech, the robot can determine the end-user's intent in order to determine the next appropriate action to be taken. As used herein and in the context of speech, the term "intent" refers to the intention of the user who provides the speech. For example, a user may intend to let the robot participate in a conversation to order pizza, where the user's intention will be expressed by the speech "order pizza". User intent can involve a specific task that the user wants the robot to perform on behalf of the user. Therefore, the speech reflecting the user's intention can be expressed as a question, command, request, etc.

[0045] In the context of the configuration of the robot, the term "intent" used herein also refers to configuration information for mapping the user's speech to a specific task / action or category of tasks / actions that the robot can perform. In order to distinguish the intent of the speech (i.e., the user's intent) from the intent of the robot, the latter is sometimes referred to as "robot intent" herein. The robot intent may include a set of one or more utterances associated with the intent. For example, the intent of ordering pizza may have various permutations of utterances expressing the expectation of placing an order to buy pizza. These associated utterances may be used to train the robot's intent classifier so that the intent classifier can then determine whether the input utterance from the user matches the intent of ordering pizza. The robot intent may be associated with one or more dialog flows for initiating a conversation with the user in a certain state. For example, the first message for the intent of ordering pizza may be the question "What kind of pizza would you like?" In addition to the associated utterances, the robot intent may further include named entities related to the intent. For example, the intent of ordering pizza may include variables or parameters (e.g., filling 1, filling 2, pizza type, pizza size, pizza quantity, etc.) for performing the task of ordering pizza. The value of the entity is usually obtained by talking to the user.

[0046] Figure 1 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to some embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102 that enables users 104 of the DABP 102 to create and deploy digital assistants or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, Figure 1 As shown, a user 104 representing a particular business can use DABP 102 to create and deploy a digital assistant 106 for the users of the particular business. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0047] For purposes of this disclosure, a "digital assistant" is a tool that assists the user of the digital assistant in completing various tasks through natural language conversation. A digital assistant may be implemented using software alone (e.g., a digital assistant is a digital tool implemented using a program, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in a variety of physical systems or devices such as computers, mobile phones, watches, appliances, vehicles, and the like. A digital assistant is also sometimes referred to as a chatbot system. Therefore, for purposes of this disclosure, the terms "digital assistant" and "chatbot system" are interchangeable.

[0048] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and obtain responses 112 in return from the digital assistant 106. The conversation can include one or more of the inputs 110 and the responses 112. Via these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond to the user with appropriate responses.

[0049] User input 110 is usually in the form of natural language and is referred to as utterance. User utterance 110 can be in text form, such as when a user types a sentence, question, text snippet, or even a single word and provides the text as input to the digital assistant 106. In some examples, user utterance 110 can be in the form of audio input or speech, such as when a user speaks or says something as input to the digital assistant 106. The utterance is usually in the form of a language spoken by the user. For example, the utterance can be English or some other language. When the utterance is in the form of speech, the speech input is converted into a textual utterance in the particular language, and the text utterance is then processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert the speech or audio input into a textual utterance, which is then processed by the digital assistant 106. In some examples, the conversion of speech to text can be done by the digital assistant 106 itself.

[0050] The utterance (which may be a text utterance or a voice utterance) may be a fragment, a sentence, a plurality of sentences, one or more words, one or more questions, a combination of the above types, and the like. The digital assistant 106 is configured to apply natural language understanding (NLU) technology to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing for understanding the meaning of the utterance, which involves identifying one or more intentions and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or operations in response to the understood meaning or intention. For the purposes of this disclosure, it is assumed that these utterances are text utterances that have been directly provided by the user of the digital assistant 106, or are the result of converting the input voice utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0051] For example, user input can request to order pizza by providing an utterance such as "I want to order a pizza". After receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate actions. Appropriate actions may involve, for example, responding to the user with questions requesting user input regarding the type of pizza that the user desires to order, the size of the pizza, any toppings of the pizza, etc. The responses provided by the digital assistant 106 may also be in natural language form, and are typically in the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). In order for the user to order pizza via a conversation between the user and the digital assistant 106, the digital assistant may guide the user to provide all necessary information for pizza ordering, and then order the pizza at the end of the conversation. The digital assistant 106 may end the conversation by outputting information indicating that the pizza has been ordered to the user.

[0052] At a conceptual level, the digital assistant 106 performs various processing in response to utterances received from the user. In some examples, the processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance, determining an action to be performed in response to the utterance, causing the action to be performed where appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing can include grammatical analysis of the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better understandable form (e.g., logical form) or structure of the utterance. Generating a response can include using NLG technology.

[0053] The NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related tasks such as sentence grammatical analysis (e.g., word segmentation, classification by inflectional form, identification of part-of-speech tags of sentences, identification of named entities in sentences, generation of dependency trees to represent sentence structure, splitting sentences into clauses, analyzing separate clauses, resolving references, executing chunks, etc.). In some examples, the NLU processing is performed by the digital assistant 106 itself. In some other examples, the digital assistant 106 may use other resources to perform part of the NLU processing. For example, sentences may be processed using a grammatical analyzer, a part-of-speech tagger, and / or NER to identify the syntax and structure of an input utterance sentence. In one embodiment, for the English language, a grammatical analyzer, a part-of-speech tagger, and a named entity recognizer provided by the Stanford NLP group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0054] Although the various examples provided in this disclosure show utterances in the English language, this is meant only as an example. In some examples, the digital assistant 106 is also capable of handling utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that may be called from an NLU core server using service calls. This makes NLU processing flexible and extensible for each language, including allowing different processing orders. Language packs may be provided for separate languages, wherein the language pack may register a list of subsystems that may provide services from an NLU core server.

[0055] The digital assistant (e.g., a user) may be enabled through a variety of different channels, such as, but not limited to, via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. Figure 1 106) is available or accessible to its user 108. A single digital assistant can configure several channels for itself, so that a single digital assistant can run on and be accessed through different services at the same time.

[0056] A digital assistant or chatbot system typically contains or is associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill bots) that are configured to interact with a user and complete specific types of tasks (such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc.). For example, Figure 1In the depicted embodiment, the digital assistant or chatbot system 106 includes skills 116 - 1 , 116 - 2 , 116 - 3 , etc. For purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skill bot” and “skill bots,” respectively.

[0057] Each skill associated with the digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, where the conversation can include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses can be in the form of text or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for selection.

[0058] There are various ways to associate skills or skill robots with digital assistants or add them to digital assistants. In some instances, skill robots can be developed by enterprises and then added to digital assistants using DABP 102. In other instances, skill robots can be developed and created using DABP 102 and then added to digital assistants created using DABP 102. In yet other instances, DABP 102 provides an online digital store (referred to as a "skill store") that provides multiple skills related to a variety of tasks. The skills provided by the skill store can also expose various cloud services. In order to add skills to a digital assistant generated using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skills, and indicate that the selected skills will be added to the digital assistant created using DABP 102. Skills from the skill store can be added to the digital assistant as is or in modified form (e.g., a user of DABP 102 can select and copy a particular skill bot provided by the skill store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0059] A digital assistant or chatbot system may be implemented using a variety of different architectures. For example, in some embodiments, a digital assistant created and deployed using DABP 102 may be implemented using a master robot / secondary (or child) robot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a master robot interacting with one or more secondary robots that are skill robots. For example, in Figure 1In the depicted embodiment, the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc. that are secondary robots to the master robot 114. In some examples, the digital assistant 106 itself is considered to act as the master robot.

[0060] A digital assistant implemented according to the master-sub robot architecture enables users of the digital assistant to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user interacts with the digital assistant, the master robot receives user input. The master robot then performs processing to determine the meaning of the user's input speech. The master robot then determines whether the task requested by the user in the speech can be handled by the master robot itself, otherwise the master robot selects an appropriate skill robot to handle the user request and routes the conversation to the selected skill robot. This enables users to have a conversation with the digital assistant through a public single interface and still have the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot of the digital assistant can be connected to a skill robot interface with specific functions, such as a customer relationship management (CRM) robot for performing functions related to customer relationship management, an enterprise resource planning (ERP) robot for performing functions related to enterprise resource planning, a human capital management (HCM) robot for performing functions related to human capital management, etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a public master robot interface, and multiple skill robots are provided in the background for handling user requests.

[0061] In some examples, in the master robot / secondary robot infrastructure, the master robot is configured to understand the list of available skill robots. The master robot can access metadata identifying various available skill robots, and for each skill robot, access the capabilities of the skill robot including tasks that can be performed by the skill robot. After receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot that can best serve or handle the user request from multiple available skill robots. The master robot then routes the utterance (or a portion of the utterance) to the specific skill robot for further handling. Thus, control flows from the master robot to the skill robot. The master robot can support multiple input channels and output channels. In some examples, routing can be performed with the aid of processing performed by one or more available skill robots. For example, as discussed below, a skill robot can be trained to infer the intent of an utterance and determine whether the inferred intent matches the intent configured for the skill robot. Therefore, routing performed by the master robot can involve the skill robot transmitting an indication to the master robot indicating whether the skill robot has been configured with an intent suitable for handling the utterance.

[0062] Although Figure 1 The embodiment of the digital assistant 106 is shown to include a master robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an embodiment using a combination of software and hardware.

[0063] DABP 102 provides an infrastructure and various services and features that enable users of DABP 102 to create digital assistants (including one or more skill robots associated with the digital assistant). In some instances, skill robots can be created by cloning existing skill robots, for example, cloning skill robots provided by a skill store. As previously described, DABP 102 provides a skill store or skill catalog that provides multiple skill robots for performing various tasks. Users of DABP 102 can clone skill robots from the skill store. The cloned skill robots can be modified or customized as needed. In some other instances, users of DABP 102 create skill robots from scratch using the tools and services provided by DABP 102. As previously described, the skill store or skill catalog provided by DABP 102 can provide multiple skill robots for performing various tasks.

[0064] In some examples, at a high level, creating or customizing a skill bot involves the following steps:

[0065] (1) Configuring settings for new skill robots

[0066] (2) Configure one or more intents for the skill robot

[0067] (3) Configure one or more entities for one or more intents

[0068] (4) Training Skill Robots

[0069] (5) Create a dialogue flow for the skill bot

[0070] (6) Add custom parts to skill robots as needed

[0071] (7) Test and deploy skill robots

[0072] Each of these steps is briefly described below.

[0073] (1) Configuring settings for a new skill bot - Various settings can be configured for a skill bot. For example, a skill bot designer can specify one or more invocation names for the skill bot being created. Users of the digital assistant can then use these invocation names to explicitly invoke the skill bot. For example, a user can enter an invocation name in the user's utterance to explicitly invoke the corresponding skill bot.

[0074] (2) Configuring one or more intents and associated example utterances for the skill robot - The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classifications that the skill robot is trained to infer for the input utterance. After receiving the utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a set of predefined intents used to train the skill robot. The skill robot then takes appropriate actions in response to the utterance based on the intent inferred for the utterance. In some instances, the intent of the skill robot represents the tasks that the skill robot can perform for the user of the digital assistant. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot may include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0075] For each intent defined for the skill robot, the skill robot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent the utterances that a user can enter into the skill robot for that intent. For example, for the CheckBalance intent, example utterances can include "What's mysavings account balance?", "How much is in mychecking account?", "How much money do I have inmy account?", etc. Therefore, various arrangements of typical user utterances can be specified as example utterances for an intent.

[0076] These intents and their associated example utterances are used as training data for training the skill robot. A variety of different training techniques can be used. As a result of this training, a prediction model is generated that is configured to take an utterance as input and output the intent inferred by the prediction model for the utterance. In some instances, the input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer the intent of the input utterance. The skill robot can then take one or more actions based on the inferred intent.

[0077] (3) Configure entities for one or more intents of a skill robot - In some instances, additional context may be required to enable a skill robot to respond appropriately to user utterances. For example, there may be situations where user input utterances resolve to the same intent in a skill robot. For example, in the above example, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests asking different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill robot, an entity called AccountType (which defines values ​​called "checking" and "saving") can enable the skill robot to parse the user request and respond appropriately. In the above example, although these utterances resolve to the same intent, the values ​​associated with the AccountType entity for the two utterances are different. This enables the skill robot to perform potentially different actions for the two utterances, even though the two utterances resolve to the same intent. One or more entities can be specified for certain intents configured for a skill robot. Therefore, entities are used to add context to the intent itself. Entities help describe the intent more fully and enable the skill bot to complete the user request.

[0078] In some examples, there are two types of entities: (a) built-in entities provided by DABP 102; and (2) custom entities that can be specified by the skill robot designer. Built-in entities are general entities that can be used with various robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by a skill robot designer to implement various banking transactions by checking user input keywords such as demand deposit, savings, and credit card, etc.

[0079] (4) Training a Skill Robot - A skill robot is configured to receive user input in the form of speech, parse or otherwise process the received input and identify or select intents associated with the received user input. As indicated above, to do this, the skill robot must be trained. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example speech associated with the intents (collectively referred to as training data) so that the skill robot can parse the user input speech into one of the intents configured for it. In some examples, the skill robot uses a predictive model that is trained using the training data and allows the skill robot to discern what the user has said (or in some cases, is attempting to say). DABP 102 provides a variety of different training techniques that can be used by skill robot designers to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some examples, a portion (e.g., 80%) of the training data is used to train the skill robot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as a trained skill robot) can be used to handle and respond to the user's speech. In some cases, the user's utterance may be a question that requires only a single answer and no additional conversation is required. To handle this situation, a Q&A (question answering) intent can be defined for the skill bot. This enables the skill bot to output a reply to the user's request without having to update the dialog definition. A Q&A intent is created in a similar manner to a regular intent. The dialog flow for a Q&A intent may be different from the dialog flow for a regular intent.

[0080] (5) Create a dialog flow for the skill robot - The dialog flow specified for the skill robot describes how the skill robot reacts when parsing different intents of the skill robot in response to received user input. The dialog flow defines the operations or actions that the skill robot will take, for example, how the skill robot responds to user utterances, how the skill robot prompts the user for input, and how the skill robot returns data. The dialog flow is like a flowchart followed by the skill robot. Skill robot designers use languages ​​such as markdown to specify the dialog flow. In some embodiments, a version of YAML called OBotML can be used to specify the dialog flow for the skill robot. The dialog flow definition of the skill robot acts as a model for the conversation itself, which is a model that enables the skill robot designer to choreograph the interaction between the skill robot and the user served by the skill robot.

[0081] In some examples, the dialog flow definition for a skill bot contains the following three parts:

[0082] (a) Context

[0083] (b) Default Transformation Section

[0084] (c) Status section

[0085] Context section - The skill bot designer can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, user variables that enable the skill bot to recognize and save user preferences, etc.

[0086] Default Transitions Section - Transitions for a skill bot can be defined in the Dialog Flow State section or in the Default Transitions section. Transitions defined in the Default Transitions section act as fallbacks and are triggered when no applicable transitions are defined within a state or when the conditions required to trigger a state transition cannot be met. The Default Transitions section can be used to define routes that allow a skill bot to gracefully handle unexpected user actions.

[0087] State Part - A dialog flow and its associated operations are defined as a sequence of transient states that govern the logic within the dialog flow. Each state node within a dialog flow definition names a component that provides the functionality required at that point in the dialog. States are thus built around components. States contain properties specific to a component and define transitions to other states that are triggered after the component executes.

[0088] Special case scenarios can be handled using the state section. For example, sometimes you may want to provide the user with the option to temporarily leave the first skill to let them do something in a second skill within the digital assistant. For example, if the user is busy in a conversation with a shopping skill (e.g., the user has made some purchase choices), the user may want to jump to the banking skill (e.g., the user may want to make sure he / she has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in the first skill can be configured to initiate an interaction with a second, different skill in the same digital assistant and then return to the original flow.

[0089] (6) Adding custom components to a skill bot - As described above, the states specified in the skill bot's dialog flow name the components that provide the required functionality corresponding to the state. Components enable the skill bot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured components for performing a variety of functions. The skill bot designer can select one or more of these pre-configured components and associate them with states in the skill bot's dialog flow. The skill bot designer can also use the tools provided by DABP 102 to create custom or new components and associate the custom components with one or more states in the skill bot's dialog flow.

[0090] (7) Testing and deploying skill robots - DABP 102 provides several features that enable skill robot designers to test the skill robots they are developing. The skill robots can then be deployed and included in the digital assistant.

[0091] Although the above description describes how to create a skill robot, similar techniques can also be used to create a digital assistant (or master robot). At the master robot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master robot) can handle without calling the skill robot associated with the digital assistant. Examples of system intents defined for the master robot include: (1) Exit: Applicable when the user signals that they want to exit the current session or context in the digital assistant; (2) Help: Applicable when the user requests help or directions; and (3) Unresolved Intent: Applicable to user input that does not quite match the exit intent and help intent. The digital assistant also stores information about one or more skill robots associated with the digital assistant. This information enables the master robot to select a specific skill robot to handle the utterance.

[0092] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing for determining how to route the utterance and the associated conversation. The digital assistant determines this using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance is to be routed to a specific skill for handling, to be handled by the digital assistant or master robot itself according to built-in system intents, or to be handled as a different state in the current conversation flow.

[0093] In some embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill robot using its invocation name. If the invocation name is present in the user input, the invocation name is considered to be an explicit invocation of the skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill robot for further processing. In some embodiments, if there is no specific invocation or explicit invocation, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and skill robot associated with the digital assistant. The score calculated for the skill robot or system intent indicates how likely it is that the user input represents the task that the skill robot is configured to perform or represents the system intent. Any system intent or skill robot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill robot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill robots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Any intent with a confidence score exceeding a threshold (e.g., 70%) is typically considered a candidate intent. If a specific skill robot is selected, the user utterance is routed to that skill robot for further processing. If a system intent is selected, one or more actions are performed by the master robot itself based on the selected system intent.

[0094] Figure 2 2 is a simplified block diagram of a master robot (MB) system 200 according to some embodiments. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multi-intention subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data storage 250. Figure 2The MB system 200 depicted in FIG. 1 is merely an example of a component arrangement in a master robot. A person skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have more than Figure 2 More or fewer systems or components than those shown, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0095] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language grammar analyzer 214. As indicated above, the utterance may be provided in various ways including audio or text. The utterance 202 may be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 may include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text.

[0096] Language detector 212 detects the language of utterance 202 based on the text of utterance 202. How utterance 202 is treated depends on the language, as each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of an utterance.

[0097] The language parser 214 performs grammatical analysis on the utterance 202 to extract part-of-speech (POS) tags for each language unit (e.g., word) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 can also tokenize the language units of the utterance 202 (e.g., convert each word into a separate token) and classify the words by inflected form. A lemma is the main form of a group of words as represented in a dictionary (e.g., "run" is the lemma of run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking of compound expressions, for example, combining "credit" and "card" into a single expression "credit card". The language parser 214 can also identify relationships between words in the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a particular noun) is a direct object, which part of the utterance is a preposition, etc. The results of the processing performed by the language parser 214 form the extracted information 205 and are provided as input to the MIS 220 along with the utterance 202 itself.

[0098] As indicated above, the utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, the utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in some embodiments, pre-processing may be performed, for example by the pre-processing subsystem 210, to identify a single sentence among the multiple sentences for multiple intent analysis and explicit call analysis. In general, the results produced by the MIS 220 and the EIS 230 are substantially the same regardless of whether the utterance 202 is processed at the level of a single sentence or as a single unit including multiple sentences.

[0099] MIS 220 determines whether utterance 202 indicates multiple intents. Although MIS 220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS 220 does not involve determining whether the intent of utterance 202 matches any intent that has been configured for the robot. Instead, the processing of determining whether the intent of utterance 202 matches the robot's intent can be performed by intent classifier 242 of MB system 200 or an intent classifier of a skill robot (e.g., Figure 3 The processing performed by MIS 220 assumes that there is a robot (e.g., a particular skill robot or the master robot itself) that can handle utterance 202. Therefore, the processing performed by MIS 220 does not need to know which robots are in the chatbot system (e.g., the identities of the skill robots registered with the master robot) or what intents have been configured for a particular robot.

[0100] To determine that the utterance 202 includes multiple intents, the MIS 220 applies one or more rules from a set of rules 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and may include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern may include a coordinating conjunction that connects two parts of a sentence (e.g., a conjunction), where the two parts correspond to different intents. If the utterance 202 matches the sentence pattern, it can be inferred that the utterance 202 represents multiple intents. It should be noted that an utterance with multiple intents does not necessarily have different intents (e.g., intents involving different robots or different intents within the same robot). Instead, an utterance can have different instances of the same intent, such as "Place a pizza order using payment account X, then place a pizza order using payment account Y".

[0101] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines which parts of utterance 202 are associated with each intent. MIS 220 constructs a new utterance for separate processing to replace the original utterance for each intent represented in the utterance containing multiple intents, for example, Figure 2 , utterance "B" 206 and utterance "C" 208 depicted in . Thus, the original utterance 202 can be split into two or more separate utterances that are handled one at a time. MIS 220 uses the extracted information 205 and / or based on an analysis of the utterance 202 itself to determine which of the two or more utterances should be handled first. For example, MIS 220 can determine that utterance 202 contains a marker word indicating that a particular intent should be handled first. The newly formed utterance corresponding to the particular intent (e.g., one of utterance 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (or has been temporarily paused), the next highest priority utterance (e.g., utterance 206 or the other of utterance 208) can then be sent to EIS 230 for processing.

[0102] EIS230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) contains a call name for a skill robot. In some embodiments, each skill robot in the chatbot system is assigned a unique call name that distinguishes the skill robot from other skill robots in the chatbot system. The list of call names can be saved in the data store 250 as part of the skill robot information 254. When the utterance contains a word that matches the call name, the utterance is considered an explicit call. If the robot is not explicitly called, the utterance received by EIS230 is considered a non-explicit call utterance 234 and is input to the intent classifier of the main robot (e.g., intent classifier 242) to determine which robot to use to handle the utterance. In some instances, the intent classifier 242 will determine that the main robot should handle the non-explicit call utterance. In other instances, the intent classifier 242 will determine the skill robot to which the utterance is routed for handling.

[0103] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that the master robot has to perform. For example, when there is an explicit call, the master robot may not have to perform any intent classification analysis (e.g., using intent classifier 242), or may have to perform a simplified intent classification analysis to select a skill robot. Thus, the explicit call analysis can enable the selection of a particular skill robot without resorting to intent classification analysis.

[0104] Moreover, there may be cases where there is functional overlap between multiple skill robots. This may happen, for example, if the intents handled by two skill robots overlap or are very close to each other. In this case, it may be difficult for the master robot to identify which of the multiple skill robots to select based solely on intent classification analysis. In this scenario, explicit calls make it clear that there is no ambiguity as to the specific skill robot to use.

[0105] In addition to determining that the utterance is an explicit call, EIS230 is also responsible for determining whether any part of the utterance should be used as input to the skill robot that is explicitly called. In particular, EIS230 can determine whether a part of the utterance is irrelevant to the call. EIS230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. EIS230 can send the part of the utterance that is irrelevant to the call to the called skill robot instead of sending the entire utterance received by EIS230. In some instances, the input of the called skill robot is simply formed by deleting any part of the utterance associated with the call. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using PizzaBot" is related to calling the pizza robot, but is not related to any processing to be performed by the pizza robot. In some cases, EIS230 can reformat the part to be sent to the called robot, for example to form a complete sentence. Thus, EIS 230 not only determines that there is an explicit call, but also determines what to send to the skill robot when there is an explicit call. In some instances, there may not be any text that can be entered into the robot being called. For example, if the utterance is "Pizza Bot", EIS 230 may determine that the pizza robot is being called, but there is no text to be processed by the pizza robot. In this scenario, EIS 230 may indicate to skill robot invoker 240 that there is nothing to send.

[0106] The skill robot caller 240 calls a skill robot in various ways. For example, the skill robot caller 240 may call a robot in response to receiving an indication 235 that a particular skill robot has been selected as a result of an explicit call. The indication 235 may be sent by the EIS 230 along with the input for the skill robot that was explicitly called. In this scenario, the skill robot caller 240 will hand over control of the session to the skill robot that was explicitly called. The skill robot that was explicitly called will determine an appropriate response to the input from the EIS 230 by treating it as an independent utterance. For example, the response may be to perform a specific action or to start a new session in a specific state, where the initial state of the new session depends on the input sent from the EIS 230.

[0107] Another way that the skill robot invoker 240 can invoke a skill robot is implicitly by using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different classifications, one classification for each skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. Parameters generated as a result of this training (e.g., a set of parameter values ​​for a machine learning model) can be stored as part of the skill robot information 254.

[0108] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training of the machine learning model can involve inputting at least a subset of utterances from example utterances associated with various skill robots to generate, as output of the machine learning model, inferences about which robot is the correct robot for handling any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., by backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0109] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot that indicates the likelihood that the skill robot can handle an utterance (e.g., a non-explicit call utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each system-level intent (e.g., help, exit) that has been configured. If a particular confidence score satisfies one or more conditions, the skill robot caller 240 will call the robot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must also exceed the next highest confidence score by a certain margin. When the confidence scores of multiple skill robots all exceed the threshold confidence score value, applying such a condition will enable routing to a particular skill robot.

[0110] After identifying the robot based on the evaluation of the confidence score, the skill robot invoker 240 hands over the processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is the skill robot. Further, the skill robot invoker 240 will determine what to provide as input 247 to the identified robot. As described above, in the case of an explicit invocation, the input 247 can be based on a portion of the utterance that is not relevant to the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invocation, the input 247 can be the entire utterance.

[0111] The data store 250 includes one or more computing devices that store data used by various subsystems of the master robot system 200. As explained above, the data store 250 includes rules 252 and skill robot information 254. The rules 252 include, for example, rules for determining by the MIS 220 when an utterance represents multiple intents and how to split utterances representing multiple intents. The rules 252 further include rules for determining by the EIS 230 which parts of the utterance that displays the calling skill robot are to be sent to the skill robot. The skill robot information 254 includes the calling names of the skill robots in the chatbot system, for example, a list of the calling names of all skill robots registered with a particular master robot. The skill robot information 254 may also include information used by the intent classifier 242 to determine the confidence score of each skill robot in the chatbot system, for example, parameters of a machine learning model.

[0112] Figure 3is a simplified block diagram of a skill robot system 300 according to some embodiments. The skill robot system 300 is a computing system that can be implemented in software only, hardware only, or a combination of hardware and software. In some embodiments, as Figure 1 In the depicted embodiment, the skill bot system 300 can be used to implement one or more skill bots within a digital assistant.

[0113] The skill robot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to Figure 2 The MIS 220 in FIG. 310 and provides similar functionality, including being operable to use rules 352 in data store 350 to determine: (1) whether an utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules for detecting multiple intents and for splitting utterances applied by MIS 310 are the same as the rules applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to Figure 1 The extracted information 205 in the skill robot system 300 may be generated using the language parser 214 or a language parser local to the skill robot system 300.

[0114] The intent classifier 320 can be combined with the above Figure 2 In some embodiments, the intent classifier 320 is trained in a similar manner to the intent classifier 242 discussed in the embodiments of the present invention and described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. For a specific skill robot, at least a subset of example utterances associated with the specific skill robot is used as training utterances to train the machine learning model of the intent classifier 320. The ground truth of each training utterance will be the specific robot intent associated with the training utterance.

[0115] The speech 302 may be received directly from the user or provided by the main robot. When the speech 302 is provided by the main robot, for example, as Figure 2As a result of the processing by MIS 220 and EIS 230 in the embodiment depicted in , MIS 310 may be bypassed to avoid duplicating processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a conversation that occurs after being routed to a skill robot, MIS 310 may process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0116] The intent classifier 320 is configured to match a received utterance (e.g., utterance 306 or 308) with an intent associated with the skill robot system 300. As explained above, a skill robot may be configured with one or more intents, each of which includes at least one example utterance associated with the intent and used to train the classifier. Figure 2 In an embodiment, the intent classifier 242 of the master robot system 200 is trained to determine confidence scores for individual skill robots and confidence scores for system intents. Similarly, the intent classifier 320 can be trained to determine confidence scores for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and is therefore more fine-grained. The intent classifier 320 can access intent information 354. For each intent associated with the skill robot system 300, the intent information 354 includes a list of utterances that represent the intent and explain the meaning of the intent and are typically associated with tasks that can be performed by the intent. The intent information 354 can further include parameters generated as a result of training on the list of utterances.

[0117] The conversation manager 330 receives, as an output of the intent classifier 320, an indication 322 of a particular intent, which is identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some instances, the intent classifier 320 is unable to determine any match. For example, if the utterance involves a system intent or an intent of a different skill robot, the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill robot system 300 may submit the utterance to the master robot for disposal, e.g., routing to a different skill robot. However, if the intent classifier 320 successfully identifies the intent within the skill robot, the conversation manager 330 will initiate a conversation with the user.

[0118] The session initiated by the session manager 330 is a session specific to the intent identified by the intent classifier 320. For example, the session manager 330 can be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine can include a default starting state (e.g., when the intent is invoked without any additional input) and one or more additional states, wherein each state is associated with an action to be performed by the skill robot (e.g., perform a purchase transaction) and / or a dialog to be presented to the user (e.g., a question, a response). Thus, the session manager 330 can determine an action / dialog 335 upon receiving an indication 322 that an intent is recognized, and can determine additional actions or dialogs in response to subsequent utterances received during the session.

[0119] The data storage 350 includes one or more computing devices that store data used by the various subsystems of the skill robot system 300. Figure 3 As depicted, data store 350 includes rules 352 and intent information 354. In some embodiments, data store 350 may be integrated into the data store of a host robot or digital assistant, e.g. Figure 2 The data is stored in 250.

[0120] Training data augmentation

[0121] It has been found that NER models are generally biased towards correctly classifying entities with categories (or classes) that are overrepresented (i.e., in the majority) in the training data used to train the NER model, rather than classifying entities with categories that are underrepresented (i.e., in the minority) in the training data. When the NER model pays more attention to entity values ​​in the input utterance that correspond to one or more overrepresented categories in the training data, the classification performance of the NER model is often affected by entity values ​​in the input utterance that correspond to one or more underrepresented categories in the training data. In some cases, the NER model may misclassify these entity values ​​(e.g., classifying London as a CURRENCY entity type instead of a LOCATION entity type). In other cases, the NER model may not even detect these entity values ​​(e.g., not even detecting New York as an entity, let alone classifying New York as a LOCATION entity type).

[0122] To overcome these problems, various embodiments relate to techniques for adaptively expanding training data that take into account the distribution of named entities of different categories within the training data. Data expansion is a technique for artificially increasing the number of examples in the training data by generating modified training examples from existing training examples within the training data. For example, the training data includes original utterances (examples of natural language texts), wherein each original utterance includes at least one named entity corresponding to a named entity category. The training data can be expanded by using adaptive expansion techniques to generate additional utterances (e.g., modified examples of natural language texts) from the original utterances in the training data and combining the generated additional utterances with the original utterances to form expanded training data. The adaptive expansion techniques described herein generate an adaptive number of expanded utterances based on the distribution of the classifications of named entities in the training data to ensure that sufficient utterances including named entities corresponding to a minority of classified entities are generated during the training data expansion process. Therefore, the distribution of named entities corresponding to each named entity classification can be normalized across the training data so that during training, the NER model is less likely to become biased towards learning entities corresponding to the majority classifications. The NER model can be trained using adaptively augmented training data, which improves performance compared to a NER model trained using only static augmented training data. The NER model trained using training data adaptively augmented using the techniques described herein can be implemented in a chatbot system, such as a chatbot system for Figure 1 , Figure 2 and Figure 3 The described chatbot system. Advantageously, the models and chatbots perform better on utterances in various text formats because the models are better able to identify and extract entities in the utterances.

[0123] Figure 4A A block diagram is shown illustrating aspects of an artificial intelligence modeling system 400 configured to train and provide one or more models. Figure 4A As shown, in this example, the modeling system 400 includes various subsystems: a prediction model training subsystem 402 for building and training models; an evaluation subsystem 414 for evaluating the performance of the trained models; and an implementation subsystem 412 for implementing the models into artificial intelligence-based solutions (e.g., deployed as part of one or more chatbots). The prediction model training subsystem 402 builds and trains one or more prediction models 416 for use by other subsystems. A -416 N('N' represents any natural number) (these prediction models may be referred to herein individually or collectively as prediction models 416). For example, prediction models 416 may include a model for identifying one or more entities in an utterance (e.g., NER model 426), another model for determining the likelihood that an utterance represents a task that a particular skill robot is configured to perform, another model for predicting intent based on an utterance of a first type of skill robot, and another model for predicting intent based on an utterance of a second type of skill robot. Other types of prediction models may be implemented in other examples according to the present disclosure.

[0124] The prediction model 416 can be a machine learning ("ML") model, such as a convolutional neural network ("CNN"), for example, an initial neural network, a residual neural network ("Resnet"); or a recurrent neural network, such as a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model; other variants of a deep neural network ("DNN") (e.g., a multi-label n-point DNN classifier or a multi-class DNN classifier for single intent classification. The prediction model 416 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, or a random forest model. Model), shallow neural network, or a combination of one or more of such techniques - for example, CNN-HMM or MCNN (multi-scale convolutional neural network). Modeling system 400 can employ the same type of prediction model 416 or different types of prediction models 416 to identify one or more entities in an utterance, determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform, predict intent based on an utterance of a first type of skill robot, and predict intent based on an utterance of a second type of skill robot.

[0125] To train various predictive models 416, the predictive model training subsystem 402 includes four main components: a dataset preparation component 404, a hyperparameter tuning component 406, a data augmentation component 408, and a model training component 410. The dataset preparation component 404 loads data assets 418 into the predictive model training system 402, pre-processes the data assets 418, and splits the data assets 418 into training, validation, and test sets 422. A-N(which may be referred to herein individually or collectively as sets 422) for configuring, training, and testing predictive models 416. Splitting data assets 418 into sets 422 may be performed randomly (e.g., 80 / 10 / 10% or 60 / 30 / 40%), or the split may be performed according to more sophisticated validation techniques (such as K-fold cross validation, leave-one-out cross validation, leave-one-out cross validation, nested cross validation, etc.) to minimize sampling bias and overfitting.

[0126] Each of these collections 422 may include at least a subset of utterances from example utterances associated with one or more skill robots. As indicated above, utterances may be provided in various ways including audio or text. Discourses may be sentence fragments, complete sentences, multiple sentences, etc. For example, if utterances are provided as audio, the data preparation component 404 may convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation marks (e.g., commas, semicolons, periods, etc.) into the result text. In some instances, example utterances are provided by a client or customer. In other instances, example utterances are automatically generated from a previous utterance library (e.g., identifying utterances specific to the skills that the chatbot is designed to learn from the library). Collection 422 may include input text or audio (or input features of text or audio frames) and labels 420 corresponding to the input text or audio (or input features) as a matrix or value table. For example, for each training utterance, an indication of the correct entity and its classification to be inferred by the prediction model 416 may be provided as basic fact information for label 420. The behavior of the predictive model 416 may then be adapted (eg, via back-propagation) to minimize the discrepancy between the inferences generated for the various entities and the ground truth information.

[0127] Hyperparameter tuning component 406 performs a hyperparameter tuning process, which selects hyperparameters for configuring prediction model 416. Hyperparameters are settings that can be adjusted or optimized to control the behavior of prediction model 416. Most models explicitly define hyperparameters that control different aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined to adapt the model to a specific scenario. For example, additional hyperparameters can be defined to determine the rate for adaptively expanding training data, the number of hidden units or hidden layers of the model, the learning rate of the model, the convolution kernel width, and / or the number of parameters of the model. The hyperparameter tuning process is performed in the following manner: identify N hyperparameter sets (e.g., via hyperparameter search techniques such as grid search, Bayesian optimization, etc.) and configure N instances of the model with these N hyperparameter sets, wherein each instance of the model is configured with a hyperparameter set in these N hyperparameter sets. The hyperparameter tuning process continues by training each instance of the model with a training data set extracted from the set 422 (e.g., a validation set of the training data), inputting test data extracted from the set 422 (e.g., a test set of the training data) into each instance of the model, generating predictions for the test data with each instance of the model, and evaluating the performance of each instance of the model by comparing the predictions generated by the corresponding instance with the labels in the corresponding training data set used to train the corresponding instance of the model (e.g., k-fold cross validation). The hyperparameter tuning process continues by determining the hyperparameter set that produces the best performing model instance among all instances of the model (i.e., which of the model instances predicts the labels of the training data of the corresponding instance with the greatest accuracy) among the N hyperparameter sets, and configuring the model using the determined hyperparameter set. The accuracy of the model can be expressed as a general numerical metric that can be maximized or minimized according to the user's preferences. Searching for a hyperparameter set to maximize or minimize the metric can be cumbersome, so search algorithms such as grid search and random search can be used. Grid search selects a grid of hyperparameter values ​​and evaluates all hyperparameter values. Requires guesswork to specify the minimum and maximum values ​​for each hyperparameter. Random search randomly samples values ​​for random points on a grid. Smart hyperparameter tuning picks several hyperparameter settings, evaluates a validation matrix, tunes the hyperparameters, and re-evaluates the validation matrix. Examples of smart hyperparameters are Spearmint (hyperparameter optimization using Gaussian processes) and Hyperopt (hyperparameter optimization using tree-based estimators).

[0128] The data augmentation component 408 augments the training data to artificially increase the amount of training data. In some instances, the data augmentation component 408 generates new data examples from existing data examples (e.g., by combining examples with minor changes with existing data examples) to increase the amount of training data and trains the prediction model 416 with the augmented training data. Data augmentation reduces overfitting (and improves generalization) of the prediction model 416 (e.g., making the prediction model 416 more resilient to discourse with entity changes and new and unseen entities). In addition, by training the prediction model 416 with augmented training data, the prediction model 416 can better consider context when detecting and classifying entities in the discourse, even when the named entities in the discourse do not match the named entities in the training data. In addition, the prediction model 416 can also learn to detect and classify entities in the discourse even if the entities in the discourse include typos and / or otherwise have spelling changes.

[0129] Additional utterances are generated from utterances in the training data by replacing named entities in the utterance with different named entities, such as those selected from a gazetteer. A template utterance can be generated by removing named entities from the utterance, and the additional utterance can be generated by populating the template utterance with named entities selected from a gazetteer that are different from the named entities of the utterance. The populated template utterance can be combined with the utterance to form augmented training data, which is then used to train the predictive model 416. A detailed description of generating additional utterances using a gazetteer is described in commonly owned U.S. patent application No. 18 / 185,675 filed on March 17, 2023, which is incorporated herein by reference in its entirety for all purposes.

[0130] The template utterances and the additional utterances are generated using an adaptive augmentation technique. As described in further detail below, the adaptive augmentation technique generates an adaptive number of augmented utterances based on the distribution of categories of named entities in the training data to ensure that sufficient utterances including named entities corresponding to minority classification entities are generated during the training data augmentation process. Thus, the distribution of named entities corresponding to each named entity category can be normalized across the training data so that during training, the prediction model 416 is less likely to become biased towards learning entities corresponding to the majority classification.

[0131] The model training component 410 trains the prediction model 416 by selecting model parameters (e.g., weights and / or biases) for the prediction model 416 and performing an iterative operation of inputting utterances from the augmented training data into the prediction model 416 so as to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes an objective function (e.g., minimizes the loss function of the prediction model 416). The objective function can be constructed to measure the difference between the output inferred using the prediction model 416 and the basic facts annotated to the sample using the label 420. For example, for a supervised learning-based model, the goal of training is to learn a function "h()" (sometimes also referred to as a hypothesis function) (h:X→Y) that maps the training input space X to the target value space Y, such that h(x) is a good predictor of the corresponding value y. A variety of different techniques can be used to learn the hypothesis function. In some techniques, as part of deriving the hypothesis function, an objective function that measures the difference between the basic fact value of the input and the predicted value of the input can be defined. As part of the training, the model parameters are updated in a way that minimizes or maximizes the objective function using techniques such as back-propagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), Hebbian learning, Adam optimization, etc.

[0132] Once the set of model parameters is identified (i.e., the prediction model 416 is trained), the prediction model 416 can be tested or validated using test / validation data extracted from the set 422. The testing or validation process includes an iterative operation of inputting utterances from the test / validation data into the prediction model 416 and evaluating the performance of the prediction model 416 using validation techniques (e.g., K-fold cross validation, leave one out cross validation, leave one group out cross validation, nested cross validation, etc.). The predictions (e.g., one or more classified named entities) generated by the prediction model 416 can be evaluated by comparing these predictions with the ground truth named entities using correlation techniques (e.g., the Bland-Altman method and the Spearman rank correlation coefficient). Further, performance indicators 428, such as error, accuracy, precision, recall, receiver operating characteristic (ROC) curves, etc., can be calculated by the evaluation subsystem 414. The performance indicators 428 can be used by the evaluation subsystem 414 to analyze the performance of the prediction model 416 in classifying named entities.

[0133] Once the prediction model 416 has been tested and / or verified, it can be output or deployed. For example, the prediction model training subsystem 402 can output or deploy the NER model 426. The NER model 426 can be output or deployed to the prediction model training subsystem 402 and / or the implementation subsystem 412 and used therein for implementing artificial intelligence-based solutions, such as chatbots. For example, the chatbot can use the NER model 426 to predict (i.e., identify and classify) named entities 430 in the input utterance 424 received by the chatbot, and use the predicted named entities 430 in downstream processing such as dialogue construction, data acquisition, intent classification, skill robot invocation, etc. In another example, the predicted named entities 430 can be used to further train the NER model 426 and / or other prediction models 416.

[0134] Although not explicitly shown, it should be understood that the modeling system 400 can further include a developer device associated with a developer. Communications from the developer device to components of the modeling system 400 can indicate what types of input data, utterances, and / or database schemas are to be used for the models, the number and type of models to be used, the hyperparameters for each model (e.g., learning rate and number of hidden layers), how data requests are formatted, which training data to use (e.g., and how to obtain the training data), and which validation techniques to use, and / or how the controller process is to be configured.

[0135] Adaptive scaling technology and speed

[0136] As discussed above, intent prediction and entity extraction help the chatbot system understand user queries and user utterances in the field of a given service or service set. Intent prediction determines the purpose (i.e., intent) of a user query or utterance. Entity extraction determines one or more entities (i.e., constraints) of a user query or utterance (if any). For example, for a user query about "the weather on Wednesday in the Poconos", intent prediction determines that the user's intent is to understand "weather", and entity extraction determines that "Wednesday" and "Poconos" are entities that focus the user's intent on a specific date and geographic location. Entity detection and classification can be done using a NER model. Entity tags can be assigned to each instance of a named entity within a user utterance. In some examples, a named entity can correspond to a named entity category, such as a location, organization, time, person, etc. For a given utterance, a word in the utterance that refers to an entity within an entity category can be marked with a tag that identifies the category. For example, for the utterance "My name is Jane Doe", the word "Jane Doe" may be detected and classified as corresponding to a person, and may be labeled with a label identifying the category into which it has been classified (e.g., [ PER JaneDoe]). Since the words "My name is" are contextually relevant, they will not be detected and classified as named entities, nor will they be assigned tags identifying them as named entities. In another example, for the utterance "What is the weather in Paris", the word "Paris" will be detected and classified as corresponding to a location and will be labeled as "[ LOC Paris]". The utterance may include named entities belonging to different named entity categories, and each named entity in the utterance may be detected, classified according to its named entity category, and tagged with a label identifying its corresponding category. For example, for the utterance "Her name is Jane Doe and she livesin Paris", the word "Jane Doe" may be detected, classified, and tagged as [ PER Jane Doe], and the word “Paris” can be detected, classified, and labeled as [ LOC Paris].

[0137] In order to predict the classification labels representing named entities, NER models are usually trained with training data including each entity category that is expected to be classified. For example, in order for a NER model to detect and classify the word "Jane Doe" as a named entity corresponding to a person and "Paris" as a named entity corresponding to a location, it will be necessary to train the NER model with training data containing named entities in the named entities of the person category and named entities in the named entities of the location category. Training data is usually obtained from public or private pre-labeled datasets. There are several datasets available for training purposes, including text with labeled named entities corresponding to the names of people, locations, organizations, and other entities that do not belong to the first three groups. An example of a publicly available dataset from which training data can be obtained is the CoNLL-2003 dataset. For additional information about the CoNLL-2003 dataset, see “Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition” by Sang et al., published on arXiv preprint cs / 0306050(2003), the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0138] But the problem with public or private pre-labeled datasets is that these datasets do not include sufficiently diverse pre-labeled data to train models to detect entities with various contexts and entity value variations (currency types, number formats, data time formats, different spelling conventions, etc.). Therefore, NER models trained with these pre-labeled datasets and other similar pre-labeled datasets usually learn to focus on the named entity values ​​they are trained on, while ignoring contextual information. As a result, it is difficult for NER models to recognize and label named entities that the NER model is not specifically trained to recognize and label (i.e., named entities that are not in the training data and / or named entities with different spellings, misspellings, variations, etc.). In addition, these datasets are usually unbalanced in their representation of different entity categories, so some entity categories are overrepresented in the dataset, while some entity categories are underrepresented. Therefore, NER models trained with these datasets are usually biased towards learning majority-class entities and often misclassify minority-class entities.

[0139] Prior attempts to address this problem include generating customized labeled datasets (e.g., by crowdsourcing workers) or users or customers of the NER model; however, generating these customized labeled datasets utilizes a large amount of resources (e.g., human and financial resources). Other previous attempts to address this problem include automatically generating additional utterances by replacing entities in the utterance with other entities selected from entity sources such as gazetteers; however, adding a static number of augmented utterances under this approach may further tilt the dataset toward the majority classified entities represented in the dataset, thereby adversely affecting the performance of the NER model for the minority classified entities represented in the dataset. To overcome these and other challenges, the method described herein utilizes an adaptive augmentation technique that takes into account the distribution of different entity types within the training data. The adaptive augmentation technique generates an adaptive number of augmented utterances based on the distribution of the entities to ensure that a sufficient number of utterances including minority classified entities are generated during the augmentation of the training data.

[0140] The process for adaptively expanding training data begins with accessing training data including utterances. The training data can be accessed from a database (not shown) that stores training data for training a predictive model. The utterances in the training data can be referred to as raw utterances. Each raw utterance can include one or more named entities corresponding to one or more named entity categories in a set of named entity categories. In some embodiments, the set of named entity categories can correspond to default or system named entities (e.g., PERSON, NUMBER, CURRENCY, DATE_TIME). In other embodiments, the set of named entity categories can be defined by users, customers, developers, etc. of the NER model. Each named entity in the utterance can include a label indicating the category to which the named entity belongs. Each category in the set of named entity categories can be associated with a different label. For example, a person named entity category can be associated with [ PER ] tag, and the location named entity category can be associated with [ LOC ] is associated with. Each utterance in the training data may also include contextual information before and / or after one or more named entities in the utterance. For example, for the utterance “His name is[ PER John Doe](His name is [ PER John Doe]), the “His name is” part of the utterance is in the named entity [ PER John Doe]. The training data can be accessed from one or more public or private pre-labeled datasets (such as the CoNLL-2003 dataset mentioned above).

[0141] The process for adaptively augmenting the training data proceeds to determine the rate at which the original utterances of each named entity category represented in the training data should be augmented (which may be referred to herein as the augmentation rate) for each named entity category represented in the training data. The augmentation rate defines how many additional instances of each named entity category represented in the training data should be included in the training data to balance the distribution of named entity categories in the training data. For example, for a utterance including the named entity category [ PER ] appears 75 times, named entity category [ LOC ] appears 72 times, named entity category [ DATE_TIME ] appeared 29 times, named entity category [ CURRENCY ] appears 21 times and the named entity category [ MERCHANT ]The training data consists of 100 utterances with 68 occurrences. The augmentation rate for each named entity category can define how many additional instances of each named entity category should be included in the training data to balance the distribution of named entity categories.

[0142] The expansion rate of each named entity category can be calculated by the hyperparameter tuning component 406. The hyperparameter tuning component 406 can calculate the expansion rate of each named entity category in the following manner: for each named entity category, a function for calculating the expansion rate is selected from a plurality of functions for calculating the expansion rate; and based on the distribution of the corresponding named entity category in the training data, the expansion rate of the corresponding named entity category is calculated using the selected function. The multiple functions for calculating the expansion rate may include a first function and a second function. The first function can calculate the expansion rate of a given named entity category based on an exponential function (exp_beta_norm) that receives a normalized distribution value and a beta value (beta_value) of a given named entity category as input. An example of the first function is exp_beta_norm: lamba: np.array([math.exp(v / sum_dist_entities*beta value)for v in dist_entities.values()]). The normalized distribution value for a given named entity category can be calculated by counting the number of occurrences of the given named entity category in the training data (dist_entities), summing the number of occurrences of all named entity categories in the training data (sum_dist_entities), and dividing the dist_entities of the given named entity category by the sum_dist_entities of all named entity categories that appear in the training data. The β value can be selected from a range of values, such as [-1.0, 1.0]. The second function can calculate the expansion rate of a given named entity category based on a base-two logarithmic function that receives the dist_entities of a given named entity category as input. An example of a second function is log2_norm: lamba: np.array([math.log2(v) for v in dist_entities.values()]).

[0143] As discussed above, the hyperparameter tuning component 406 defines hyperparameters that can be selected by a hyperparameter tuning process that determines the hyperparameters for configuring the NER model 426. The hyperparameter tuning component 406 can define a function selection hyperparameter (approach_select) for selecting a function for calculating an expansion rate from a plurality of functions for calculating an expansion rate. For example, using the approach_select hyperparameter (approach_select={first function, second function, ...), the hyperparameter tuning component 406 can select a function for calculating an expansion rate from a plurality of functions for calculating an expansion rate during the hyperparameter tuning process. The hyperparameter tuning component 406 can also define a β value selection hyperparameter for selecting beta_value during the hyperparameter tuning process. The function selection hyperparameter and the β value selection hyperparameter can be one of the above-mentioned N hyperparameter sets identified using a hyperparameter search technique. For each hyperparameter set in the N hyperparameter sets, the training data for the corresponding instance of the training model can be expanded according to the expansion rate calculated based on the corresponding hyperparameter set. In this manner, during a hyperparameter tuning process, an augmentation rate for named entity categories in the training data may be determined based on which augmentation rate produces a best performing model for instances of the model configured during the hyperparameter tuning process.

[0144] The process for adaptively expanding the training data continues to generate additional utterances that can be combined with the original utterances in the training data to form expanded training data. The additional utterances can be generated at an expansion rate for each named entity category calculated by the hyperparameter tuning component 406. In order to generate additional utterances for each named entity category, the original utterances in the training data that include at least one instance of the corresponding named entity category can be transformed into template utterances, which can be filled with selected named entities of the same named entity category as the corresponding named entity category. The expansion rate of the corresponding named entity category can determine the number of template utterances required to balance the training data of the corresponding named entity category. The named entities used to fill the template can be selected from a named entity source (such as a place name dictionary). The original utterance in the training data can be transformed into a template utterance by removing one or more named entities from the original utterance and inserting a placeholder identifier for each of the one or more removed named entities. The placeholder identifier can represent the named entity category of the named entity removed for which the placeholder identifier is inserted. For example, the original utterance "My name is[ PER Jane Doe](My name is [ PER Jane Doe])" can be transformed into the template utterance "My name is[ PER ](My name is[ PER ])", where the named entity corresponding to the named entity category of personPER ] placeholder identifier is inserted into the original utterance because the named entity [JaneDoe] corresponds to a person. The template utterance can be populated with one or more named entities selected from one or more entity sources by replacing each placeholder identifier in the template utterance with the selected named entity corresponding to the named entity category of the placeholder identifier. For example, continuing the example above, to populate the template utterance "My name is [ PER ](My name is[ PER ])", replace the placeholder identifier [ PER ] is replaced by a place name from the place dictionary that includes the person entity (e.g., GZ [PER] : {[Jane Doe]; [Jae Doe]; [Jane Do]; [Janes Doey]; ...). Once the template utterances are populated, the generated additional utterances are added to the original utterances in the training data to form augmented training data.

[0145] NER Model

[0146] As mentioned above about Figure 4A As discussed, the predictive model training subsystem 402 outputs trained models that have been trained with augmented training data, including the NER model 426. The NER model 426 may be implemented in a system or an artificial intelligence based solution such as a chatbot system. Figure 4B A simplified block diagram of a system 470 is shown in FIG. 4 , in which a trained NER model 472 may be implemented. Figure 4B As shown, the system 470 may include a trained NER model 472. Using the trained NER model 472, the system 470 may receive an input utterance 480 (e.g., My name is John and I live in Texas) and generate an output utterance 490, which includes the input utterance 480 labeled with tags that identify the named entities in the input utterance 480 and the categories to which the identified named entities belong (e.g., My name is [ PER My name is [John] PER John]) and I live in[ LOCATION Texas](I live in LOCATION Texas])).

[0147] The trained NER model 472 can be based on a transformer-based model, such as a bidirectional transformer-based encoder representation (BERT) model. In some embodiments, the trained NER model 472 can include a BERT layer 474, a convolutional neural network / bidirectional long short-term memory (CNN / BiLSTM) layer 476, and a conditional random field (CRF) layer 478.

[0148] The BERT layer 474 may include an algorithm that accepts a sequence of words of an utterance (such as the input utterance 480) as input and generates a feature vector or word embedding for the words of the sequence. In some embodiments, the BERT layer 474 includes a transformer layer that includes an encoder (e.g., an attention mechanism and a feedforward network). The attention mechanism can generate attention scores for the words of the sequence, and the feedforward network can encode the words of the sequence into word embeddings based on the attention scores.

[0149] The CNN / BiLSTM layer 476 may include an algorithm that accepts word embeddings from the BERT layer 474 and generates sentence or string embeddings for a sequence of words. The CNN portion may accept word embeddings from the BERT layer 474 and generate character-level vector representations for each character of each word in the sequence of words, and the BiLSTM portion may accept word embeddings from the BERT layer 474 and character-level vector representations from the CNN portion and generate sentence or string embeddings for a sequence of words.

[0150] The CRF layer 478 may include an algorithm that accepts a sentence or string embedding as input, detects whether a word in a word sequence is a named entity, and classifies the detected named entity into the named entity type to which it belongs. In some embodiments, the CFR layer 478 may output the classified detected named entities and identifiers that identify which words in the word sequence the detected named entities correspond to.

[0151] In some implementations, system 470 may receive detected named entities and identifiers from NER model 472 and generate an output utterance, such as output utterance 490 labeled with the named entities contained therein (e.g., My name is [ PER My name is [John] PER John]) and I live in[ LOCATION Texas](I live in LOCATIONTexas])). The system 470 can use the labeled output utterance 490 to perform operations such as querying a database, generating a response, and displaying the generated response on a display device or client device (not shown) of the system 470. Using the training data augmentation techniques described herein, the NER model 472 can accurately identify named entities in the input utterance, which enables the system 470 to generate improved responses and improve the performance of other functions (e.g., understanding language).

[0152] The described construction, operation, function and advantages of the system 470 and the NER model 472 are not intended to be limiting, and other constructions, operations, functions and advantages are possible and included in this specification. In addition, although not explicitly shown, it should be understood that the system 470 may further include a developer device associated with a developer. Communication from the developer device to the components of the system 470 can indicate which types of input data and / or utterances will be used for the model, the number and type of models to be used, the hyperparameters of each model (e.g., the learning rate and the number of hidden layers), the formatting of data requests, which training data to use (e.g., and how to obtain the training data), and which verification technique to use, and / or the configuration of the controller process.

[0153] Illustrative methods

[0154] Figure 5A is a flow chart illustrating an example process 500 for adaptively augmenting training data in accordance with certain embodiments. Figure 5A The depicted processes may be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the respective systems. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 5A The methods presented and described below are intended to be illustrative rather than limiting. Figure 5A The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order or some steps may be performed in parallel. In some embodiments, such as Figures 1 to 4B In the embodiment depicted in Figure 5A The processing depicted in can be performed by a pre-processing subsystem (eg, pre-processing subsystem 210 or predictive model training subsystem 402) to generate augmented training data for training a NER model.

[0155] At box 502, training data is accessed. The training data can be accessed from a database that stores training data for training a prediction model. The utterances in the training data can be referred to as raw utterances. Each raw utterance can include one or more named entities corresponding to one or more named entity categories in a set of named entity categories. In some embodiments, the set of named entity categories can correspond to default or system named entities (e.g., PERSON, NUMBER, CURRENCY, DATE_TIME). In other embodiments, the set of named entity categories can be defined by a user, customer, developer, etc. of the NER model. Each named entity in the utterance can include a label indicating the category to which the named entity belongs. Each category in the set of named entity categories can be associated with a different label. For example, a person named entity category can be associated with [ PER ] tag, and the location named entity category can be associated with [ LOC ] is associated with. Each utterance in the training data may also include contextual information before and / or after one or more named entities in the utterance. For example, for the utterance “His name is[ PER John Doe] (His name is [ PER John Doe]), the “His name is” part of the utterance is in the named entity [ PER John Doe]. The training data can be accessed from one or more public or private pre-labeled datasets (such as the CoNLL-2003 dataset mentioned above).

[0156] At block 504, a hyperparameter tuning process for calculating an expansion rate is performed. An expansion rate may be calculated for each named entity class represented in the training data, and may be a rate at which original utterances in the training data that include these named entity classes should be expanded. The expansion rate may define how many additional instances of each named entity class represented in the training data should be included in the training data to balance the distribution of named entity classes in the training data. For example, for a named entity class [ PER ] appears 75 times, named entity category [ LOC ] appears 72 times, named entity category [ DATE_TIME ] appeared 29 times, named entity category [ CURRENCY ] appears 21 times and the named entity category [ MERCHANT ]The training data consists of 100 utterances with 68 occurrences. The augmentation rate for each named entity category can define how many additional instances of each named entity category should be included in the training data to balance the distribution of named entity categories.

[0157] The expansion rate of each named entity category can be calculated by performing a hyperparameter tuning process. The hyperparameter tuning process can calculate the expansion rate of each named entity category in the following manner: for each named entity category, a function for calculating the expansion rate is selected from a plurality of functions for calculating the expansion rate; and based on the distribution of the corresponding named entity category in the training data, the expansion rate of the corresponding named entity category is calculated using the selected function. The multiple functions for calculating the expansion rate may include a first function and a second function. The first function can calculate the expansion rate of a given named entity category based on an exponential function (exp_beta_norm) that receives a normalized distribution value and a beta value (beta_value) of a given named entity category as input. An example of the first function is exp_beta_norm: lamba: np.array([math.exp(v / sum_dist_entities*beta value)for v in dist_entities.values()]). The normalized distribution value for a given named entity category can be calculated by counting the number of occurrences of the given named entity category in the training data (dist_entities), summing the number of occurrences of all named entity categories in the training data (sum_dist_entities), and dividing the dist_entities of the given named entity category by the sum_dist_entities of all named entity categories that appear in the training data. The β value can be selected from a range of values, such as [-1.0, 1.0]. The second function can calculate the expansion rate of a given named entity category based on a base-two logarithmic function that receives the dist_entities of a given named entity category as input. An example of a second function is log2_norm: lamba: np.array([math.log2(v) for v in dist_entities.values()]).

[0158] Hyperparameters that can be selected by a hyperparameter tuning process can be defined. The hyperparameter tuning process can define a function selection hyperparameter (approach_select) for selecting a function for calculating an expansion rate from a plurality of functions for calculating an expansion rate. For example, using the approach_select hyperparameter (approach_select={first function, second function, ...), the hyperparameter tuning process can select a function for calculating an expansion rate from a plurality of functions for calculating an expansion rate during the hyperparameter tuning process. The hyperparameter tuning process can also define a β value selection hyperparameter for selecting beta_value during the hyperparameter tuning process. The function selection hyperparameter and the β value selection hyperparameter can be one of the above-mentioned N hyperparameter sets identified using a hyperparameter search technique. For each hyperparameter set in the N hyperparameter sets, the training data for the corresponding instance of the training model can be expanded according to the expansion rate calculated based on the corresponding hyperparameter set. In this way, during the hyperparameter tuning process, the expansion rate of the named entity category in the training data can be determined based on which expansion rate produces the best performance model of the instance of the model configured during the hyperparameter tuning process.

[0159] At box 508, additional utterances are generated based on the calculated expansion rate. The additional utterances may be generated at the expansion rate for each named entity category calculated during the hyperparameter tuning process. In order to generate additional utterances for each named entity category, original utterances in the training data that include at least one instance of the corresponding named entity category may be transformed into template utterances, which may be populated with selected named entities of the same named entity category as the corresponding named entity category. The expansion rate of the corresponding named entity category may determine the number of template utterances required to balance the training data for the corresponding named entity category. The named entities used to populate the template may be selected from a named entity source (such as a place name dictionary). The original utterance in the training data may be transformed into a template utterance by removing one or more named entities from the original utterance and inserting a placeholder identifier for each of the removed one or more named entities. The placeholder identifier may represent the named entity category of the named entity for which the placeholder identifier was inserted. For example, the original utterance "My name is[ PER Jane Doe](My name is [ PER Jane Doe])" can be transformed into the template utterance "My name is[ PER ](My name is[ PER ])", where the named entity corresponding to the named entity category of person PER] placeholder identifier is inserted into the original utterance because the named entity [Jane Doe] corresponds to a person. By replacing each placeholder identifier in the template utterance with the selected named entity corresponding to the named entity category of the placeholder identifier, the template utterance can be populated with one or more named entities selected from one or more entity sources. For example, continuing the example above, in order to populate the template utterance "My name is [ PER ](My name is[ PER ])", replace the placeholder identifier [ PER ] is replaced by a place name from the place dictionary that includes the person entity (e.g., GZ [PER] : The named entities selected from {[Jane Doe]; [Jae Doe]; [Jane Do]; [Janes Doey]; …).

[0160] At block 510, the training data is augmented to form augmented training data. The training data is augmented by adding the generated additional utterances to original utterances in the training data.

[0161] Figure 5B is a flow chart illustrating an example process 550 for providing a NER model in accordance with some embodiments. Figure 5B The depicted processes may be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 5B The methods presented and described below are intended to be illustrative rather than limiting. Figure 5B The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order or some steps may be performed in parallel. In some embodiments, such as Figures 1 to 4B In the embodiment depicted in Figure 5B The processing depicted in can be performed by a processing subsystem (e.g., the predictive model training subsystem 402, the evaluation subsystem 414, and / or the implementation subsystem 412) to train a NER model with augmented training data and deploy the trained NER model.

[0162] At box 552, the NER model is trained. The NER model can be trained using the obtained and / or expanded training data, as described above. Training the NER model includes: selecting model parameters for the NER model, and performing an iterative operation of inputting the utterances from the expanded training data into the NER model, so as to find a set of model parameters (e.g., weights and / or biases) for the NER model, which maximizes or minimizes the objective function, for example, minimizes the loss function. Each training iteration may involve finding a set of model parameters for the NER model so that the objective function value using the model parameter set is less than the objective function value using a different model parameter set in the previous iteration. The objective function can be constructed to measure the difference between the output inferred using the NER model and the expanded training data using the label. Once the model parameter set is identified, the NER model can be tested or verified using a test data subset (test or validation data set). The testing or validation process includes an iterative operation of inputting the utterances from the test data subset into the NER model using a validation technique (such as K-fold cross validation, leave one cross validation, leave one group cross validation, nested cross validation, etc.). Once the best set of model parameters is obtained, a reserved test set from the test data subset can be input into the NER model to obtain the output (in this example, one or more recognized entities), and the output and the ground truth entities are evaluated using correlation techniques (such as the Bland-Altman method and the Spearman rank correlation coefficient). Further, performance indicators such as error, accuracy, precision, recall, receiver operating characteristic (ROC) curves, etc. can be calculated. Performance indicators can be used to analyze the performance of the NER model.

[0163] At box 554, a trained NER model is provided. The NER model can be provided to a system or an artificial intelligence-based solution (such as a chatbot system) or deployed or implemented therein. For example, the trained NER model can be deployed within a chatbot system (e.g., as part of a skill robot) for identifying one or more entities in utterances in various languages ​​and / or text formats. In some instances, an utterance can be received (e.g., by a skill robot), the utterance can be tokenized into word units, and the utterance and word units can be analyzed by a trained NER model to predict classification labels for the word units. In some instances, word units are identified as named entities, and the predicted classification labels represent the named entities. Using the trained NER model, the system can receive an input utterance and generate an output utterance, which includes the input utterance labeled with predicted classification labels, which identify the named entities in the input utterance and the classification to which the identified named entities belong. The system can use the labeled output utterance to perform operations such as querying a database (e.g., looking up information related to a word-unit based on the predicted classification label and / or the value of the named entity), generating a response (e.g., generating a dialogue with the user based on the predicted classification label and / or the value of the named entity), displaying the labeled output utterance and / or the generated response on a display device of the system or a client device (not shown), or any combination thereof.

[0164] Descriptive System

[0165] Figure 6 Depicted is a simplified diagram of a distributed system 600. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608 coupled to a server 612 via one or more communication networks 610. The client computing devices 602, 604, 606, and 608 may be configured to execute one or more applications.

[0166] In various examples, server 612 can be adapted to run one or more services or software applications that implement one or more embodiments described in the present disclosure. In some examples, server 612 can also provide other services or software applications that can include non-virtual environments and virtual environments. In some examples, these services can be provided to users of client computing devices 602, 604, 606, and / or 608 as web-based services or cloud services (such as under a software as a service (SaaS) model). Users operating client computing devices 602, 604, 606, and / or 608 can then utilize one or more client applications to interact with server 612 to utilize the services provided by these components.

[0167] exist Figure 6In the depicted configuration, the server 612 may include one or more components 618, 620, and 622 that implement the functions performed by the server 612. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be appreciated that a variety of different system configurations that may differ from the distributed system 600 are possible. Thus, Figure 6 The illustrated example is one example of a distributed system for implementing the example system and is not intended to be limiting.

[0168] A user may use client computing devices 602, 604, 606, and / or 608 to execute one or more applications, models, or chatbots that may generate one or more events or models that may then be implemented or serviced according to the teachings of the present disclosure. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although Figure 6 Only four client computing devices are depicted, but any number of client computing devices may be supported.

[0169] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, such as Google Chrome TM OS, including various mobile operating systems (e.g., Microsoft Windows Windows Android TM , Palm Portable handheld devices may include cellular phones, smart phones (e.g., ), tablet computers (e.g. ), personal digital assistants (PDAs), etc. Wearable devices may include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Microsoft Gesture Input Device Game consoles, Sony System, by Various game systems provided by and others), etc. The client device may be able to execute various different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications) and may use various communication protocols.

[0170] The network(s) 610 may be any type of network familiar to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), By way of example only, the network(s) 610 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., in accordance with the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, and / or any other wireless protocol) and / or any combination of these networks and / or other networks.

[0171] The server 612 may be composed of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, The server 612 may include a server, a mid-range server, a mainframe computer, a rack server, etc.), a server farm, a server cluster, or any other suitable arrangement and / or combination. The server 612 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that may be virtualized to maintain the server's virtual storage devices. In various examples, the server 612 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0172] The computing system in server 612 may run one or more operating systems, including any of those discussed above and any commercially available server operating system. Server 612 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transport Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines Corporation), etc.

[0173] In some implementations, server 612 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. By way of example, data feeds and / or event updates may include, but are not limited to: feed, The server 612 may also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client computing devices 602, 604, 606, and 608.

[0174] The distributed system 600 may also include one or more data repositories 614, 616. In some examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 614, 616 may be used to store information (such as information related to chatbot performance or generated models) for use by the chatbot used by the server 612 when performing various functions according to various embodiments. The data repositories 614, 616 may reside in various locations. For example, the data repository used by the server 612 may be local to the server 612 or may be remote from the server 612 and communicate with the server 612 via a network-based or dedicated connection. The data repositories 614, 616 may be of different types. In some examples, the data repository used by the server 612 may be a database, such as a relational database, such as an Oracle database. And other databases provided by other suppliers. One or more of these databases can be adapted to respond to SQL format commands to implement storage, update and retrieval of data to and from the database.

[0175] In some examples, one or more of data repositories 614, 616 may also be used by applications to store application data. The data repositories used by applications may be of different types, such as, for example, key-value storage repositories, object storage repositories, or general storage repositories backed by a file system.

[0176] In some examples, the functionality described in this disclosure can be provided as a service via a cloud environment. Figure 7 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services according to certain examples. Figure 7 In the depicted example, cloud infrastructure system 702 may provide one or more cloud services that may be requested by a user using one or more client computing devices 704, 706, and 708. Cloud infrastructure system 702 may include one or more computers and / or servers, which may include those described above with respect to server 612. The computers in cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0177] The network(s) 710 can facilitate data communication and exchange between the clients 704, 706, and 708 and the cloud infrastructure system 702. The network(s) 710 can include one or more networks. The networks can be of the same or different types. The network(s) 710 can support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0178] Figure 7 The depicted example is only one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in some other examples, the cloud infrastructure system 702 may have more Figure 7 Those components depicted may have more or fewer components, may combine two or more components, or may have a different configuration or arrangement of components. Figure 7 Three client computing devices are depicted, but in alternative examples any number of client computing devices may be supported.

[0179] The term cloud service is generally used to refer to services that are made available to users on demand and via a communication network such as the Internet through a service provider's system (e.g., cloud infrastructure system 702). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can make use of the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle of Redwood Shores, California, offers cloud services. It provides several cloud services such as middleware services, database services, Java cloud services and other services.

[0180] In some examples, cloud infrastructure system 702 may provide one or more cloud services using different models, such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), and other models, including hybrid service models. Cloud infrastructure system 702 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0181] The SaaS model enables applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase the hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system 702. Oracle Examples of SaaS services provided include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and others.

[0182] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services to provide elastic computing and storage capabilities. Provides various IaaS services.

[0183] The PaaS model is typically used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without the customer having to purchase, build, or maintain such resources. Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

[0184] Cloud services are typically provided on an on-demand self-service basis, subscription-based, elastically scalable, reliable, highly available, and secure. For example, a customer may order one or more services provided by the cloud infrastructure system 702 via a subscription order. The cloud infrastructure system 702 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use speech to request that the cloud infrastructure system take a certain action (e.g., intent) as described above and / or provide services to a chatbot system as described herein. The cloud infrastructure system 702 may be configured to provide one or even multiple cloud services.

[0185] The cloud infrastructure system 702 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 702 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In some other examples, under a private cloud model, the cloud infrastructure system 702 can be operated within an organization (e.g., within an enterprise organization) and provide services to customers within the organization. For example, customers can be various departments of an enterprise such as a human resources department, a payroll department, or even individuals within an enterprise. In some other examples, under a community cloud model, the cloud infrastructure system 702 and the services provided can be shared by several organizations in the relevant community. Various other models such as a mix of the models mentioned above can also be used.

[0186] Client computing devices 704, 706, and 708 may be of different types (eg, Figure 6 702) and may be capable of operating one or more client applications. A user may use a client device to interact with the cloud infrastructure system 702, such as requesting services provided by the cloud infrastructure system 702. For example, a user may use a client device to request information or actions from a chatbot as described in the present disclosure.

[0187] In some examples, the processing performed by the cloud infrastructure system 702 for providing services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating data sets to train and deploy one or more models. The analysis may be performed by one or more processors, thereby potentially processing data in parallel, performing simulations using data, and the like. For example, big data analysis may be performed by the cloud infrastructure system 702 for generating and training one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0188] like Figure 7 , cloud infrastructure system 702 may include infrastructure resources 730 used to facilitate provisioning of various cloud services provided by cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that may be used to service storage requested from an application may be part of cloud infrastructure system 702. In other examples, the storage virtual machine may be part of a different system.

[0189] In some examples, to facilitate efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 702 for different customers, the resources can be bundled into resource groups or resource modules (also referred to as "pods"). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods can be pre-provisioned for different types of cloud services. For example, a first group of pods can be provisioned for database services, a second group of pods can be provisioned for Java services (the second group of pods can include a different combination of resources than the pods in the first group of pods), etc. For some services, the resources allocated for provisioning the services can be shared between services.

[0190] The cloud infrastructure system 702 itself may internally use services 732 that are shared by different components of the cloud infrastructure system 702 and that facilitate the provision of services by the cloud infrastructure system 702. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for implementing cloud support, email services, notification services, file transfer services, and the like.

[0191] The cloud infrastructure system 702 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. Figure 7 As depicted, the subsystem may include a user interface subsystem 712 that enables users or customers of the cloud infrastructure system 702 to interact with the cloud infrastructure system 702. The user interface subsystem 712 may include a variety of different interfaces, such as a web interface 714, an online store interface 716 (where cloud services provided by the cloud infrastructure system 702 are advertised and can be purchased by consumers), and other interfaces 718. For example, a customer may use a client device to request (service request 734) one or more services provided by the cloud infrastructure system 702 using one or more of the interfaces 714, 716, and 718. For example, a customer may visit an online store, browse cloud services provided by the cloud infrastructure system 702, and place a subscription order for one or more services provided by the cloud infrastructure system 702 that the customer wishes to subscribe to. The service request may include information identifying the customer and the one or more services that the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 702. As part of the order, the customer may provide information identifying a chatbot system for which the services are to be provided and optionally provide one or more credentials for the chatbot system.

[0192] In some examples (such as Figure 7In the depicted example), cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this processing, OMS 720 may be configured to: create an account for a customer (if one has not already been created); receive billing and / or billing information from the customer to be used to bill the customer for providing the requested service to the customer; verify the customer information; after verification, book the order for the customer; and orchestrate various workflows to prepare the order for fulfillment.

[0193] Once properly authenticated, the OMS 720 may then call an order provisioning subsystem (OPS) 724 that is configured to provision resources for the order, including processing resources, memory resources, and networking resources. Provisioning may include allocating resources to an order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for an order and the type of resources provisioned may depend on the type of cloud services that the customer has ordered. For example, according to a workflow, the OPS 724 may be configured to determine a specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated for an order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users to be supported by the service, the duration of the service being requested, and the like. The allocated groups may then be customized for the specific requesting customer for providing the requested service.

[0194] In some examples, the setup phase processing as described above can be performed as part of the provisioning process by cloud infrastructure system 702. Cloud infrastructure system 702 can generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by cloud infrastructure system 702 itself or from storage virtual machines provided by systems other than cloud infrastructure system 702.

[0195] The cloud infrastructure system 702 may send a response or notification 744 to the requesting customer to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) may be sent to the customer that enables the customer to begin using and taking advantage of the benefits of the requested service. In some examples, for the customer requesting the service, the response may include a chatbot system ID generated by the cloud infrastructure system 702 and information identifying the chatbot system selected by the cloud infrastructure system 702 as the chatbot system corresponding to the chatbot system ID.

[0196] The cloud infrastructure system 702 can provide services to multiple customers. For each customer, the cloud infrastructure system 702 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the order, and providing the requested services to the customer. The cloud infrastructure system 702 can also collect usage statistics about the customer's use of the subscribed services. For example, statistics can be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system up time and system down time. The usage information can be used to bill the customer. Billing can be done, for example, on a monthly basis.

[0197] The cloud infrastructure system 702 can provide services to multiple customers in parallel. The cloud infrastructure system 702 can store information (possibly including proprietary information) of these customers. In some examples, the cloud infrastructure system 702 includes an identity management subsystem (IMS) 728 configured to manage customer information and provide separation of managed information so that information related to one customer cannot be accessed by another customer. The IMS 728 can be configured to provide various security-related services such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles and related functions, etc.

[0198] Figure 8 An example of a computer system 800 is illustrated. In some examples, the computer system 800 can be used to implement any digital assistant or chatbot system in a distributed environment and the various servers and computer systems described above. Figure 8 As shown, computer system 800 includes various subsystems, including a processing subsystem 804 that communicates with multiple other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. The storage subsystem 818 may include non-transitory computer-readable storage media, including storage media 822 and system memory 810.

[0199] The bus subsystem 802 provides a mechanism for allowing the various components and subsystems of the computer system 800 to communicate with each other as expected. Although the bus subsystem 802 is schematically shown as a single bus, alternative examples of the bus subsystem can utilize multiple buses. The bus subsystem 802 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such an architecture can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus (which can be implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0200] The processing subsystem 804 controls the operation of the computer system 800 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core processor or a multi-core processor. The processing resources of the computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 804 may include one or more dedicated coprocessors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 804 may be implemented using custom circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0201] In some examples, the processing unit in the processing subsystem 804 can execute instructions stored in the system memory 810 or on the computer-readable storage medium 822. In various examples, the processing unit can execute various programs or code instructions and can maintain multiple programs or processes that are executed simultaneously. At any given time, some or all of the program code to be executed can reside in the system memory 810 and / or on the computer-readable storage medium 822 (potentially including residing on one or more storage devices). Through suitable programming, the processing subsystem 804 can provide the various functions described above. In the instance of the computer system 800 executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0202] In some examples, a processing acceleration unit 806 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 804 , thereby speeding up the overall processing performed by the computer system 800 .

[0203] The I / O subsystem 808 may include devices and mechanisms for inputting information to the computer system 800 and / or for outputting information from or via the computer system 800. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information to the computer system 800. User interface input devices may include, for example, keyboards, pointing devices such as a mouse or trackball, touch pads or touch screens incorporated into a display, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Windows, which enable a user to control and interact with the input devices. Motion Sensor, Microsoft 360 game controllers, devices that provide an interface for receiving input using gestures and spoken commands. The user interface input device may also include an eye gesture recognition device, such as detecting eye activity from the user (e.g., "blinking" when taking a picture and / or making a menu selection) and translating eye gestures into input devices (e.g., Google ) Blink detector. In addition, the user interface input device may include an interface that enables the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0204] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing rods, game controllers and graphics boards, and audio / visual devices (such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser rangefinders, eye gaze tracking devices). In addition, the user interface input device may include, for example, medical imaging input devices, such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. The user interface input device may also include, for example, audio input devices, such as MIDI keyboards, digital musical instruments, etc.

[0205] In general, the term output device is used to include all possible types of devices and mechanisms for outputting information from the computer system 800 to a user or other computers. The user interface output device may include a display subsystem, indicator lights, or non-visual displays such as audio output devices. The display subsystem may be a cathode ray tube (CRT), a flat panel device (such as a flat panel device using a liquid crystal display (LCD) or a plasma display), a projection device, a touch screen, etc. For example, the user interface output device may include, but is not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0206] The storage subsystem 818 provides a repository or data storage for storing information and data used by the computer system 800. The storage subsystem 818 provides a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide some of the example functionality. The storage subsystem 818 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 804, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 804. The storage subsystem 818 can also provide authentication in accordance with the teachings of the present disclosure.

[0207] The storage subsystem 818 may include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. Figure 8 As shown, the storage subsystem 818 includes a system memory 810 and a computer-readable storage medium 822. The system memory 810 may include multiple memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines that help transfer information between elements within the computer system 800 during startup can typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by the processing subsystem 804. In some embodiments, the system memory 810 may include a variety of different types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0208] By way of example and not limitation, Figure 8As depicted, system memory 810 may load executing applications 812 (which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc.), program data 814, and an operating system 816. By way of example, operating system 816 may include various versions of Microsoft Apple and / or Linux operating system, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or iOS, Telephone, OS, OS, OS operating system and other mobile operating systems, as well as other operating systems.

[0209] Computer-readable storage media 822 may store programming and data structures that provide some example functionality. Computer-readable media 822 may provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 800. Software (programs, code modules, instructions) that provide the functionality described above when executed by processing subsystem 804 may be stored in storage subsystem 818. By way of example, computer-readable storage media 822 may include, for example, a hard drive, a magnetic disk drive, an optical disk drive (e.g., a CD ROM, a DVD, Computer readable storage media 822 may include, but are not limited to, The computer readable storage medium 822 may include a solid-state drive (SSD) such as a flash memory-based solid-state drive (SSD), an enterprise-class flash memory drive, a solid-state ROM, or other non-volatile memory-based SSD, a volatile memory-based SSD such as a solid-state RAM, a dynamic RAM, a static RAM, or other volatile memory-based SSD, a DRAM-based SSD, a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD using a combination of a DRAM and a flash memory-based SSD.

[0210] In some examples, the storage subsystem 818 may also include a computer-readable storage medium reader 820 that may be further connected to a computer-readable storage medium 822. The reader 820 may receive data from a storage device such as a disk, a flash memory drive, etc. and is configured to read data from the storage device.

[0211] In some examples, the computer system 800 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, the computer system 800 can provide support for executing one or more virtual machines. In some examples, the computer system 800 can execute programs such as management programs that promote the configuration and management of virtual machines. Each virtual machine can be allocated with memory, computing (e.g., processor, core), I / O and networking resources. Each virtual machine is usually independent of other virtual machines. The virtual machine usually runs its own operating system, which can be the same or different from the operating system executed by other virtual machines executed by the computer system 800. Therefore, multiple operating systems can potentially be run simultaneously by the computer system 800.

[0212] The communication subsystem 824 provides an interface to other computer systems and networks. The communication subsystem 824 is used as an interface for receiving data from other systems and transmitting data from the computer system 800 to other systems. For example, the communication subsystem 824 can enable the computer system 800 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and sending information to the client devices. For example, when the computer system 800 is used to implement Figure 1 When the robot system 120 is depicted, the communication subsystem can be used to communicate with the chat robot system selected for the application.

[0213] The communication subsystem 824 can support both wired communication protocols and / or wireless communication protocols. In some examples, the communication subsystem 824 may include a radio frequency (RF) transceiver component for accessing a wireless voice and / or data network (e.g., using cellular phone technology, advanced data network technologies such as 3G, 4G or EDGE (Enhanced Data Rates for Global Evolution), Wi-Fi (IEEE 802.XX Home Standard, or other mobile communication technology, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, in addition to or instead of a wireless interface, the communication subsystem 824 may provide wired network connectivity (e.g., Ethernet).

[0214] The communication subsystem 824 can receive and transmit various forms of data. In some examples, the communication subsystem 824 can receive input communications in the form of structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., among other forms. For example, the communication subsystem 824 can be configured to receive (or send) data feeds 826 in real time from users of social media networks and / or other communication services, such as feed, Updates, web feeds (such as Rich Site Summary (RSS) feeds), and / or real-time updates from one or more third-party information sources.

[0215] In some examples, the communication subsystem 824 may be configured to receive data in the form of continuous data streams, which may include event streams 828 of real-time events (which may be continuous or unbounded in nature without explicit termination) and / or event updates 830. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0216] The communication subsystem 824 may also be configured to transmit data from the computer system 800 to other computer systems or networks. The data may be transmitted in various forms such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc. to one or more databases that may be in computer communication with one or more stream data source computers coupled to the computer system 800.

[0217] Computer system 800 may be one of various types, including a handheld portable device (e.g., Cellular phone, computing tablets, PDAs), wearable devices (e.g., Google head mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, the Figure 8 The description of the depicted computer system 800 is intended to be a specific example only. Figure 8 Many other configurations of more or fewer components of the depicted systems are possible.Based on the disclosure and the teachings provided herein, it should be appreciated that there are other ways and / or methods to implement the various examples.

[0218] Although specific examples have been described, various modifications, changes, alternative constructions and equivalents are possible. Examples are not limited to operations in certain specific data processing environments, but are freely operated in multiple data processing environments. In addition, although certain examples have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be restrictive. Although some flow charts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. The process can have additional steps not included in the figure. The various features and aspects of the examples described above can be used individually or jointly.

[0219] Further, although certain examples have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are possible. Certain examples may be implemented in hardware only or software only or using a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors.

[0220] Where an apparatus, system, component or module is described as being configured to perform certain operations or functions, such configuration may be accomplished, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0221] Specific details are given in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures and techniques have been shown without unnecessary details to avoid obscuring the examples. This description only provides illustrative examples and is not intended to limit the scope, applicability or configuration of other examples. Instead, the previous description of the examples will provide enabling descriptions for implementing various examples for those skilled in the art. Various changes can be made to the functions and arrangements of the elements.

[0222] Therefore, the present specification and drawings should be viewed in an illustrative rather than a restrictive sense. However, it will be apparent that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, although specific examples have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0223] In the foregoing description, various aspects of the present disclosure are described with reference to specific examples of the present disclosure, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used individually or in combination. Further, the examples may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0224] In the foregoing description, for the purpose of illustration, the method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order. It should also be understood that the method described above can be performed by hardware components or can be embodied in a sequence of machine executable instructions, which can be used to make a machine (such as a general or special processor or logic circuit programmed with the instructions) perform the method. These machine executable instructions can be stored on one or more machine-readable media, such as CD-ROM or other types of optical disks, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic cards or optical cards, flash memory or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the method can be performed by a combination of hardware and software.

[0225] Where a component is described as being configured to perform certain operations, such configuration may be accomplished, for example, by designing an electronic circuit or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or any combination thereof.

[0226] While illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be interpreted to include such variations, except to the extent limited by the prior art.

Claims

1. A computer-implemented method comprising: accessing training data comprising a plurality of utterances, wherein each utterance of the plurality of utterances comprises at least one named entity corresponding to a named entity category of a plurality of named entity categories; calculating an expansion rate for each of the plurality of named entity categories based on a distribution of each of the plurality of named entity categories in the training data and a function selected from a plurality of functions in a hyperparameter tuning process of a named entity recognition (NER) model; generating a plurality of additional utterances based on the expansion rate calculated for each of the plurality of named entity categories; augmenting the training data by combining the plurality of additional utterances with the plurality of utterances to form augmented training data; and The NER model is trained using the augmented training data.

2. The computer-implemented method of claim 1, wherein: The hyperparameter tuning process includes: dividing the training data into training data sets, each of the training data sets comprising a corresponding utterance set from the plurality of utterances; For each training dataset in the training datasets: Select candidate hyperparameters from the set of candidate hyperparameters, Based on the candidate hyperparameter, generating an augmented utterance set for a corresponding training data set from the corresponding utterance set based on a function associated with the candidate hyperparameter among the plurality of functions, training the NER model with the expanded utterance set, and calculating a performance metric for the NER model trained with the expanded utterance set; and A set of hyperparameters for the NER model is determined based on the performance metric calculated for the NER model trained with the corresponding augmented utterance set.

3. The computer-implemented method of claim 1, wherein: A number of additional utterances generated for a first named entity category in the plurality of named entity categories is different than a number of additional utterances generated for a second named entity category in the plurality of named entity categories.

4. The computer-implemented method of claim 1, wherein: Determining the expansion rate of each of the multiple named entity categories includes calculating the number of occurrences of each of the multiple named entity categories in the training data and normalizing the number of occurrences of each of the multiple named entity categories in the training data.

5. The computer-implemented method of claim 4, wherein: Determining the expansion rate of each of the multiple named entity categories includes calculating a normalized distribution value for each of the multiple named entity categories, wherein, for each of the multiple named entity categories, the normalized distribution value is calculated by dividing the number of occurrences calculated for the corresponding named entity category by the sum of the number of occurrences of all of the multiple named entity categories in the training data.

6. The computer-implemented method of claim 1, wherein: The multiple functions include at least one of a first function and a second function, wherein the first function calculates the expansion rate of the given named entity category based on an exponential function that receives the normalized distribution value and β value of the given named entity category as input, and the second function calculates the expansion rate of the given named entity category based on a logarithmic function with base 2 that receives the number of occurrences of the given named entity category in the training data as input.

7. The computer-implemented method of claim 1 , further comprising: The trained NER model is provided to a system, wherein providing the trained NER model includes detecting and classifying named entities in an utterance received by the system from a user.

8. A system comprising: one or more processors; as well as One or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations including: accessing training data comprising a plurality of utterances, wherein each utterance of the plurality of utterances comprises at least one named entity corresponding to a named entity category of a plurality of named entity categories; calculating an expansion rate for each of the plurality of named entity categories based on a distribution of each of the plurality of named entity categories in the training data and a function selected from a plurality of functions in a hyperparameter tuning process of a named entity recognition (NER) model; generating a plurality of additional utterances based on the expansion rate calculated for each of the plurality of named entity categories; augmenting the training data by combining the plurality of additional utterances with the plurality of utterances to form augmented training data; and The NER model is trained using the augmented training data.

9. The system of claim 8, wherein: The hyperparameter tuning process includes: dividing the training data into training data sets, each of the training data sets comprising a corresponding utterance set from the plurality of utterances; For each training dataset in the training datasets: Select candidate hyperparameters from the set of candidate hyperparameters, Based on the candidate hyperparameter, generating an augmented utterance set for a corresponding training data set from the corresponding utterance set based on a function associated with the candidate hyperparameter among the plurality of functions, training the NER model with the expanded utterance set, and calculating a performance metric for the NER model trained with the expanded utterance set; and A set of hyperparameters for the NER model is determined based on the performance metric calculated for the NER model trained with the corresponding augmented utterance set.

10. The system of claim 8, wherein: A number of additional utterances generated for a first named entity category in the plurality of named entity categories is different than a number of additional utterances generated for a second named entity category in the plurality of named entity categories.

11. The system of claim 8, wherein: Determining the expansion rate of each of the multiple named entity categories includes calculating the number of occurrences of each of the multiple named entity categories in the training data and normalizing the number of occurrences of each of the multiple named entity categories in the training data.

12. The system of claim 11, wherein: Determining the expansion rate of each of the multiple named entity categories includes calculating a normalized distribution value for each of the multiple named entity categories, wherein, for each of the multiple named entity categories, the normalized distribution value is calculated by dividing the number of occurrences calculated for the corresponding named entity category by the sum of the number of occurrences of all of the multiple named entity categories in the training data.

13. The system of claim 8, wherein: The multiple functions include at least one of a first function and a second function, wherein the first function calculates the expansion rate of the given named entity category based on an exponential function that receives the normalized distribution value and β value of the given named entity category as input, and the second function calculates the expansion rate of the given named entity category based on a logarithmic function with base 2 that receives the number of occurrences of the given named entity category in the training data as input.

14. The system of claim 8, the operations further comprising: The trained NER model is provided to a system, wherein providing the trained NER model includes detecting and classifying named entities in an utterance received by the system from a user.

15. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a system to perform operations including: Access training data consisting of multiple utterances, where Each utterance of the plurality of utterances includes at least one named entity corresponding to a named entity category of the plurality of named entity categories; calculating an expansion rate for each of the plurality of named entity categories based on a distribution of each of the plurality of named entity categories in the training data and a function selected from a plurality of functions in a hyperparameter tuning process of a named entity recognition (NER) model; generating a plurality of additional utterances based on the expansion rate calculated for each of the plurality of named entity categories; augmenting the training data by combining the plurality of additional utterances with the plurality of utterances to form augmented training data; as well as The NER model is trained using the augmented training data.

16. The one or more non-transitory computer readable media of claim 15, wherein: The hyperparameter tuning process includes: dividing the training data into training data sets, each of the training data sets comprising a corresponding utterance set from the plurality of utterances; For each training dataset in the training datasets: Select candidate hyperparameters from the set of candidate hyperparameters, Based on the candidate hyperparameter, generating an augmented utterance set for a corresponding training data set from the corresponding utterance set based on a function associated with the candidate hyperparameter among the plurality of functions, training the NER model with the expanded utterance set, and calculating a performance metric for the NER model trained with the expanded utterance set; and A set of hyperparameters for the NER model is determined based on the performance metric calculated for the NER model trained with the corresponding augmented utterance set.

17. The one or more non-transitory computer readable media of claim 15, wherein: A number of additional utterances generated for a first named entity category in the plurality of named entity categories is different than a number of additional utterances generated for a second named entity category in the plurality of named entity categories.

18. The one or more non-transitory computer readable media of claim 15, wherein: Determining the expansion rate of each of the multiple named entity categories includes calculating the number of occurrences of each of the multiple named entity categories in the training data and normalizing the number of occurrences of each of the multiple named entity categories in the training data.

19. The one or more non-transitory computer readable media of claim 15, wherein: The multiple functions include at least one of a first function and a second function, wherein the first function calculates the expansion rate of the given named entity category based on an exponential function that receives the normalized distribution value and β value of the given named entity category as input, and the second function calculates the expansion rate of the given named entity category based on a logarithmic function with base 2 that receives the number of occurrences of the given named entity category in the training data as input.

20. The one or more non-transitory computer readable media of claim 15, the operations further comprising: The trained NER model is provided to a system, wherein providing the trained NER model includes detecting and classifying named entities in an utterance received by the system from a user.

Images