Entity-level data augmentation for robust named entity recognition in chatbots

Abstract

Data augmentation techniques for training chatbot systems in natural language processing. In one particular aspect, a method is provided that includes generating a list of values to cover for an entity, selecting utterances from a dataset having a context of the entity, converting the utterances into templates, where each of the templates includes slots mapped to the list of values for the entity, selecting a template from the templates, selecting a value from the list of values based on a mapping between a slot within the selected template and the list of values for the entity, and creating an artificial utterance based on the selected template and the selected value, where creating the artificial utterance includes inserting the selected value into a slot of the selected template that is mapped to the list of values for the entity.

Description

[0001] Priority requirements This application is a non-provisional application filed on June 12, 2020, under U.S. Provisional Application No. 63 / 038,337, and claims the benefit of the aforementioned U.S. Provisional Application, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0002] This disclosure generally relates to chatbot systems, and more specifically to data augmentation techniques for training chatbot systems in natural language processing. Background Technology

[0003] To achieve immediate responses, many users around the world utilize instant messaging or chat platforms. Organizations frequently use these platforms to engage in real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots or bots have begun to be developed to simulate conversations with end users, especially via the Internet. End users can interact with bots through messaging applications they have already installed and are using. Intelligent bots (typically powered by artificial intelligence (AI)) can communicate more intelligently and context-sensitively in real-time conversations, thus allowing for more natural conversations between bots and end users to improve the conversational experience. Instead of end users learning a fixed set of keywords or commands to respond to, intelligent bots can understand the end user's intent based on natural language utterances and respond accordingly.

[0004] However, chatbots are difficult to build because these automated solutions require domain-specific knowledge and the application of certain technologies that may only be within the capabilities of specialized developers. As part of building such a chatbot, developers can begin by understanding the needs of the enterprise and end-users. Developers can then analyze and make decisions related to, for example: selecting the dataset to be used for analysis; preparing the input dataset for analysis (e.g., cleaning, extracting, formatting, and / or transforming data before analysis, performing data feature engineering, etc.); identifying appropriate machine learning (ML) techniques or (multiple) ML models for performing the analysis; and improving the techniques or models to improve results / effects based on feedback. The task of identifying appropriate models may include developing multiple models (possibly in parallel), iteratively testing and experimenting with these models, and then identifying specific models (or multiple models) for use. Furthermore, supervised learning-based solutions typically involve a training phase, followed by an application (i.e., inference) phase, and an iterative cycle between the training and application phases. Developers can be responsible for carefully implementing and monitoring these phases to achieve the optimal solution. For example, to train (multiple) ML techniques or models, precise training data is needed so that the algorithms can understand and learn certain patterns or features (e.g., for chatbots—intent extraction and careful syntactic analysis are required, not just raw language processing). These ML techniques or models will use these patterns or features to predict desired outcomes (e.g., inferring intent from utterances). To ensure that (multiple) ML techniques or models correctly learn these patterns and features, developers can be responsible for selecting, enriching, and optimizing the training dataset for (multiple) ML techniques or models. Summary of the Invention

[0005] This disclosure relates to data augmentation techniques for training chatbot systems in natural language processing.

[0006] In various embodiments, a method is provided, the method comprising: generating a list of values ​​to be covered for an entity by a data processing system; selecting utterances by the data processing system from a dataset having context of the entity; converting the utterances into templates by the data processing system, wherein each template includes slots for the entity mapped to the list of values; selecting a template by the data processing system from the templates; selecting values ​​from the list of values ​​by the data processing system based on the mapping between slots within the selected template and the list of values ​​for the entity; and generating artificial utterances by the data processing system using the selected templates and the selected values, wherein generating the artificial utterances includes inserting the selected values ​​into the slots of the selected templates.

[0007] In some embodiments, the method further includes: training a model using a training dataset by the data processing system before selecting utterances from the dataset, wherein the training dataset is the dataset or a subset thereof; testing the model using a test dataset or a validation dataset by the data processing system; evaluating the performance of the model by the data processing system based on the tests on the model; and identifying the entity by the data processing system based on the evaluation.

[0008] In some embodiments, the method further includes: augmenting the dataset or the training dataset with the artificial utterance by the data processing system to generate an augmented training dataset; training the model using the augmented training dataset by the data processing system; retesting the model using the test dataset or the validation dataset or another test dataset or validation dataset by the data processing system; evaluating the performance of the model based on the retesting of the model by the data processing system; determining that the performance of the model meets a predefined criterion by the data processing system; and deploying the model as part of a chatbot system in response to the model meeting the predefined criterion.

[0009] In some embodiments, the method further includes: the data processing system determining that the slots within the selected template include a first element mapped to the value list and a second element mapped to a function; the data processing system processing the function based on the mapping between the slots and the function within the selected template to generate a function value; and the data processing system combining the selected value and the function value to obtain a slot value, wherein generating the artificial discourse includes inserting the slot value into the slots of the selected template.

[0010] In some embodiments, the method further includes: determining, by the data processing system, that the selected template is mapped to a nested slot mapping set including the slot and another slot; determining, by the data processing system, that the other slot in the nested slot mapping set is mapped to another value list; and selecting, by the data processing system, another value from the other value list based on the mapping between the other slot within the selected template and the other value list, wherein generating the artificial discourse includes inserting the selected value into the slot of the selected template and inserting the selected other value into the other slot of the selected template.

[0011] In some embodiments, the method further includes augmenting the dataset or training dataset with the artificial utterance by the data processing system to generate an augmented training dataset for training a model to identify and classify multiple entities including the entity.

[0012] In some embodiments, the template is selected based on a heuristic approach that includes the following criteria: (i) choosing between templates with multiple entity classifications and templates with a single entity classification; (ii) weighing template variety against template quality; (iii) categorizing templates according to context; (iv) whether the model requires a minimum number of occurrences of each template / a unique context that the user wants to teach the model; or (v) any combination of the above.

[0013] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods disclosed herein.

[0014] In various embodiments, a computer program product is provided, which is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0015] The techniques described above and below can be implemented in various ways and in various contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts are only a few of many implementations and contexts. Attached Figure Description

[0016] Figure 1 This is a simplified block diagram of a distributed environment incorporating exemplary embodiments.

[0017] Figure 2 This is a simplified block diagram of a computing system for implementing a master robot according to certain embodiments.

[0018] Figure 3 This is a simplified block diagram of a computing system for a skill-implementing robot according to certain embodiments.

[0019] Figure 4 This is a simplified block diagram of a chatbot training and deployment system according to various embodiments.

[0020] Figure 5A The illustration shows a process flow for training a predictive model with augmented training data, according to various embodiments.

[0021] Figure 5B The diagram illustrates a process flow for generating augmented training data according to various embodiments.

[0022] Figure 5CThe illustrations depict the process flow for generating artificial discourse from a template, according to various embodiments.

[0023] Figure 6 A simplified diagram of a distributed system for implementing various embodiments is depicted.

[0024] Figure 7 It is a simplified block diagram of one or more components of a system environment according to various embodiments, through which services provided by one or more components of the embodiment system can be provided as cloud services.

[0025] Figure 8 An example computer system that can be used to implement various embodiments is illustrated. Detailed Implementation

[0026] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent, however, that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other embodiments or designs.

[0027] introduction A digital assistant is an AI-driven interface that helps users complete various tasks in natural language conversations. For each digital assistant, a client can assemble one or more skills. Skills (also described in this document as chatbots, bots, or skill bots) are individual bots focused on specific types of tasks such as tracking inventory, submitting timecards, and creating expense reports. When an end user interacts with a digital assistant, the assistant evaluates the end user's input and routes the conversation to and from the appropriate chatbot. This can be done through platforms such as Facebook. ® Messenger (instant messaging), Skype Mobile ® Various channels, such as Messenger or Short Message Service (SMS), make digital assistants available to end users. Channels enable chat to be sent back and forth between end users and digital assistants and their chatbots via various messaging platforms. Channels can also support user agent upgrades, event-initiated sessions, and testing.

[0028] Intents allow chatbots to understand what users want them to do. Intents comprise a sequence of typical user requests and statements, also known as utterances (e.g., getting account balance, making a purchase, etc.). As used herein, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. Intents can be created by providing a name describing a user action (e.g., ordering pizza) and compiling a set of real-life user statements or utterances typically associated with triggering the action. Because the chatbot's cognition stems from these intents, each intent can be created and varied from a robust dataset (one to two dozen utterances), allowing the chatbot to interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what a user wants when it receives messages such as "Forget this order!" or "Cancel delivery!" (meaning the same thing expressed differently). Intents and the utterances that belong to intents together constitute the chatbot's training corpus. By training the model with the corpus, the client essentially turns the model into a reference tool for parsing end-user input into individual intents. Customers can improve the cognitive acuity of chatbots through multiple rounds of intent testing and intent training.

[0029] However, building a chatbot that can determine the end-user's intent based on user utterances is a challenging task, partly due to the subtlety and ambiguity of natural language, as well as the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (the number of intents). Thus, improving chatbot performance and the user experience may require training, monitoring, debugging, and retraining. In traditional systems, training systems are provided for training and retraining machine learning models for digital assistants or chatbots in Spoken Language Understanding (SLU) and Natural Language Processing (NLP). Traditionally, models for chatbot systems are trained in NLP using utterances “manufactured” for any intent. For example, the utterance “Do you do price changes?” might be used to train a classifier model for the chatbot system to classify this type of utterance as the intent—“Do you offer a price match?”. Using manufactured utterances to train the model helps in initially training the chatbot system to provide a service, and then, once the chatbot system is deployed and begins to receive real utterances from users, it can be retrained.

[0030] As part of NLP processing for discourse, digital assistants are trained to understand the meaning of discourse, which involves identifying one or more intentions and one or more entities corresponding to the discourse. Entity extraction in digital assistants has two phases: named entity recognition via a named entity recognizer and entity parsing. The specific training problem addressed here involves named entity recognition. Entities can be understood as nouns and are often referred to as slots. Entities are typically like dates, times, cities, names, brands, etc. (Several common entities used as examples in this paper are domain-independent and are referred to as system entities: PERSON, NUMBER, CURRENCY, DATE_TIME; however, it should be understood that this disclosure is not limited to these system entities and applies to any entity type, such as composite entities, entity roles, entity lists, etc.). For example, for a travel robot, capturing the departure city, destination, mode of transport, price, date, and time is fundamental to the interface. However, capturing entities is a challenging task for digital assistants because users input data in various contexts, in various languages, and in random, unpredictable order. Table 1 below shows examples of inputs (discourses) and outputs (entities) for several entity extraction problems:

[0031] However, capturing these entities is important for digital assistants to act based on user intent.

[0032] Traditionally, entity recognition models are trained starting with pre-labeled data. In supervised machine learning environments, particularly on this issue, the core challenge is the lack of sufficient training data with pre-labeled information for the model to learn from. Further specific challenges in building robust named entity recognition models for digital assistants include: (i) a lack of coverage of a wider variety of CURRENCY symbols in the training data (conditions supporting different currencies); (ii) a lack of coverage of different CURRENCY monetary writing rules in the training data (conditions supporting different monetary writing rules); (iii) a lack of specific CURRENCY units and the number of combined utterances in the training data (conditions addressing utterances where each currency has a primary unit, secondary unit, and utterances with the correct primary unit, secondary unit, and monetary format); (iv) a lack of utterances with abbreviated TIME_DATE formats (conditions addressing abbreviations such as abbreviating September 5 to "Sep 5 (9.5)"); and (v) a lack of utterances with different writing formats for PERSON and NUMBER (conditions supporting all formats for names and numbers). Any machine learning model will only perform well if the pre-labeled training data it is trained on is good. Therefore, the quality of the training data plays a decisive role in the model's performance.

[0033] The system entities described in this paper are quite common, and therefore many publicly available datasets are available. However, a problem with publicly available pre-labeled data is that it lacks diversity, making it difficult to train models to detect entities with various variations (currency type, number format, data time format, etc.). One option for obtaining diverse labeled data is to manually write labeled additional utterances into the training data. Another option is to outsource the writing of labeled additional utterances to freelancers or companies dedicated to data labeling. Yet another option is to use crowdsourcing, which essentially leverages crowdsourced workers to scale up manual work. However, these methods may be difficult to implement for enterprise systems using many chatbot systems that are trained in multiple languages ​​for many different tasks and receive a wide variety of utterances (many of which are very short) for each task. In systems employing chatbots like this, there is a need to automatically obtain entity diversity in a synthetically agnostic manner in order to quickly and efficiently generate large training corpora in multiple languages ​​for many different chatbots.

[0034] Therefore, different approaches are needed to address these issues. The developed methods utilize data augmentation techniques to generate more utterances with all the necessary features to achieve entity diversity and train models to detect entities with various variations. In various embodiments, a method is provided comprising: generating a list of values ​​to be covered for entities by a data processing system; selecting utterances from a dataset having the context of the entities by the data processing system; converting the utterances into templates by the data processing system, wherein each template includes slots mapped to the list of values ​​for the entities; selecting templates by the data processing system from the templates; selecting values ​​from the list of values ​​by the data processing system based on the mapping between slots within the selected templates and the list of values ​​for the entities; and generating artificial utterances by the data processing system using the selected templates and the selected values, wherein generating the artificial utterances includes inserting the selected values ​​into the slots of the selected templates.

[0035] Robots and Analysis Systems A bot (also known as a skill, chatbot, conversational bot, or talkative bot) is a computer program capable of engaging in conversations with an end user. Bots typically respond to natural language messages (e.g., questions or comments) via messaging applications that use natural language messaging. Businesses can use one or more bot systems to communicate with end users through messaging applications. The messaging application (which may be referred to as a channel) can be a preferred messaging application that the end user has already installed and is familiar with. Therefore, end users do not need to download and install a new application to chat with the bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web applications, responsive mobile or web applications with chat capabilities, or voice-based input (such as devices or applications with interfaces that use Siri, Cortana, Google Voice, or other voice inputs for interaction).

[0036] In some examples, a robot system may be associated with a Uniform Resource Identifier (URI). A URI can identify a robot system using a string of characters. A URI can be used as a webhook for one or more messaging application systems. A URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The robot system may be designed to receive messages from a messaging application system (e.g., a Hypertext Transfer Protocol (HTTP) POST message). An HTTP POST message may involve a URI from a messaging application system. In some embodiments, a message may be different from an HTTP POST message. For example, the robot system may receive messages from a Short Message Service (SMS). While the discussion herein may refer to communication received by the robot system as a message, it should be understood that a message can be an HTTP POST message, an SMS message, or any other type of communication between two systems.

[0037] End users can interact with robot systems through conversational interactions (sometimes called conversational user interface (UI)), much like human interactions. In some cases, the interaction may include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking how the robot can help. In other cases, the interaction may be transactional with, for example, a banking robot, such as transferring money from one account to another; informational with, for example, an HR robot, such as checking holiday balances; or with, for example, a retail robot, such as discussing returning purchased goods or seeking technical support.

[0038] In some embodiments, a robot system can intelligently handle end-user interactions without interaction with the robot system's administrator or developer. For example, an end-user can send one or more messages to the robot system to achieve a desired goal. Messages may include some form of content, such as text, emojis, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., using appropriate parameters for a Representational State Transition (REST) ​​call to an enterprise service) and generate a natural language response. The robot system may also prompt the end-user for additional input parameters or request additional information. In some embodiments, the robot system may also initiate communication with the end-user rather than passively responding to the end-user's utterances. Various techniques are described herein for identifying explicit calls to a robot system and determining the input of the called robot system. In some embodiments, explicit call analysis is performed by the master robot based on the detected call name in the utterance. In response to the detection of the call name, the utterance can be refined for input to the skill robot associated with the call name.

[0039] A conversation with a chatbot can follow a specific conversation flow that includes multiple states. This flow can define what will happen next based on input. In some embodiments, the chatbot system can be implemented using a state machine that includes user-defined states (e.g., end-user intentions) and actions to be taken in or between states. The conversation can take different paths based on end-user input, which can affect the chatbot's decisions regarding the flow. For example, in each state, based on end-user input or utterances, the chatbot can determine the end-user's intention in order to determine the next appropriate action to take. As used herein and in the context of utterances, the term "intention" refers to the intention of the user providing the utterance. For example, a user might intend to involve the chatbot in a conversation about ordering pizza, so the user's intention can be expressed through the utterance "Order pizza." User intentions can relate to specific tasks that the user wants the chatbot to perform on their behalf. Therefore, utterances can be expressed as questions, commands, requests, etc., that reflect the user's intention. Intentions can include goals that the end-user wants to accomplish.

[0040] In the context of chatbot configuration, the term "intent" as used herein refers to configuration information used to map a user's utterances to a specific task / action or a specific type of task / action that the chatbot can perform. To distinguish the intent of the utterances (i.e., the user's intent) from the intent of the chatbot, the latter is sometimes referred to as "bot intent." A bot intent can include one or more utterances associated with the intent. For example, the intent to order pizza can have various permutations of utterances expressing the expectation of placing an order for pizza. These associated utterances can be used to train the chatbot's intent classifier so that the intent classifier can subsequently determine whether an input utterance from the user matches the pizza-ordering intent. A bot intent can be associated with one or more conversational flows used to initiate a conversation with the user in a certain state. For example, the first message for the pizza-ordering intent could be the question "What kind of pizza would you like?". In addition to associated utterances, bot intents can further include named entities related to the intent. For example, the pizza-ordering intent can include variables or parameters for performing the task of ordering pizza, such as topping 1, topping 2, pizza type, pizza size, number of pizzas, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0041] Figure 1This is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. Environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistant or chatbot systems. DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as... Figure 1 As shown, a user 104 representing a specific business can use DABP 102 to create and deploy a digital assistant 106 for that business's users. For example, DABP 102 can be used by a bank to create one or more digital assistants for its customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, a restaurant owner (e.g., a pizzeria) can use DABP 102 to create and deploy a digital assistant that enables the restaurant's customers to order food (e.g., order pizza).

[0042] For the purposes of this disclosure, a "digital assistant" is an entity that assists a user in performing various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. Digital assistants can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, and vehicles. Digital assistants are sometimes also referred to as chatbot systems. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system are used interchangeably.

[0043] A digital assistant (such as digital assistant 106 built using DABP 102) can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and receive a response 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and the response 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the task requested by the user and respond to the user with an appropriate response.

[0044] User input 110 is typically in the form of natural language and is referred to as speech. User speech 110 can be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user speech 110 can be in the form of audio input or speech, such as when a user speaks or utters something as input to the digital assistant 106. Speech is typically in the form of the language spoken by user 108. For example, speech can be English or some other language. When speech is in speech form, the speech input is converted into speech in text form in that particular language, and then the text speech is processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert speech or audio input into text speech, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion can be performed by the digital assistant 106 itself.

[0045] The discourse (which may be text discourse or speech discourse) may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the above types, etc. Digital Assistant 106 is configured to apply Natural Language Understanding (NLU) technology to the discourse to understand the meaning of user input. As part of the NLU processing for the discourse, Digital Assistant 106 is configured to perform processing for understanding the meaning of the discourse, which involves identifying one or more intentions and one or more entities corresponding to the discourse. After understanding the meaning of the discourse, Digital Assistant 106 may perform one or more actions or operations in response to the understood meaning or intention. For the purposes of this disclosure, it is assumed that these discourses are text discourses directly provided by the user 108 of Digital Assistant 106, or the result of converting input speech discourse into text form. However, this is not intended to limit or constrain in any way.

[0046] For example, user 108 can request to order a pizza by providing a statement such as "I want to order a pizza." Upon receiving such a statement, digital assistant 106 is configured to understand the meaning of the statement and take an appropriate action. The appropriate action may involve responding to the user's input, for example, by requesting information about the type of pizza the user wishes to order, its size, and any toppings. The responses provided by digital assistant 106 can also be in natural language and are typically in the same language as the input statement. As part of generating these responses, digital assistant 106 can perform natural language generation (NLG). To enable the user to order a pizza via a conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the necessary information for the pizza order and then allow the pizza to be ordered at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0047] At the conceptual level, the digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as Natural Language Understanding (NLU), determining the action to be performed in response to the utterance, performing the action where appropriate, generating a response to be output to the user in response to the user's utterance, outputting the response to the user, etc. NLU processing may include performing syntactic analysis on the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better comprehensible form (e.g., logical form) or structure of the utterance. Generating a response may include using NLG techniques.

[0048] The NLU processing performed by the digital assistant (such as digital assistant 106) may include various NLP-related processes such as sentence parsing (e.g., tokenization, classification by inflectional form, identification of part-of-speech tags, identification of named entities in sentences, generation of dependency trees to represent sentence structure, segmentation of sentences into clauses, analysis of individual clauses, resolution of pronouns, execution of chunks, etc.). In some embodiments, the NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of the input utterance sentence may be identified by processing the sentence using a parser, part-of-speech tagger, and / or named entity recognizer. In one implementation, for the English language, a parser, part-of-speech tagger, and named entity recognizer provided by the Stanford Natural Language Processing (NLP) Group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0049] While the various examples provided in this disclosure illustrate utterances in the English language, this is merely illustrative. In some embodiments, the digital assistant 106 is also capable of processing utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked from the NLU core server using service calls. This makes NLU processing flexible and scalable for each language, including allowing different processing sequences. Language packs may be provided for individual languages, wherein the language packs may register a list of subsystems that can be served from the NLU core server.

[0050] Digital assistants (such as...) can be enabled through various channels (such as, but not limited to, certain applications, social media platforms, various messaging services and applications, and other applications or channels). Figure 1The digital assistant 106 depicted is available or accessible to its user 108. A single digital assistant can be configured with several channels, allowing it to run on different services simultaneously and access them through different services.

[0051] Digital assistant or chatbot systems typically contain or are associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill-based bots) configured to interact with users and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc. For example, for... Figure 1 In the depicted embodiments, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the terms "skill" and "skills" are used synonymously with the terms "skill bot" and "skill bots," respectively.

[0052] Each skill associated with the digital assistant helps the user complete tasks through a conversation with the user, wherein the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may take the form of text or audio messages to the user and / or simple user interface elements (e.g., a selection list) presented to the user for selection.

[0053] There are various methods to associate or add skills or skill bots to a digital assistant. In some instances, a business can develop skill bots and then add them to a digital assistant using DABP 102. In other instances, skill bots can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet another instance, DABP 102 provides an online digital store (referred to as the "Skill Store") that offers multiple skills covering a wide variety of tasks. Skills offered through the Skill Store can also be exposed to various cloud services. To add a skill to a digital assistant created using DABP 102, a DABP 102 user can access the Skill Store via DABP 102, select the desired skill, and instruct the selected skill to be added to the digital assistant created using DABP 102. Skills from the Skill Store can be added to the digital assistant as is or with modifications (e.g., a DABP 102 user can select and copy a specific skill bot offered by the Skill Store, customize or modify the selected skill bot, and then add the modified skill bot to the digital assistant created using DABP 102).

[0054] Various architectures can be used to implement digital assistant or chatbot systems. For example, in some embodiments, a digital assistant created and deployed using DABP 102 can be implemented using a master bot / secondary (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more secondary bots that are skill-based bots. For example, in... Figure 1 In the depicted embodiments, digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are secondary robots of the master robot 114. In some embodiments, digital assistant 106 itself is considered to act as the master robot.

[0055] Digital assistants implemented using a master-secondary robot architecture enable users to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user engages with the digital assistant, the master robot receives the user input. The master robot then performs processing to determine the meaning of the user's input. It then determines whether the task requested in the user's utterance can be handled by the master robot itself; otherwise, it selects the appropriate skill robot to handle the user's request and routes the session to the selected skill robot. This allows users to interact with the digital assistant through a common, single interface while still having the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot can interface with skill robots with specific functionalities, such as a CRM robot for performing functions related to Customer Relationship Management (CRM), an ERP robot for performing functions related to Enterprise Resource Planning (ERP), an HCM robot for performing functions related to Human Capital Management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through the common master robot interface, while multiple skill robots are available in the background to handle user requests.

[0056] In some embodiments, within a master robot / sub-robot infrastructure, the master robot is configured to know a list of available skill robots. The master robot can access metadata identifying the various available skill robots, and for each skill robot, the skill robot's capabilities include tasks that can be performed by the skill robot. Upon receiving a user request in the form of a utterance, the master robot is configured to identify or predict from among the multiple available skill robots that a specific skill robot can best serve or handle the user request. The master robot then routes the utterance (or a portion of the utterance) to that specific skill robot for further handling. Thus, control flows from the master robot to the skill robot. The master robot may support multiple input and output channels.

[0057] Although Figure 1 The embodiments shown illustrate a digital assistant 106 including a main robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. A digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations using a combination of software and hardware.

[0058] The DABP 102 provides the infrastructure, services, and features that enable DABP 102 users to create digital assistants, including one or more skill bots associated with them. In some instances, skill bots can be created by cloning existing skill bots, for example, cloning skill bots provided by a skill store. As mentioned earlier, the DABP 102 provides a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP 102 users can clone skill bots from the skill store. The cloned skill bot can be modified or customized as needed. In some other instances, DABP 102 users create skill bots from scratch using the tools and services provided by the DABP 102. As mentioned earlier, the skill store or skill catalog provided by the DABP 102 can offer multiple skill bots for performing various tasks.

[0059] In some embodiments, creating or customizing a skill robot at a high level involves the following steps: (1) Configure settings for new skill robots (2) Configure one or more intentions for the skill robot (3) Configure one or more entities for one or more intents. (4) Training Skills Robot (5) Create dialogue flow for skill robots (6) Add custom parts to the skill robot as needed. (7) Test and deploy skill robots Each of the above steps is briefly described below.

[0060] (1) Configure settings for new skill robots—Various settings can be configured for skill robots. For example, skill robot designers can specify one or more invocation names for the skill robot being created. These invocation names can then be used by the digital assistant user to explicitly invoke the skill robot. For example, the user can enter the invocation name in their speech to explicitly invoke the corresponding skill robot.

[0061] (2) Configure one or more intents and associated example utterances for the skill robot—The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent categories or classifications that the skill robot is trained to infer in response to input utterances. Upon receiving a utterance, the trained skill robot infers the intent of the utterance, wherein the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes an appropriate action to respond to the utterance based on the intent inferred for that utterance. In some instances, the intents of the skill robot represent tasks that the skill robot can perform for a digital assistant user. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot could include “CheckBalance”, “TransferMoney”, “DepositCheck”, etc.

[0062] For each intent defined for a skill bot, the skill bot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are designed to represent the utterances a user would input to the skill bot in response to that intent. For example, for the CheckBalance intent, example utterances could include “What's my savings account balance?”, “How much is in my checking account?”, “How much money do I have in my account?”, and so on. Therefore, various permutations of typical user utterances can be designated as example utterances for the intent.

[0063] These intentions and their associated example utterances are used as training data for training a skill-based robot. Various training techniques can be used. As a result of this training, a predictive model is generated, configured to take utterances as input and output the intentions inferred by the predictive model in response to those utterances. In some instances, the input utterances are provided to an intention analysis engine, which is configured to use the trained model to predict or infer the intentions of the input utterances. The skill-based robot can then take one or more actions based on the inferred intentions.

[0064] (3) Configuring Entities for One or More Intents of a Skill Robot—In some instances, additional context may be needed for the skill robot to respond appropriately to user utterances. For example, there may be cases where user input utterances are parsed into the same intent within the skill robot. For instance, in the example above, the utterances “What's my savings account balance?” and “How much is in my checking account?” both resolve to the same CheckBalance intent, but these utterances are different requests for different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill robot, an entity called AccountType (which defines values ​​called “checking” and “saving”) enables the skill robot to perform syntactic analysis on user requests and respond appropriately. In the example above, although these utterances resolve to the same intent, the values ​​associated with the AccountType entity for the two utterances are different. This allows the skill robot to perform potentially different actions for the two utterances, even though they resolve to the same intent. One or more entities can be specified for certain intents configured for the skill robot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe intent and enable skill-based robots to fulfill user requests.

[0065] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with various bots. Examples of built-in entities include, but are not limited to, entities relating to time, date, address, number, email address, duration, cycle time, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for banking skills, the AccountType entity can be defined by the skill bot designer to enable various banking transactions by examining keywords entered by the user (such as current account, savings, credit card, etc.).

[0066] (4) Training the Skill Robot—The skill robot is configured to receive user input in utterance form, process the received input through syntactic analysis or otherwise, and identify or select an intent associated with the received user input. As indicated above, this requires training the skill robot. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example utterances associated with those intents (collectively, training data), such that the skill robot can parse user input utterances into one of its configured intents. In some embodiments, the skill robot uses a predictive model trained with the training data that allows the skill robot to discern what the user says (or, in some cases, is attempting to say). DABP 102 provides a variety of different training techniques that skill robot designers can use to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion of the training data (e.g., 80%) is used to train the skill robot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as the trained skill robot) can be used to process and respond to the user's utterances. In some cases, a user's utterance may be a question that requires only a single answer and no further conversation. To handle this, a Q&A (question and answer) intent can be defined for the skill bot. This allows the skill bot to output a response to the user's request without having to update the dialog definition. Q&A intents are created in a similar manner to regular intents. The dialogue flow used for Q&A intents may differ from that used for regular intents.

[0067] (5) Creating a Dialogue Flow for the Skilled Robot—The dialogue flow specified for the skilled robot describes how it reacts as it parses different intentions in response to received user input. The dialogue flow defines the actions or behaviors the skilled robot will take, such as how it responds to user utterances, prompts for user input, and returns data. The dialogue flow is like the process followed by the skilled robot. Figure 1 Skill bot designers specify the dialogue flow using languages ​​such as Markdown. In some embodiments, a version of YAML called OBotML can be used to specify the dialogue flow for the skill bot. The dialogue flow definition for the skill bot acts as a model of the session itself, a model that allows the skill bot designer to carefully orchestrate the interactions between the skill bot and the users it serves.

[0068] In some embodiments, the dialogue flow definition of a skill robot includes the following three parts: (a) Contextual part (b) Default Transformation Section (c) State section The context section—skill bot designers can define variables used in the session flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, and user variables that enable the skill bot to recognize and save user preferences.

[0069] The Default Transition section—transitions for the skill bot can be defined either in the dialog flow state section or in the Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when no applicable transition is defined within the state or when the conditions required to trigger a state transition cannot be met. The Default Transition section can be used to define routes that allow the skill bot to handle unexpected user actions appropriately.

[0070] The state component—the dialogue flow and its associated operations—is defined as a sequence of temporary states that manage the logic within the dialogue flow. Each state node within the dialogue flow definition is named a component, which provides the functionality required at that point in the dialogue. Therefore, states are built around components. States contain component-specific properties and define transitions to other states that are triggered after a component's execution.

[0071] Special scenarios can be handled using the state section. For example, you might sometimes want to provide users with the option to temporarily leave the first skill they are interacting with and perform an action in a second skill within the digital assistant. For instance, if a user is busy conversing with the shopping skill (e.g., the user has made some purchase choices), the user might want to jump to the banking skill (e.g., the user might want to ensure he / she has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, the action in the first skill can be configured to initiate an interaction with a second, different skill within the same digital assistant and then return to the original stream.

[0072] (6) Adding Custom Parts to the Skill Robot—As described above, the states specified in the skill robot's dialogue flow are named for parts that provide the required functionality corresponding to those states. Parts enable the skill robot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured parts for performing a wide variety of functions. The skill robot designer can select one or more of these pre-configured parts and associate them with states in the skill robot's dialogue flow. The skill robot designer can also use the tools provided by DABP 102 to create custom parts or new parts and associate custom parts with one or more states in the skill robot's dialogue flow.

[0073] (7) Testing and Deploying Skilled Robots – DABP 102 provides several features that enable skilled robot designers to test the skilled robots they are developing. The skilled robots can then be deployed and included in digital assistants.

[0074] While the above description outlines how to create skill bots, similar techniques can be used to create digital assistants (or master bots). At the master bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents identify general tasks that the digital assistant (i.e., the master bot) can handle without invoking the skill bots associated with it. Examples of system intents defined for the master bot include: (1) Exit: applicable when a user signals that they want to exit the current session or context within the digital assistant; (2) Help: applicable when a user requests help or direction; and (3) UnresolvedIntent: applicable to user input that doesn't quite match the Exit and Help intents. The digital assistant also stores information about one or more skill bots associated with it. This information enables the master bot to select the specific skill bot to handle utterances.

[0075] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user's input utterance should be routed to a specific skill for handling, handled by the digital assistant or master robot itself according to built-in system intent, or handled as a different state within the current conversation flow.

[0076] In some embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If the invocation name is present in the user input, the invocation name is considered an explicit invocation of the skill bot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In some embodiments, if no specific invocation or explicit invocation exists, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and skill bot associated with the digital assistant. The score calculated for the skill bot or system intent indicates how likely the user input represents a task that the skill bot is configured to perform or represents a system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Any intent with a confidence score exceeding a threshold (e.g., 70%) is generally considered a candidate intent. If a specific skill bot is selected, the user's utterance is routed to that skill bot for further processing. If a system intent is selected, the main bot itself performs one or more actions based on the selected system intent.

[0077] Figure 2 This is a simplified block diagram of a master robot (MB) system 200 according to certain embodiments. The MB system 200 may be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, multiple intention subsystems (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data storage 250. Figure 2 The depicted MB system 200 is merely an example of component arrangement in the main robot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have a more... Figure 2 The systems or components shown may have more or fewer subsystems, may combine two or more subsystems, or may have different subsystem configurations or arrangements.

[0078] The preprocessing subsystem 210 receives the utterance “A” 202 from the user and processes it through a language detector 212, a language parser 214, and a named entity recognizer 216. As described above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation marks. For example, if the utterance 202 is provided as audio, the preprocessing subsystem 210 can use a speech-to-text converter (not shown) that inserts punctuation marks (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio into text.

[0079] The language detector 212 detects the language of discourse 202 based on the text of discourse 202. The processing of discourse 202 depends on the language, because each language has its own grammar and semantics. Differences between languages ​​are considered when analyzing the syntax and structure of discourse.

[0080] Language parser 214 performs syntactic analysis on discourse 202 to extract part-of-speech (POS) tags for individual language units (e.g., words) within discourse 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. Language parser 214 can also tokenize the language units of discourse 202 (e.g., convert each word into a separate token) and categorize words according to inflectional forms. A lemma is the principal form of a group of words as represented in a dictionary (e.g., "run" is a lemma of run, runs, ran, running, etc.). Other types of preprocessing that language parser 214 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". Language parser 214 can also identify relationships between words in discourse 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which part of a utterance (e.g., a specific noun) is a direct object, which part of the utterance is a preposition, etc.

[0081] Named entity recognizer 216 identifies and classifies named entities in text (e.g., utterances) into predefined categories such as people, organizations, places, time expressions, currencies, common resource language addresses, etc. Identity recognition involves detecting a single word or string of words that constitutes a predefined entity. For example, each word represents a token: “The Pacific Ocean” is a string of three tokens that represents an entity. In some instances, to facilitate entity recognition, an inside-outside-start tokenization method can be used to indicate the start and end positions of an entity. Each of the identified entities can then be classified into a predefined category. For example, a named entity recognition machine learning model might detect the word “Oracle” in a text string, identify that word as an entity, and classify it as “Company”. To learn what is and what is not a relevant entity and how to classify them, the named entity recognition machine learning model implemented by named entity recognizer 216 is trained on a training data corpus. The greater the relevance of the training data to the task or skill, the more accurate each model will be in recognizing named entities for the task or skill. Once entities and categories are defined, they can be used to label data and create a training dataset (in some instances, named entity recognition data programs can automate this process). The training dataset can then be used to train one or more algorithms to predictively label entities and categories in emerging discourse.

[0082] The result of the processing performed by the language detector 212, the language parser 214 and the named entity recognizer 216 forms the extracted information 205, which is provided as input to the MIS 220 along with the utterance 202 itself.

[0083] As described above, utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, utterance 202 may be treated as a single unit, even if it comprises multiple sentences. However, in some embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210, to identify individual sentences among multiple sentences for use in multiple intent analysis and explicit call analysis. Generally, whether utterance 202 is processed at the level of a single sentence or as a single unit comprising multiple sentences, the results produced by MIS 220 and EIS 230 are substantially the same.

[0084] MIS 220 determines whether utterance 202 represents multiple intentions. Although MIS 220 can detect the presence of multiple intentions in utterance 202, the processing performed by MIS 220 does not involve determining whether the intention of utterance 202 matches any intention already configured for the robot. Instead, the processing to determine whether the intention of utterance 202 matches a robot intention can be performed by the intention classifier 242 of MB system 200 or (e.g., as...) Figure 3 The intent classifier of the skill robot (as shown in the embodiment) is executed. The processing performed by MIS 220 assumes the existence of a robot that can handle utterance 202 (e.g., a specific skill robot or the main robot itself). Therefore, the processing performed by MIS 220 does not need to know which robots are in the chatbot system (e.g., the identity of skill robots registered with the main robot) or what intents have been configured for a specific robot.

[0085] To determine that utterance 202 includes multiple intentions, MIS 220 applies one or more rules from a set of rules 252 in data storage 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include sentence patterns indicating the presence of multiple intentions. For example, a sentence pattern may include co-conjunctions connecting two parts of a sentence (e.g., conjunctions) that correspond to different intentions. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intentions. It should be noted that utterances with multiple intentions do not necessarily have different intentions (e.g., intentions involving different robots or different intentions within the same robot). Instead, utterances can have different instances of the same intention, such as “Place a pizza order using payment account X, then place a pizza order using payment account Y”.

[0086] As part of determining that discourse 202 represents multiple intentions, MIS 220 also determines which parts of discourse 202 are associated with each intention. MIS 220 constructs new discourses for individual processing in place of the original discourses for each intention represented in a discourse containing multiple intentions, for example, such as... Figure 2The original utterance 202 can be divided into two or more separate utterances, processed one at a time, as depicted in the diagram. The MIS 220 uses the extracted information 205 and / or analysis of the utterance 202 itself to determine which of the two or more utterances should be processed first. For example, the MIS 220 may determine that utterance 202 contains markers indicating a specific intention that should be processed first. The newly formed utterance corresponding to that specific intention (e.g., one of utterances 206 or 208) will be sent first for further processing by the EIS 230. After the session triggered by the first utterance has ended (or been temporarily suspended), the next highest priority utterance (e.g., another utterance of utterances 206 or 208) can then be sent to the EIS 230 for processing.

[0087] EIS 230 determines whether a received utterance (e.g., utterance 206 or utterance 208) contains a call name for a skill bot. In some embodiments, each skill bot in the chatbot system is assigned a unique call name that distinguishes it from other skill bots in the chatbot system. A list of call names may be stored in data storage 250 as part of skill bot information 254. When a utterance contains a word that matches a call name, the utterance is considered an explicit call. If a bot is not explicitly called, the utterance received by EIS 230 is considered a non-explicit call utterance 234 and is fed to the main bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to handle the utterance. In some instances, intent classifier 242 will determine that the main bot should handle non-explicit call utterances. In other instances, intent classifier 242 will determine which skill bot to route the utterance to for handling.

[0088] The explicit invocation functionality provided by EIS 230 offers several advantages. It reduces the amount of processing the master robot must perform. For example, when explicit invocation is present, the master robot may not need to perform any intent classification analysis (e.g., using intent classifier 242), or may need to perform a simplified intent classification analysis to select the skill robot. Therefore, explicit invocation analysis can achieve the selection of a specific skill robot without relying on intent classification analysis.

[0089] Furthermore, there may be functional overlap between multiple skill robots. This can occur, for example, if two skill robots handle overlapping or very similar intentions. In such cases, the master robot may struggle to identify which of the multiple skill robots to select based solely on intention classification analysis. In this scenario, explicit invocation ensures unambiguity regarding the specific skill robot to be used.

[0090] In addition to determining whether a utterance is an explicit invocation, EIS 230 is also responsible for determining whether any part of the utterance should be used as input to the explicitly invoked skill robot. Specifically, EIS 230 can determine whether a portion of the utterance is irrelevant to the invocation. EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. EIS 230 may send the irrelevant portion of the utterance to the invoked skill robot instead of sending the entire utterance received by EIS 230. In some instances, the input to the invoked skill robot is simply formed by removing any utterance portion associated with the invocation. For example, “I want to order pizza using Pizza Bot” can be shortened to “I want to order pizza” because “using Pizza Bot” is associated with the invocation of the Pizza Bot but not with any processing to be performed by the Pizza Bot. In some instances, EIS 230 can reformat the portion to be sent to the invoked robot, for example, to form a complete sentence. Therefore, EIS 230 not only determines that an explicit call exists, but also, when an explicit call exists, determines what to send to the skill robot. In some instances, there may be no text to be input into the called robot. For example, if the utterance is "Pizza Bot," EIS 230 can determine that the pizza robot is being called, but there is no text to be processed by the pizza robot. In such a scenario, EIS 230 can indicate to the skill robot caller 240 that there is no content to send.

[0091] Skill robot invoker 240 invokes skill robots in various ways. For example, skill robot invoker 240 may invoke a robot in response to receiving an instruction 235 indicating that a specific skill robot has been selected as a result of an explicit invocation. Instruction 235 may be sent by EIS 230 along with the input of the explicitly invoked skill robot. In this scenario, skill robot invoker 240 relinquishes control of the session to the explicitly invoked skill robot. The explicitly invoked skill robot will determine the appropriate response to the input from EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new session in a specific state, where the initial state of the new session depends on the input sent from EIS 230.

[0092] Another way the skill robot invoker 240 can invoke a skill robot is through implicit invocation using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that a utterance represents a task configured to be performed by a particular skill robot. The intent classifier 242 is trained on different categories, one category per skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. The parameters produced as a result of this training (e.g., a set of parameter values ​​for a machine learning model) can be stored as part of the skill robot information 254.

[0093] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described further in detail herein. Training the machine learning model may involve taking input from at least a subset of example utterances associated with various skill robots to generate an inference as the output of the machine learning model about which robot is the correct robot for disposing of any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as basic fact information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the discrepancy between the generated inference and the basic fact information.

[0094] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot, indicating the likelihood that the skill robot can handle a utterance (e.g., a non-explicit invocation utterance 234 received from EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score meets one or more conditions, the skill robot invoker 240 invokes the robot associated with that particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain margin. Imposing such a condition when the confidence scores of multiple skill robots all exceed the threshold confidence score value will enable routing to a particular skill robot.

[0095] After identifying a robot based on a confidence score assessment, the skill robot invoker 240 hands over processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is a skill robot. Further, the skill robot invoker 240 determines what to provide as input 247 to the identified robot. As described above, in the case of explicit invoking, input 247 can be based on a utterance portion unrelated to the invoking, or input 247 can be nothing (e.g., an empty string). In the case of implicit invoking, input 247 can be the entire utterance.

[0096] Data storage 250 includes one or more computing devices that store data used by various subsystems of the main robot system 200. As explained above, data storage 250 includes rules 252 and skill robot information 254. Rules 252 include, for example, rules for determining, by MIS 220, when a utterance represents multiple intents and how to segment the utterance representing multiple intents. Rules 252 further include rules for determining, by EIS 230, which parts of the utterance explicitly invoking the skill robot should be sent to the skill robot. Skill robot information 254 includes the invoking names of skill robots in the chatbot system, for example, a list of invoking names of all skill robots registered through a specific main robot. Skill robot information 254 may also include information used by intent classifier 242 to determine the confidence score of each skill robot in the chatbot system, such as parameters of a machine learning model.

[0097] Figure 3 This is a simplified block diagram of a skill robot system 300 according to certain embodiments. The skill robot system 300 is a computing system that can be implemented solely in software, solely in hardware, or a combination of hardware and software. In some embodiments, such as Figure 1 In the depicted embodiments, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0098] The skill robot system 300 includes a MIS 310, an intent classifier 320, and a session manager 330. The MIS 310 is similar to... Figure 2 MIS 220 provides similar functionality, including operablely using rules 352 in data storage 350 to determine: (1) whether a utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by MIS 310 for detecting multiple intents and for segmenting utterances are the same as those applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to... Figure 2The information extracted from the system 205 can be generated using a language detector 212, a language parser 214 and / or a named entity recognizer 216, or a language detector, language parser and / or named entity recognizer native to the skill robot system 300.

[0099] The intent classifier 320 can be combined with the above. Figure 2 The intent classifier 242 discussed in the embodiments is trained in a similar manner and is described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. For a specific skill robot, the machine learning model of the intent classifier 320 is trained using at least a subset of example utterances associated with that specific skill robot as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with the training utterance.

[0100] Phras 302 can be received directly from the user or provided through the main bot. When phrasing 302 is provided through the main bot, for example, as via... Figure 2 The results processed by MIS 220 and EIS 230 in the embodiments depicted can bypass MIS 310 to avoid repeating the processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a session after routing to a skill bot, MIS 310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance “D” 306 and utterance “E” 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0101] Intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skill robot system 300. As explained above, the skill robot may be configured with one or more intents, each intent including at least one example utterance associated with that intent and used to train the classifier. Figure 2In one embodiment, the intent classifier 242 of the main robot system 200 is trained to determine confidence scores for each skill robot and confidence scores for system intents. Similarly, an intent classifier 320 can be trained to determine confidence scores for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and therefore more granular. The intent classifier 320 can access intent information 354. For each intent associated with the skill robot system 300, the intent information 354 includes a list of utterances that represent the intent and explain its meaning, and are generally associated with a task that can be performed by that intent. The intent information 354 may further include parameters generated as a result of training on this list of utterances.

[0102] The session manager 330 receives a specific intent indication 322 as the output of the intent classifier 320, which identifies the indication as the best match for the utterance input to the intent classifier 320. In some instances, the intent classifier 320 may not determine any match. For example, if the utterance relates to a system intent or the intent of a different skill robot, the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill robot system 300 may submit the utterance to the master robot for processing, e.g., to route it to a different skill robot. However, if the intent classifier 320 successfully identifies an intent within a skill robot, the session manager 330 will initiate a session with the user.

[0103] A session initiated by session manager 330 is a session specific to an intent identified by intent classifier 320. For example, session manager 330 may be implemented using a state machine configured to perform a dialogue flow in response to an identified intent. The state machine may include a default starting state (e.g., when an intent is invoked without any additional input) and one or more additional states, each associated with an action to be performed by the skill robot (e.g., performing a purchase transaction) and / or a dialogue to be presented to the user (e.g., a question, a response). Thus, session manager 330 may determine an action / dialogue 335 upon receiving an indication 322 that an intent has been identified, and may determine additional actions or dialogues in response to subsequent utterances received during the session.

[0104] Data storage 350 includes one or more computing devices that store data used by various subsystems of the skill robot system 300. For example... Figure 3 As depicted, data storage 350 may include rules 352 and intent information 354. In some embodiments, data storage 350 may be integrated into the data storage of the main robot or digital assistant, for example... Figure 2The data storage in the middle is 250.

[0105] Entity data augmentation It has been found that models for entity recognition and extraction perform poorly on text containing entities that can be expressed in a variety of languages ​​and formats. To overcome this problem, various embodiments involve techniques for augmenting training data with a diverse set of entities in any number of languages ​​and formats (e.g., adding various texts and text formats used to describe one or more entities) so that the model can detect entities with a variety of variations (currency types, number formats, data-time formats, etc.). By augmenting the training data with a diverse set of entities in any number of languages ​​and formats, these models improve in entity recognition and extraction. Models trained on augmented entity data can be implemented in chatbot systems, such as those described above. Figure 1 , Figure 2 and Figure 3 As described. Advantageously, these models and chatbots perform better on utterances in various languages ​​and / or text formats because they are better able to identify and extract entities from the utterances. Furthermore, since the augmentation is applied automatically in a synthesis-agnostic manner, clients do not need to worry about manually adding utterances in various languages ​​and / or text formats to the training data.

[0106] Figure 4 A block diagram illustrating various aspects of a chatbot system 400 configured to train one or more models based on text data is shown, said one or more models being implemented, for example, as named entity recognizers. Figure 4 As shown, the entity recognition performed by the chatbot system 400 in this example includes various stages: a prediction model training stage 410 for building and training a model, an evaluation stage 415 for evaluating the performance of the trained model, and a chatbot implementation stage 420 for implementing one or more chatbots. The prediction model training stage 410 builds and trains one or more prediction models 425a-425n ('n' represents any natural number) for use in the other stages (the prediction models may be individually referred to as prediction model 425 or collectively as prediction model 425 herein). For example, prediction model 425 may include a model for identifying one or more entities in a utterance, another model for determining the probability that the utterance represents a task configured to be performed by a robot with a particular skill, another model for predicting intent based on the utterance of a first-type skill robot, and another model for predicting intent based on the utterance of a second-type skill robot. Other types of prediction models may be implemented in other examples according to this disclosure.

[0107] Predictive model 425 may be a machine learning (“ML”) model, such as a convolutional neural network (“CNN”), for example, an initial neural network, a residual neural network (“ResNet”); or a recurrent neural network, such as a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model; or other variants of deep neural networks (“DNN”), such as a multi-label n-binary DNN classifier or a multi-class DNN classifier for single-intent classification. Predictive model 425 may also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model. The chatbot system 400 may employ the same or different types of predictive models to identify one or more entities in a utterance, determine the probability that the utterance represents a task that a robot with a specific skill is configured to perform, predict intent based on the utterance of a first type of skill robot, and predict intent based on the utterance of a second type of skill robot. Other types of predictive models may be implemented in other examples according to this disclosure.

[0108] To train various prediction models 425, the training phase 410 consists of three main parts: dataset preparation 430, template engineering 435, and model training 440. Dataset preparation 430 includes loading data assets 445, splitting the data assets 445 into training and validation sets 445 an so that the system can train and test the prediction models 425, and performing basic preprocessing. Splitting the data assets 445 into training and validation sets 445 an can be done randomly (e.g., 90% / 10% or 70% / 30%), or the split can be performed according to more complex validation techniques (such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc.) to minimize sampling bias and overfitting.

[0109] Training data 445a may include at least a subset of utterances from example utterances associated with one or more skill-based bots. As described above, utterances can be provided in various ways, including audio or text. A utterance can be a sentence fragment, a complete sentence, multiple sentences, etc. For example, if the utterance is provided as audio, data preparation 430 can use a speech-to-text converter (not shown) to insert punctuation marks (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio to text. In some instances, example utterances are provided by a client or customer. In other instances, example utterances are automatically generated from a previous utterance library (e.g., utterances from a library specific to the skills the chatbot is designed to learn). Training data 445a for prediction model 425 may include input text or audio (or input features of text or audio frames) and labels 450 corresponding to the input text or audio (or input features) as a matrix or value table. For example, for each training utterance, basic factual information as labels 450 can be provided as indications of the correct entities to be inferred by prediction model 425 and their classifications. The behavior of the prediction model 425 can then be adapted (e.g., through backpropagation) to minimize the discrepancy between the inferences generated for various entities and the basic factual information.

[0110] Template engineering 435 includes the process of augmenting training data 445a with artificial utterances comprising a diverse set of entities in any number of languages ​​and formats, to make predictive model 425 more resilient to utterances in various languages ​​and / or text formats. By augmenting training data 445a with artificial utterances comprising a diverse set of entities in any number of languages ​​and formats, predictive model 425 improves its ability to recognize entities in various languages ​​and text formats. Augmentation comprises several operations, broadly: (i) extracting templates from the original training dataset, which ensures that the augmented training data has a similar distribution of predictor variables to the original training dataset; and (ii) using a slot mapping method based on a custom grammar to generate augmented training data comprising artificial utterances comprising a diverse set of entities in any number of languages ​​and formats, such as regarding Figures 5A to 5C Detailed description. With the introduction of entity-level data augmentation, prediction model 425 performs better on utterances in various languages ​​and / or text formats because it is better able to recognize entities in any number of languages ​​and formats. Furthermore, since entity-level data augmentation is applied automatically in a synthesis-agnostic manner, clients do not need to worry about manually adding utterances in various languages ​​and / or text formats to the training data.

[0111] The training process for model 425 includes: selecting hyperparameters for model 425 and performing iterative operations by inputting utterances from a subset of the augmented training data 445a into model 425 to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes the objective function, such as minimizing the loss function. Hyperparameters are settings that can be tuned or optimized to control the behavior of model 425. Most models explicitly define hyperparameters that control different aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined to adapt the model to specific scenarios. For example, hyperparameters may include the number of hidden units or layers of the model, the learning rate of the model, the kernel width of the convolutions, or the number of parameters of the model. Each training iteration may involve finding a set of model parameters for model 425 (configured with the defined set of hyperparameters) such that the objective function value using this set of model parameters is less than the objective function value using a different set of model parameters in the previous iteration. The objective function can be constructed to measure the difference between the output inferred using model 425 and the underlying facts labeled using label 150 to a subset of the augmented training data 445a.

[0112] Once the model parameter set is identified, model 425 is trained and can be tested or validated using a subset of test data 445b (the test or validation dataset). The testing or validation process involves iteratively inputting utterances from a subset of test data 445b into model 425 using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc., to tune hyperparameters and ultimately find the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set from a subset of test data 445a can be input into model 425 to obtain outputs (in this example, one or more identified entities), and the outputs can be evaluated against the underlying fact entities using relevance techniques such as the Brand-Ottman method and Spearman rank correlation coefficient. Furthermore, performance metrics 450, such as error, accuracy, precision, recall, receiver operating characteristic (ROC) curve, etc., can be calculated in the entity recognition evaluation phase 415. Metric 450 can be used in the entity recognition evaluation phase 415 to analyze the performance of the model 425 used for entity recognition, such as regarding Figures 5A to 5C Detailed description.

[0113] The model training phase 410 outputs a trained model, including one or more trained entity recognition models 455. These one or more trained entity recognition models 455 can be deployed and used in the chatbot implementation phase 420 for implementing one or more chatbots. For example, one or more chatbots can be configured with one or more trained entity recognition models 455 to receive text data 460 from one or more users and to identify and extract entities 465 from various utterances received by the one or more chatbots. Entities 465 can be information extracted from the text data 460 (e.g., ...). Figure 2 and Figure 3 The extracted information (205 and 304) described in the text is a portion of the information and can be used for downstream processing, such as intent classification.

[0114] Entity data augmentation technology Figure 5A This is a flowchart illustrating the process of adjusting the overall prediction model using entity data augmentation. Figure 5A The described process can be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 5A The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 5A The individual processing steps, occurring in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as Figures 1 to 4 In the embodiments depicted, Figure 5A The processing described herein can be performed by a preprocessing subsystem (e.g., preprocessing subsystem 210 or prediction model training phase 410) to create, train, update, and deploy one or more prediction models (implemented as named entity recognizers) using the augmented dataset.

[0115] At step 505, the data processing system (e.g., regarding...) Figure 4The described chatbot system 400 obtains a dataset comprising utterances. The dataset includes realistic, task-oriented dialogue data (consisting of utterances) that can be obtained from corpora such as question-and-answer datasets (e.g., including Wikipedia articles, manually generated factual questions from them, and manually generated answers to these questions), customer support datasets (e.g., two-person conversations extracted from chat logs for receiving technical support for various issues), dialogue datasets (e.g., a collection of fictional dialogues extracted from original movie scripts, including dialogue between pairs of movie characters), multilingual datasets (e.g., feedback from customers in various languages ​​stating the reasons for dissatisfaction with a given company, product, or service), and so on.

[0116] At step 510, the dataset is split into a training dataset and a test and / or validation dataset (the evaluation dataset serves as the target for one or more predictive models). Although the following processing is explained with reference to a single training dataset and a single test and / or validation dataset, it should be understood that the dataset can be split into one or more training datasets and one or more test and / or validation datasets, and processed in a similar manner. Splitting the data assets into training and validation sets can be performed randomly (e.g., 90% / 10% or 70% / 30%), or the split can be performed based on more sophisticated validation techniques (such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc.) to minimize sampling bias and overfitting. The training dataset can be labeled before or after it is acquired. Labeling may include predefined lists of categories for entities, locating entities in each utterance, and labeling entities with categories from the predefined list of categories. For example, entity categories can be predefined as PRODCUT, COMPANY, CURRENCY, and TIME. Indexes can identify utterances in which entities fall between these categories, and labels can be created for entity categories within index locations (e.g., if entities like Oracle, which are categorized under COMPANY, fall between indexes 9 and 17, and entities like 10 / 25 / 20, which are categorized under TIME, fall between indexes 35 and 45, then the utterance can be labeled as (entity: (9, 17, 'COMPANY'), (35, 45, 'TIME')).

[0117] At step 515, a prediction model is trained on the training dataset to learn parameters for recognizing and classifying entities in the utterance. The training process may include selecting hyperparameters for the model and performing iterative operations by inputting utterances from the training dataset into the model to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes an objective function, such as minimizing a loss function. Each training iteration may involve finding a set of model parameters (configured with a defined set of hyperparameters) such that the objective function value using that set is less than the objective function value used in the previous iteration with a different set of model parameters. The objective function may be constructed to measure the difference between the output inferred using the model and the underlying facts labeled onto the training dataset.

[0118] At step 520, one or more predictive models are tested and / or validated on test and / or validation datasets to identify one or more entities in a discourse. The testing or validation process involves iterative operations that input discourse from the test and / or validation datasets into the model using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc., to tune hyperparameters and ultimately find the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set from the test and / or validation datasets can be input into the model to obtain output (e.g., one or more identified entities), and the output can be evaluated against basic fact entities using correlation techniques such as the Brand-Ottman method and Spearman rank correlation coefficient. Furthermore, performance metrics such as error (e.g., mean absolute error or mean squared error), accuracy (e.g., classification accuracy), precision (e.g., F1 score), recall (e.g., F1 score), receiver operating characteristic (ROC) curve (e.g., area under the curve (AUC) - ROC curve), etc., can be calculated. These metrics can be used to analyze the performance of the model used for entity identification. For example, evaluation tools can be used to understand false positives and false negatives on the retained test set to assess the model's behavior. Testing and / or validation are performed and analyzed to obtain a baseline evaluation of the model and (if any) to gain an understanding that additional information (e.g., additional nouns, text formats, currency types, URL types, etc.) needs to be added to or diversified in the training dataset to improve the performance of the model for recognizing entities in utterances of various languages ​​and / or text formats. For example, testing and / or validation can demonstrate that the model performs poorly when attempting to identify and classify the entity CURRENCY (especially for utterances without the US dollar currency), and thus gain an understanding that additional information (e.g., additional currency types) needs to be added to or diversified in the training dataset to improve the performance of the model for the entity CURRENCY.

[0119] In instances where additional information needs to be added to or diversified in the training dataset to improve model performance, at step 525, one or more templates are created or extracted from the original corpus—the dataset obtained in step 505 before splitting—or a subset thereof (such as the split training dataset), as per [reference to...]. Figure 5B Detailed description. In step 530, the data processing system uses one or more templates to create artificial utterances, such as about Figure 5BDetailed description. At step 535, the data processing system augments the dataset or a subset thereof with human utterance to obtain an augmented dataset. The rate at which the dataset or subset is augmented with human utterance (the augmentation rate) can be set based on one or more factors. In some instances, the augmentation rate is selected based on analysis of test and validation data; for example, the augmentation rate is selected based on its perception / deterministic ability to improve the performance of the named entity recognition model. In other instances, the augmentation rate is randomly selected and can be optimized thereafter. In still other instances, the augmentation rate is selected based on a predefined pattern to prevent degradation in the ability of one or more prediction models to predict entities from one or more previously observed classifications. For example, the augmentation rate can be defined by the following pattern(s): CURRENCY: 3 The number of times monetary terms appeared in the original training dataset, and / or TIME_DATE: 0.75 |Number of times the punctuation mark "time-date" appears in the original training dataset|.

[0120] At step 540, the augmented dataset is split into an augmented training dataset and an augmented test and / or validation dataset (the evaluation dataset serves as the target for one or more predictive models). Although the following processing is explained with reference to a single training dataset and a single test and / or validation dataset, it should be understood that the augmented dataset can be split into one or more augmented training datasets and one or more augmented test and / or validation datasets, and processed in a similar manner. Splitting the augmented data assets into training and validation sets can be performed randomly (e.g., 90% / 10% or 70% / 30%), or the split can be performed based on more sophisticated validation techniques (such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc.) to minimize sampling bias and overfitting. The augmented training dataset is labeled based on a predefined list of classifications for the entities, as described with respect to step 510.

[0121] At step 545, the model is trained using the expanded training dataset (or a new base model is trained). The training process may include: selecting hyperparameters for the model and performing iterative operations that input utterances from the expanded training dataset into the model to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes the objective function, e.g., minimizing the loss function. Each training iteration may involve finding a set of model parameters (configured with a defined set of hyperparameters) such that the objective function value using that set is less than the objective function value used in the previous iteration with a different set of model parameters. The objective function may be constructed to measure the difference between the output inferred using the model and the underlying facts labeled onto the expanded training dataset.

[0122] At step 550, the model is tested and / or validated on the expanded test and / or validation dataset to identify one or more entities in the discourse. The testing or validation process involves iterative operations using validation techniques (such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc.) to input discourse from the expanded test and / or validation dataset into the model to tune hyperparameters and ultimately find the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set from the test and / or validation dataset can be input into the model to obtain the output (e.g., one or more identified entities), and the output can be evaluated against the underlying fact entities using correlation techniques (such as the Brand-Ottman method and Spearman rank correlation coefficient). Furthermore, performance metrics such as error (e.g., mean absolute error or mean squared error), accuracy (e.g., classification accuracy), precision (e.g., F1 score), recall (e.g., F1 score), receiver operating characteristic (ROC) curve (e.g., area under the curve (AUC) - ROC curve), etc., can be calculated. The metrics can be used to analyze the performance of the model used to identify entities. Compared to the baseline evaluation of the model in step 520, the metrics can also be used to analyze the model's performance. For example, evaluation tools can be used to understand false positives and false negatives on the retained test set to evaluate the model's behavior. Differentiation tools can be used to understand the differences between the new model and the previous version of the model, and / or the differences between the new model and the base model. To determine if any degradation exists, degradation tools can be used to ensure that there is no degradation between the two model versions and / or between the new model and the base model.

[0123] Subsequently, based on the evaluation of model training and testing, steps 525 to 550 can be iteratively repeated until the evaluation demonstrates that one or more predictive models for identifying one or more entities in utterances of various languages ​​and / or text formats have optimal performance. The optimal performance of the model can be defined by the user or system as achieving one or more performance metrics (e.g., accuracy of x%, achieving an error score of x, an increase in accuracy of x% compared to the baseline evaluation or previous version of the model, a reduction in the error score for classifying a given entity of at least x%, an improvement of at least x% in identifying and classifying a given entity compared to the baseline evaluation, etc.). Once the model has been trained and / or has reached optimal performance, at step 555, the model is deployed within a chatbot system (e.g., as part of a skills bot) for identifying one or more entities in utterances of various languages ​​and / or text formats. For example, the chatbot may receive utterances that can be analyzed by one or more predictive models to determine whether the utterances contain one or more entities, and the identified entities can be used by the chatbot in downstream processes to, for example, infer the intent of the utterances.

[0124] Figure 5B This is a flowchart illustrating a process 560 for creating or extracting one or more templates from one or more training datasets and creating artificial discourse based on said one or more templates, as shown in the section on... Figure 5A The process described is implemented in 500. Figure 5B The described process can be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 5B The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 5B The individual processing steps, occurring in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figures 1 to 4 In the embodiments depicted, Figure 5B The processing described herein can be performed by a data processing system comprising a template engine configured to automatically generate templates and utterances for training a predictive model. The template engine automatically generates a large amount of augmented data that is similar in context to the training data already used within the training system, while also possessing sufficient variation to ensure the model does not learn specific sentence patterns. This is more advantageous than crowdsourced data (which provides quantity but dissimilar context) or manually adding more utterances (which provides similar context but cannot manually generate enough utterances or ensure some diversity of data).

[0125] The following discussion of process 560 explains the data augmentation process (which applies to the entire process 500) by using CURRENCY as an example entity, which is identified by the user or computing system (e.g., via the analysis performed in steps 520 and 550) as potentially benefiting from data augmentation (e.g., the training data lacks sufficient coverage for certain currency units). Although CURRENCY is used as an example entity, it should be understood that the process described herein can also be implemented for other categories of entities and combinations thereof. At step 565, a list of values ​​to be covered for a given entity is generated or identified. For example, the list of units to be covered for the CURRENCY entity is determined by the data processing system (e.g., regarding...) Figure 4 The chatbot system described (400) generates or identifies names. A list of place names (geographical dictionary or directory) can be used to generate or identify popular terms, such as currencies and their synonyms, primary and secondary units (e.g., dollars and cents), and standard text formats (e.g., lists of currency types categorized by currency number format, country, language, or region). The list may look like this: List of covered currencies SGD$ Singapore Dollar Singapore Dollars Sing (Singapore Dollar) SGD Rupee (Rs.) Indian rupee (Rs) Australian Dollar (AUS Dollar) Australian Dollars Thai Baht R$ EGP SDG LBP ARS COP PHP CLP DOP CUP UYU etc. At step 570, utterances are selected from a dataset having a context of a given entity (e.g., a dataset obtained at step 505 before splitting from the original corpus, or a subset thereof such as a split training dataset). For example, a plurality of utterances may be selected manually or by the data processing system from the dataset having the correct context of the CURRENCY entity. An example utterance that may be selected for the CURRENCY entity is: "He paid $ 100.78 for dinner"; however, it should be understood that, alternatively or additionally, many other example utterances having the CURRENCY entity may be selected.

[0126] At step 575, the selected utterances are converted into templates, wherein each of the templates includes slots mapped to a list of values for the given entity. For example, the data processing system executes a script to template all selected utterances. More particularly, the script will cause the selected utterances to be converted into a list of templates, which are essentially sentences with slots. The templates are stored as a list associated with the given entity and optionally with the context of the given entity, and the templates from the list will then be selected and populated with different values for the entity by a template engine, as described with reference to Figure 5C the detailed description. The templates may be defined with a custom syntax to customize the behavior of the model for identifying the entity classifications defined in the utterances. For example, a context-free grammar (a set of recursive rules for generating string patterns) defined by the syntax of a programming language (such as the metalanguage BNF (Backus-Naur Form)) may be used to model complex entities like currency that have multiple components (e.g., symbols, codes, major and minor units, etc.) and support nested slot mappings. An example template for the selected utterance "He paid $ 100.78 for dinner" may be "He paid for dinner (He paid for dinner)". As another example, for the selected utterance "Create an expense for $ 10 AUD on May 21 st (Create an expense of 10 Australian dollars on May 21)", the template may be Create an expense (Create an (expense|exp) for Each slot consists of one or more elements, which are mapped to one or more values ​​(e.g., a set of potential values ​​or an operation to be performed). For example, continuing the example above, slots: Mapped to the following elements: and .element It is a list of values; and the element is a list of values. These are function type elements. Each element: and Mapped to individual values, for example, The list of units mapped to the CURRENCY entity, and Mapped to a function used to generate decimal numbers for CURRENCY entities.

[0127] At step 580, a template is selected from the template list, values ​​are selected from the value list based on the mapping between slots within the selected template and the value list of a given entity, and an artificial utterance is created based on the selected template and the selected values. Creating the artificial utterance involves inserting the selected values ​​into slots in the selected template that are mapped to the value list of the given entity. For example, a template engine is run with a template list (e.g., a template list including the entity CURRECNY) and a list of values ​​(e.g., units) to be covered for the entity (e.g., CURRENCY) to generate an artificial utterance. The template engine has a configuration file in which the user specifies which slots map to which entity files, and the template engine then follows the mapping to generate the artificial utterance. Multiple templates and / or values ​​can be selected from the list, and multiple artificial utterances can be generated (based on any selected or set expansion rate) by iteratively running the template engine with the template list and the value list to be covered for the entity.

[0128] Figure 5C The flowchart in the image illustrates the usage relative to the example template: "He paid for dinner." "to generate artificial discourse sub-processes (respectively about Figure 5A and Figure 5B (Described 530, 580). At step (A), based on the analysis of test and verification data (e.g., regarding...) Figure 5AThe analyses performed in steps 520 and 550 (described above) determine that entity CURRENCY needs augmentation to improve the performance of the named entity recognition model. Additionally, an augmentation rate is set for the process of creating artificial utterances to augment the training data. In some instances, the augmentation rate is selected based on analysis of test and validation data; for example, it is selected based on its perceived / deterministic ability to improve the performance of the named entity recognition model. In other instances, the augmentation rate is randomly selected and can be optimized subsequently. In still other instances, the augmentation rate is selected based on a predefined pattern to prevent degradation in the ability of one or more predictive models to predict entities from one or more previously observed classifications.

[0129] At step (B), a template is selected from a list of templates associated with the determined entity: CURRENCY and optionally with the context of the given entity (this list of templates can be based on information about the entity). Figure 5B Step 575 is described as generating the template list. When the template list is created, it is associated with entities and optionally with the context of a given entity (e.g., a mapping between the list and entities is created in a file or database). For example, the template engine has a configuration file in which the user specifies which templates map to which entity files, and the template engine follows this mapping to identify the template list associated with the identified entity: CURRENCY. Then, a template is selected from the identified template list either in a controlled manner (i.e., based on a predefined selection protocol, such as sequential or using an applicability function) or randomly (e.g., he paid for dinner). In some instances, template creation and extraction are performed to ensure that the augmented training dataset has the same distribution as the original dataset used to train the model (e.g., from the original corpus—the dataset obtained in step 505 before splitting—or a subset thereof, such as the split training dataset)). For example, heuristics can be used to determine how to select templates. Heuristics may include: (i) choosing between templates with multiple entity classifications and templates with a single entity classification; (ii) weighing template variety against template quality; (iii) categorizing templates based on some context, such as related to account balances; (iv) whether the model requires a minimum number of occurrences of each template / a unique context that the user wants to teach the model; or (v) any combination of the above.

[0130] At step (C), one or more slot elements within the selected template are identified. Slots can be identified by parsing the template and locating the text defining the slots based on a custom syntax used to generate the template, for example, / Any text between the symbols ( / ) can be identified as a slot, including any text with identifiers such as _slot_, or any combination thereof. When slots are created, they are associated with one or more elements (e.g., a mapping between slots and elements is created in a file or database). For example, a template engine has a configuration file in which a user can specify which elements map to which slots, and the template engine will then follow that mapping to identify the elements associated with the identified slots. As illustrated, slots: He was identified from the following template: He paid for dinner. And slots: Mapped to the following elements: and .element It is a list of values; and the element is a list of values. It is a function type element.

[0131] At steps (D) and (E), the values ​​of each element are identified. When slot elements are created, they are associated with values ​​(e.g., a mapping between elements and values ​​is created in a file or database). For example, a template engine has a configuration file in which a user can specify which values ​​map to which elements, and the template engine can then use that mapping to identify the values ​​associated with the identified elements. As illustrated, elements Mapped to a list of various values ​​for entity: CURRENCY; and mapped to a function for generating random decimal numbers as values ​​for entity: CURRENCY.

[0132] At step (F), one or more operations are performed based on the identified values, and the results of the operations are combined to generate a slot value for each identified slot. For example, for a list-of-values ​​element, a value is selected from a list of various values ​​identified for the entity: CURRENCY (i.e., based on a predefined selection protocol, such as sequential or using an applicability function) or randomly (e.g., SGD$). For a function-of-values ​​element, the function is run to obtain its output; for example, a random decimal number generator can be run to obtain 20.45. The value of the list-of-values ​​element: SGD$ is combined with the value of the function-of-values ​​element to generate the slot value: SGD$ 20.45. It should be understood that the combination of values ​​can be more complex than a simple sequential arrangement of values. For example, in some instances, values ​​can be combined using a separate function or a random order generator.

[0133] At step (G), the slot value of each identified slot is then input into the associated slot within the identified template to generate artificial utterance. For example, SGD$20.45 is input into the template: He paid for dinner. Word slots: In the example, artificial utterances are generated: He paid SGD$20.45 for dinner. Multiple templates and / or values ​​can be selected from a list, and multiple artificial utterances can be generated (based on any augmentation rate selected or set) by iteratively running the template engine with the template list and the list of values ​​to be covered for the entity.

[0134] Template engines are highly extensible—while this is a simple example with one template, in reality, a script may provide many different templates, as well as a longer hierarchy of further templates, files, or functions that the template engine can map to.

[0135] Explanatory System Figure 6 A simplified diagram of a distributed system 600 is depicted. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608 coupled to a server 612 via one or more communication networks 610. Client computing devices 602, 604, 606, and 608 can be configured to execute one or more applications.

[0136] In various examples, server 612 may be adapted to run one or more services or software applications implementing one or more embodiments described in this disclosure. In some examples, server 612 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided as web-based services or cloud services (such as under a Software as a Service (SaaS) model) to users of client computing devices 602, 604, 606, and / or 608. Users operating client computing devices 602, 604, 606, and / or 608 may then use one or more client applications to interact with server 612 to utilize the services provided by these components.

[0137] exist Figure 6 In the depicted configuration, server 612 may include one or more components 618, 620, and 622 that implement the functions performed by server 612. These components may include software components that can be executed by one or more processors, hardware components, or a combination thereof. It should be understood that various different system configurations, different from distributed system 600, are possible. Therefore, Figure 6 The example shown is an example of a distributed system for implementing the example system and is not intended to be restrictive.

[0138] Users can use client computing devices 602, 604, 606, and / or 608 to execute one or more applications, models, or chatbots, which can generate one or more events or models that can then be implemented or serviced according to the teachings of this disclosure. The client device can provide an interface that enables a user of the client device to interact with it. The client device can also output information to the user via this interface. Although Figure 6 It describes only four client computing devices, but can support any number of client computing devices.

[0139] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Windows). ® Apple Macintosh ® UNIX ® Or UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome™ OS), including various mobile operating systems (e.g., Microsoft Windows Mobile). ® (iOS®, Windows Phone®, Android™, BlackBerry®, PalmOS®). Portable handheld devices can include cellular phones, smartphones (e.g., iPhone). ® ), tablet computers (e.g., iPad) ® Wearable devices include personal digital assistants (PDAs), etc. Google Glass is another example. ® Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices and internet-enabled gaming devices (e.g., with or without Kinect). ® Microsoft Xbox gesture input device ® Game consoles, Sony PlayStation® systems, various game systems provided by Nintendo®, and others. Client devices can be able to run a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, Short Message Service (SMS) applications), and can use various communication protocols.

[0140] The (multiple) networks 610 can be any type of network familiar to those skilled in the art that supports data communication using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. By way of example only, the (multiple) networks 610 can be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, or a wireless network (e.g., according to the IEEE 1002.11 protocol suite, Bluetooth). ® (and / or any other wireless protocol operating on a network) and / or any combination of these networks and / or other networks.

[0141] Server 612 may consist of one or more general-purpose computers, special-purpose server computers (including, by way of example, PC (personal computer) servers, UNIX servers, etc.). ® Server 612 may be a server, mid-range server, mainframe computer, rack server, server cluster, or any other suitable arrangement and / or combination thereof. Server 612 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as logical storage devices that can be virtualized to maintain one or more flexible pools of virtual storage devices for the server. In various examples, server 612 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0142] The computing system in server 612 can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. Server 612 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA... ® Servers, database servers, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM®, etc.

[0143] In some implementations, server 612 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. As an example, data feeds and / or event updates may include, but are not limited to, Twitter. ® Feedback, Facebook ® The server 612 may receive real-time updates from one or more third-party information sources and continuous data streams. These real-time updates may include real-time events related to sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. The server 612 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client computing devices 602, 604, 606, and 608.

[0144] The distributed system 600 may also include one or more data repositories 614, 616. In some examples, these data repositories may be used to store data and other information. For example, one or more of data repositories 614, 616 may be used to store information (such as information related to chatbot performance or generated models) for use by the chatbot, which is used by server 612 in performing various functions according to various embodiments. Data repositories 614, 616 may reside in various locations. For example, the data repository used by server 612 may be local to server 612 or may be located remotely to server 612 and communicate with server 612 via a network-based or dedicated connection. Data repositories 614, 616 may be of different types. In some examples, the data repository used by server 612 may be a database, such as a relational database, such as a database provided by Oracle Corporation® and other vendors. One or more of these databases may be adapted to respond to SQL-formatted commands to store, update, and retrieve data from the database.

[0145] In some examples, one or more of the data stores 614 and 616 can also be used by the application to store application data. The data store used by the application can be of different types, such as a key-value store, an object store, or a general-purpose storage store supported by a file system.

[0146] In some examples, the functionality described in this disclosure may be provided as a service via a cloud environment. Figure 7 This is a simplified block diagram of a cloud-based system environment in which various services, based on certain examples, can be provided as cloud services. Figure 7In the depicted example, cloud infrastructure system 702 can provide one or more cloud services that can be requested by users using one or more client computing devices 704, 706, and 708. Cloud infrastructure system 702 may include one or more computers and / or servers, which may include those computers and / or servers described above with respect to server 612. The computers in cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server groups, server clusters, or any other suitable arrangement and / or combination.

[0147] Multiple networks 710 can facilitate data communication and exchange between clients 704, 706, and 708 and cloud infrastructure system 702. Multiple networks 710 may include one or more networks. The networks may be of the same or different types. Multiple networks 710 may support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0148] Figure 7 The example depicted is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other examples, the cloud infrastructure system 702 may have more... Figure 7 The depicted components may have more or fewer components, may combine two or more components, or may have different component configurations or arrangements. For example, although... Figure 7 Three client computing devices are depicted, but in alternative examples, any number of client computing devices can be supported.

[0149] The term cloud service is generally used to refer to services that become available to users on demand through a service provider's systems (e.g., cloud infrastructure systems 702) and via communication networks such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's systems differ from the customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. Therefore, customers can utilize cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's systems can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase the infrastructure resources to run the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle Corporation® of Redwood Shores, California, offers several cloud services such as middleware services, database services, Java Cloud services, and others.

[0150] In some examples, cloud infrastructure system 702 may provide one or more cloud services using different models, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 702 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0151] The SaaS model enables applications or software to be delivered to customers as a service via communication networks such as the Internet, without requiring customers to purchase the underlying application's hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, a variety of services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0152] The IaaS model is typically used to provide customers with infrastructure resources (such as servers, storage, hardware, and networking resources) as cloud services to offer elastic computing and storage capabilities. Various IaaS services are provided by Oracle Corporation®.

[0153] The PaaS model is typically used to provide a platform and environment of resources as a service that enables customers to develop, run, and manage applications and services without requiring them to purchase, build, or maintain such resources. Examples of PaaS services provided by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solutions, and other services.

[0154] Cloud services are typically delivered in an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may subscribe to one or more services provided by cloud infrastructure system 702 via a subscription order. Cloud infrastructure system 702 then performs processing to provide the service requested in the customer's subscription order. For instance, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent) as described above and / or to provide services for a chatbot system as described herein. Cloud infrastructure system 702 can be configured to provide one or more cloud services.

[0155] Cloud infrastructure system 702 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 702 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, which can be an individual or a business. In some other examples, under a private cloud model, cloud infrastructure system 702 can operate within an organization (e.g., within a business organization), and services are provided to customers within the organization. For example, customers can be various departments within the organization, such as human resources or payroll departments, or even individuals within the organization. In some other examples, under a community cloud model, cloud infrastructure system 702 and the services provided can be shared by several organizations in the relevant community. Various other models, such as hybrids of the models mentioned above, can also be used.

[0156] Client computing devices 704, 706, and 708 can be of different types (e.g. Figure 6 The client computing devices 602, 604, 606, and 608 depicted may also be capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 702, such as requesting services provided by the cloud infrastructure system 702. For example, a user can use the client device to request information or actions from a chatbot as described in this disclosure.

[0157] In some examples, the processing performed by cloud infrastructure system 702 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, potentially processing data in parallel, performing simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0158] like Figure 7 As illustrated in the examples, cloud infrastructure system 702 may include infrastructure resources 730 used to facilitate the provision of various cloud services offered by cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to service storage requested from an application may be part of cloud infrastructure system 702. In other examples, the storage virtual machine may be part of a different system.

[0159] In some examples, to facilitate the efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 702 to different customers, resources can be bound to resource groups or resource modules (also known as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods may be pre-provisioned for different types of cloud services. For example, a first pod may be provisioned for a database service, and a second pod may be provisioned for a Java service (the second pod may include a different combination of resources than the pods in the first pod), and so on. For some services, resources allocated for provisioning the service may be shared between services.

[0160] The cloud infrastructure system 702 itself can internally use services 732 shared by different components of the cloud infrastructure system 702 and that facilitate the provision of services by the cloud infrastructure system 702. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0161] Cloud infrastructure system 702 may include multiple subsystems. These subsystems may be implemented in software or hardware, or a combination thereof. Figure 7 The depicted subsystem may include a user interface subsystem 712 that enables users or customers of the cloud infrastructure system 702 to interact with the cloud infrastructure system 702. The user interface subsystem 712 may include various interfaces, such as a web interface 714, an online store interface 716 (where advertising and customers can purchase cloud services provided by the cloud infrastructure system 702), and other interfaces 718. For example, a customer may use a client device to request (service request 734) one or more services provided by the cloud infrastructure system 702 using one or more of the interfaces 714, 716, and 718. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 702, and place a subscription order for one or more services provided by the cloud infrastructure system 702 that the customer wishes to subscribe to. A service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 702. As part of the order, the customer may provide information identifying the service to be provided to a chatbot system and optionally provide one or more credentials for the chatbot system.

[0162] In some examples (such as) Figure 7In the illustrated example, cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this process, OMS 720 may be configured to: create an account for a customer (if not already created); receive billing and / or billing information from the customer to be used to bill the customer for the requested services; verify the customer information; after verification, place an order for the customer; and schedule various workflows to prepare the order for delivery.

[0163] Once correctly verified, OMS 720 can invoke the Order Provisioning Subsystem (OPS) 724, configured as order provisioning resources (including processing resources, storage resources, and networking resources). Provisioning may include allocating resources for an order and configuring resources to facilitate the service requested by the customer's order. The manner in which resources are provisioned for an order and the type of resources provided may depend on the type of cloud service the customer has already subscribed to. For example, according to a workflow, OPS 724 may be configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated to an order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users the service is to support, the duration of the requested service, etc. The allocated groups can then be customized for a specific requesting customer to provide the requested service.

[0164] In some examples, the setup phase processing described above can be performed by cloud infrastructure system 702 as part of the provisioning process. Cloud infrastructure system 702 can generate application IDs and select storage virtual machines for applications from storage virtual machines provided by cloud infrastructure system 702 itself or from storage virtual machines provided by other systems besides cloud infrastructure system 702.

[0165] Cloud infrastructure system 702 can send a response or notification 744 to the requesting client to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) enabling the client to begin using and taking advantage of the benefits of the requested service can be sent. In some examples, for the client requesting the service, the response may include a chatbot system ID generated by cloud infrastructure system 702 and information identifying the chatbot system selected by cloud infrastructure system 702 corresponding to the chatbot system with the chatbot system ID.

[0166] Cloud infrastructure system 702 can provide services to multiple customers. For each customer, cloud infrastructure system 702 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 702 can also collect usage statistics about customers' use of subscribed services. For example, it can collect statistics such as storage usage, data transfer volume, number of users, system uptime, and system downtime. This usage information can be used to issue bills to customers. Billing can be done, for example, on a monthly basis.

[0167] Cloud infrastructure system 702 can provide services to multiple customers in parallel. Cloud infrastructure system 702 can store information about these customers (potentially including proprietary information). In some examples, cloud infrastructure system 702 includes an Identity Management Subsystem (IMS) 728 configured to manage customer information and provide separation of the managed information so that information related to one customer cannot be accessed by another customer. IMS 728 can be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0168] Figure 8 An example of a computer system 800 is illustrated. In some examples, computer system 800 can be used to implement any digital assistant or chatbot system within a distributed environment, as well as the various servers and computer systems described above. Figure 8 As shown, the computer system 800 includes various subsystems, including a processing subsystem 804 that communicates with multiple other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. The storage subsystem 818 may include non-transitory computer-readable storage media, including storage medium 822 and system memory 810.

[0169] Bus subsystem 802 provides a mechanism for allowing the various components and subsystems of computer system 800 to communicate with each other as intended. While bus subsystem 802 is schematically shown as a single bus, alternative examples of bus subsystems may utilize multiple buses. Bus subsystem 802 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (which may be implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0170] Processing subsystem 804 controls the operation of computer system 800 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Processors may include single-core or multi-core processors. The processing resources of computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 804 may include one or more dedicated coprocessors such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of processing subsystem 804 may be implemented using custom circuitry such as ASICs or FPGAs.

[0171] In some examples, the processing units in processing subsystem 804 can execute instructions stored in system memory 810 or on computer-readable storage medium 822. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in system memory 810 and / or on computer-readable storage medium 822 (potentially including residing on one or more storage devices). With appropriate programming, processing subsystem 804 can provide the various functions described above. In an instance where computer system 800 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0172] In some examples, a processing acceleration unit 806 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 804, thereby accelerating the overall processing performed by the computer system 800.

[0173] I / O subsystem 808 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 800. Generally, the term input device is intended to include all possible types of devices and mechanisms for inputting information to computer system 800. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect, which enables users to control and interact with input devices. ® Motion sensor, Microsoft Xbox ® The 360 ​​game controller provides an interface for receiving input using gestures and spoken commands. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translating the eye gestures into the input device (such as Google Glass). ® Google Glass input ® Blink detector. Additionally, user interface input devices may include those enabling users to interact with a voice recognition system (e.g., Siri) via voice commands. ® A voice recognition sensing device for interaction with navigators.

[0174] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointing sticks, game controllers and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0175] Generally, the term "output device" is intended to encompass all possible types of devices and mechanisms for outputting information from a computer system 800 to a user or other computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tubes (CRTs), flat panel devices (such as those using liquid crystal displays (LCDs) or plasma displays), projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headsets, car navigation systems, plotters, voice output devices, and modems.

[0176] Storage subsystem 818 provides a repository or data store for storing information and data used by computer system 800. Storage subsystem 818 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Storage subsystem 818 may store software (e.g., programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 804. The software may be executed by one or more processing units of processing subsystem 804. Storage subsystem 818 may also provide certification according to the teachings of this disclosure.

[0177] The storage subsystem 818 may include one or more non-transitory memory devices, which may include volatile memory devices and non-volatile memory devices. For example... Figure 8 As shown, the storage subsystem 818 includes system memory 810 and computer-readable storage medium 822. System memory 810 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines that help transfer information between elements within the computer system 800 during startup may typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by processing subsystem 804. In some embodiments, system memory 810 may include various types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0178] By using examples rather than restrictions, such as Figure 8As depicted, system memory 810 can load an executing application 812 (which may include various applications such as web browsers, middleware applications, relational database management systems (RDBMS), etc.), program data 814, and operating system 816. By way of example, operating system 816 may include various versions of Microsoft Windows. ® Apple Macintosh ® and / or Linux operating system, various commercially available UNIX systems ® Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google Chrome) ® OS, etc.) and / or such as iOS, Windows ® Phone, Android ® OS, BlackBerry ® OS, Palm ® OS operating system and other mobile operating systems.

[0179] Computer-readable storage medium 822 can store programming and data structures that provide some example functionality. Computer-readable storage medium 822 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 800. Software (programs, code modules, instructions) that provides the functionality described above, when executed by processing subsystem 804, can be stored in storage subsystem 818. By way of example, computer-readable storage medium 822 may include, for example, hard disk drives, disk drives, optical disc drives (such as CD ROMs, DVDs, Blu-ray discs). ® Non-volatile memory such as disks or other optical media. Computer-readable storage media 822 may include, but is not limited to, Zip... ® Drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 822 may also include SSDs based on flash memory, such as flash memory-based solid-state drives (SSDs), enterprise-class flash memory drives, SSDs based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0180] In some examples, the storage subsystem 818 may also include a computer-readable storage medium reader 820 that can be further connected to a computer-readable storage medium 822. The reader 820 may receive data from a storage device such as a disk, flash memory drive, etc., and is configured to read data from said storage device.

[0181] In some examples, computer system 800 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 800 may provide support for executing one or more virtual machines. In some examples, computer system 800 may execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing (e.g., processor, core), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. Virtual machines typically run their own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 800. Therefore, multiple operating systems may potentially be run simultaneously by computer system 800.

[0182] The communication subsystem 824 provides interfaces to other computer systems and networks. The communication subsystem 824 serves as an interface for receiving data from other systems and transmitting data from computer system 800 to other systems. For example, the communication subsystem 824 can enable computer system 800 to establish communication channels to one or more client devices via the Internet for receiving and sending information to client devices. For example, when computer system 800 is used to implement... Figure 1 When the robot system 120 is described, the communication subsystem can be used to communicate with a chatbot system selected for the application.

[0183] The communication subsystem 824 may support both wired and / or wireless communication protocols. In some examples, the communication subsystem 824 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies such as cellular phone technology, 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 802.XX Home Standard, or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some examples, in addition to or as an alternative to a wireless interface, the communication subsystem 824 may provide wired network connectivity (e.g., Ethernet).

[0184] The communication subsystem 824 can receive and transmit data in various forms. In some examples, among others, the communication subsystem 824 can also receive input communications in the form of structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc. For example, the communication subsystem 824 can be configured to receive (or send) data feeds 826 in real time from users of social media networks and / or other communication services, such as Twitter. ® Feedback, Facebook ® Updates, web feeds (such as rich site summary (RSS) feeds) and / or real-time updates from one or more third-party information sources.

[0185] In some examples, the communication subsystem 824 may be configured to receive data in the form of a continuous data stream, which may include an event stream 828 and / or event updates 830 of real-time events (which may be inherently continuous or unbounded and may not have an explicit end). Examples of applications that generate continuous data may include, for example, sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0186] The communication subsystem 824 can also be configured to transmit data from computer system 800 to other computer systems or networks. Data can be transmitted in various forms, such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to computer system 800.

[0187] The computer system 800 can be of a variety of types, including handheld portable devices (e.g., iPhone). ® Cellular phone, iPad ® Computing tablets, PDAs, and wearable devices (e.g., Google Glass) ® Head-mounted displays, personal computers, workstations, mainframes, self-service kiosks, server racks, or any other data processing systems. Due to the constantly evolving nature of computers and networks, [the following is relevant:] Figure 8 The description of the computer system 800 is intended only as a concrete example. It has a higher... Figure 8 Many other configurations with more or fewer components are possible for the system depicted. Based on this disclosure and the teachings provided herein, it should be understood that there are other ways and / or methods to implement the various examples.

[0188] While specific examples have been described, various modifications, alterations, alternative constructions, and equivalents are possible. The examples are not limited to operations in a particular data processing environment but are free to operate in multiple data processing environments. Furthermore, although certain examples have been described using specific series of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be restrictive. While some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. Processes may have additional steps not included in the diagrams. Various features and aspects of the examples described above can be used individually or in combination.

[0189] Furthermore, while certain examples have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. Some examples may be implemented using only hardware, only software, or a combination thereof. The various processes described herein can be implemented on the same or different processors in any combination.

[0190] When a device, system, component, or module is described as being configured to perform certain operations or functions, this configuration can be accomplished, for example, by designing electronic circuitry to perform operations, by programming programmable electronic circuitry (such as a microprocessor) to perform operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0191] Specific details are set forth in this disclosure to provide a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides exemplary examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the foregoing description of the examples will provide those skilled in the art with enabling descriptions for implementing the various examples. Various changes can be made to the function and arrangement of the elements.

[0192] Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be restrictive. Various modifications and equivalents are within the scope of the following claims.

[0193] In the foregoing description, various aspects of this disclosure have been described with reference to specific examples; however, those skilled in the art will recognize that this disclosure is not limited thereto. Various features and aspects of the disclosure described above may be used individually or in combination. Furthermore, the examples may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0194] In the foregoing description, the methods have been described in a specific order for illustrative purposes. It should be understood that, in alternative examples, the methods may be performed in a different order than described. It should also be understood that the methods described above may be executed by hardware components or may be embodied in a sequence of machine-executable instructions that can be used to cause a machine (such as a general-purpose or special-purpose processor or logic circuit programmed with the instructions) to execute the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as CD-ROMs or other types of optical discs, floppy disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the methods may be executed by a combination of hardware and software.

[0195] When a component is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuitry or other hardware for performing the operations, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) for performing the operations, or any combination thereof.

[0196] Although illustrative examples of this application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in other ways, and the appended claims are intended to be construed as including such variations, except where limited by the prior art.

Claims

1. A computer-implemented method, comprising: The data processing system uses the training dataset or a subset of the training dataset to train the machine learning model; The data processing system determines that the training dataset should be expanded to include variants of a given entity, wherein the determination is based on the performance of the machine learning model on test data; In response to determining that the training dataset should be expanded, the data processing system selects discourses from the training dataset, which includes the entity and the context of the entity; The data processing system converts the utterance into a template list mapped to the entity, wherein the conversion includes maintaining the context and replacing the entity in each utterance with slots associated with one or more elements of the entity, and wherein the elements in the one or more elements are mapped to a list of values ​​of the entity; The data processing system generates the artificial discourse set by iteratively performing the following operations: The data processing system selects a template from the template list based on the mapping between the entity and the template list; The data processing system identifies slots and one or more associated elements within the selected template. The data processing system identifies the value list of the entity based on the mapping between the element in the one or more elements and the value list; The data processing system selects a value from the identified list of values; and The data processing system generates artificial discourse using the selected template and selected values, wherein generating the artificial discourse includes inserting the selected values ​​into the slots of the selected template; The data processing system augments the training dataset with the artificial discourse set to generate an augmented training dataset; and The data processing system trains the machine learning model using the expanded training dataset, wherein the training includes: iterative operations to find a set of parameters of the machine learning model that minimizes the loss or error function of the machine learning model, wherein each iteration includes finding the set of parameters of the machine learning model such that the value of the loss or error function using the set of parameters is less than the value of the loss or error function using another set of parameters in a previous iteration, and wherein the loss or error function is configured to measure: (i) the output inferred for the entity in each utterance or artificial utterance in the expanded training dataset, and (ii) the difference between the labels that provide basic factual information for the entity in each utterance or artificial utterance in the expanded training dataset.

2. The method of claim 1, further comprising: The data processing system tests the machine learning model using a test dataset or a validation dataset; The data processing system evaluates the performance of the machine learning model based on the tests performed on the model. as well as The data processing system identifies the entity based on the assessment.

3. The method of claim 2, further comprising: After training the machine learning model using the expanded training dataset, the data processing system retests the machine learning model using the test dataset, the validation dataset, or another test dataset or validation dataset. The data processing system evaluates the performance of the machine learning model based on the retesting of the machine learning model; The data processing system determines that the performance of the machine learning model meets predefined standards; as well as In response to the machine learning model meeting the predefined criteria, the machine learning model is deployed as part of a chatbot system.

4. The method of claim 1, wherein, The generation of the artificial discourse set further includes: The data processing system identifies the function based on the mapping between another element of the one or more elements and the function; The data processing system processes the function to generate a function value; and The data processing system uses the selected template, the selected value, and the function value to generate the artificial utterance, wherein generating the artificial utterance includes inserting the selected value and the function value into the slots of the selected template.

5. The method of claim 1, wherein, The generation of the artificial discourse set further includes: The data processing system determines that the selected template is mapped to a nested slot mapping set that includes the slot and another slot; The data processing system determines that another slot in the nested slot mapping set is mapped to another list of values; and The data processing system selects another value from the other value list based on the mapping between another slot within the selected template and the other value list. Generating the artificial utterance includes inserting the selected value into the slot of the selected template and inserting another selected value into the other slot of the selected template.

6. The method of claim 2, wherein, The training dataset is augmented with the artificial discourse set using an augmentation rate determined based on the evaluation.

7. The method of claim 1, wherein, The templates are selected based on a heuristic approach that includes the following criteria: (i) choosing between templates with multiple entity classifications and templates with a single entity classification; and (ii) weighing template variety against template quality. (iii) Categorize templates according to context; (iv) Whether the model requires a minimum number of occurrences of each template / a unique context that the user wants to teach the model; or (v) any combination of the above.

8. A system for data augmentation, comprising: One or more processors; as well as A memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions including, when executed by the one or more processors, causing the one or more processors to perform processing including the following operations: Use the training dataset or a subset of the training dataset to train the machine learning model; The training dataset is determined to be expanded to include variants of a given entity, wherein the determination is based on the performance of the machine learning model on test data; In response to determining that the training dataset should be expanded, utterances are selected from the training dataset including the entity and the context of the entity; The utterance is converted into a template list mapped to the entity, wherein the conversion includes maintaining the context and replacing the entity in each utterance with slots associated with one or more elements of the entity, and wherein the elements in the one or more elements are mapped to a list of values ​​of the entity; Generate artificial discourse sets by iteratively performing the following operations: A template is selected from the template list based on the mapping between the entity and the template list; Identify slots and one or more associated elements within the selected template; The value list of the entity is identified based on the mapping between the element in the one or more elements and the value list; Select a value from the list of identified values; and Generating artificial utterances using the selected template and selected values, wherein generating the artificial utterances includes inserting the selected values ​​into the slots of the selected template; The training dataset is augmented with the artificial utterance set to generate an augmented training dataset; and The machine learning model is trained using the expanded training dataset, wherein the training includes: iterative operations to find a set of parameters of the machine learning model that minimizes the loss or error function of the machine learning model, wherein each iteration includes finding the set of parameters of the machine learning model such that the value of the loss or error function using the set of parameters is less than the value of the loss or error function using another set of parameters in a previous iteration, and wherein the loss or error function is configured to measure: (i) the output inferred for the entity in each utterance or artificial utterance in the expanded training dataset, and (ii) the difference between the labels that provide basic factual information for the entity in each utterance or artificial utterance in the expanded training dataset.

9. The system of claim 8, wherein, The process further includes: Test the machine learning model using a test dataset or a validation dataset; The performance of the machine learning model is evaluated based on the tests conducted on the model; and The entity is identified based on the assessment.

10. The system of claim 9, wherein, The process further includes: After training the machine learning model using the expanded training dataset, the machine learning model is retested using the test dataset, the validation dataset, or another test dataset or validation dataset. The performance of the machine learning model is evaluated based on the retesting of the machine learning model; Determine that the performance of the machine learning model meets predefined criteria; and In response to the machine learning model meeting the predefined criteria, the machine learning model is deployed as part of a chatbot system.

11. The system of claim 8, wherein, The generation of the artificial discourse set further includes: The function is identified based on the mapping between another element of the one or more elements and the function; Process the function to generate a function value; and The artificial utterance is generated using the selected template, the selected value, and the function value, wherein generating the artificial utterance includes inserting the selected value and the function value into the slots of the selected template.

12. The system of claim 8, wherein, The generation of the artificial discourse set further includes: Determine that the selected template is mapped to a nested slot mapping set that includes the slot and another slot; Determine that another slot in the nested slot mapping set is mapped to another list of values; and Another value is selected from the other value list based on the mapping between another slot within the selected template and the other value list. Generating the artificial utterance includes inserting the selected value into the slot of the selected template and inserting another selected value into the other slot of the selected template.

13. The system of claim 9, wherein, The training dataset is augmented with the artificial discourse set using an augmentation rate determined based on the evaluation.

14. The system of claim 8, wherein, The templates are selected based on a heuristic approach that includes the following criteria: (i) choosing between templates with multiple entity classifications and templates with a single entity classification; and (ii) weighing template variety against template quality. (iii) Categorize templates according to context; (iv) Whether the model requires a minimum number of occurrences of each template / a unique context that the user wants to teach the model; or (v) any combination of the above.

15. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions including, when executed by the one or more processors, causing the one or more processors to perform processing comprising the following operations: Use the training dataset or a subset of the training dataset to train the machine learning model; The training dataset should be expanded to include variants of a given entity, wherein, The determination is based on the performance of the machine learning model on the test data; Select discourses from the training dataset that includes the entity and the context of the entity; The utterance is converted into a template list mapped to the entity, wherein the conversion includes maintaining the context and replacing the entity in each utterance with slots associated with one or more elements of the entity, and wherein the elements in the one or more elements are mapped to a list of values ​​of the entity; Generate artificial discourse sets by iteratively performing the following operations: A template is selected from the template list based on the mapping between the entity and the template list; Identify slots and one or more associated elements within the selected template; The value list of the entity is identified based on the mapping between the element in the one or more elements and the value list; Select a value from the list of identified values; and Generating artificial utterances using the selected template and selected values, wherein generating the artificial utterances includes inserting the selected values ​​into the slots of the selected template; The training dataset is augmented with the artificial utterance set to generate an augmented training dataset; and The machine learning model is trained using the expanded training dataset, wherein the training includes: iterative operations to find a set of parameters of the machine learning model that minimizes the loss or error function of the machine learning model, wherein each iteration includes finding the set of parameters of the machine learning model such that the value of the loss or error function using the set of parameters is less than the value of the loss or error function using another set of parameters in a previous iteration, and wherein the loss or error function is configured to measure: (i) the output inferred for the entity in each utterance or artificial utterance in the expanded training dataset, and (ii) the difference between the labels that provide basic factual information for the entity in each utterance or artificial utterance in the expanded training dataset.

16. The non-transitory computer-readable storage device as claimed in claim 15, wherein, The process further includes: Test the machine learning model using a test dataset or a validation dataset; The performance of the machine learning model is evaluated based on the tests conducted on the model; and The entity is identified based on the assessment.

17. The non-transitory computer-readable storage device as claimed in claim 16, wherein, The process further includes: After training the machine learning model using the expanded training dataset, the machine learning model is retested using the test dataset, the validation dataset, or another test dataset or validation dataset. The performance of the machine learning model is evaluated based on the retesting of the machine learning model; Determine that the performance of the machine learning model meets predefined criteria; and In response to the machine learning model meeting the predefined criteria, the machine learning model is deployed as part of a chatbot system.

18. The non-transitory computer-readable storage device as claimed in claim 15, wherein, The generation of the artificial discourse set further includes: The function is identified based on the mapping between another element of the one or more elements and the function; Process the function to generate a function value; and The artificial utterance is generated using the selected template, the selected value, and the function value, wherein generating the artificial utterance includes inserting the selected value and the function value into the slots of the selected template.

19. The non-transitory computer-readable storage device as claimed in claim 15, wherein, The generation of the artificial discourse set further includes: Determine that the selected template is mapped to a nested slot mapping set that includes the slot and another slot; Determine that another slot in the nested slot mapping set is mapped to another list of values; and Another value is selected from the other value list based on the mapping between another slot within the selected template and the other value list. Generating the artificial utterance includes inserting the selected value into the slot of the selected template and inserting another selected value into the other slot of the selected template.

20. The non-transitory computer-readable storage device as claimed in claim 16, wherein, The training dataset is augmented with the artificial discourse set using an augmentation rate determined based on the evaluation.

Images