Multi-factor modeling for natural language processing

Abstract

Techniques for systems are disclosed, including techniques for multi-factor modeling for natural language processing with chatbot systems. In embodiments, a method includes receiving a set of utterance data corresponding to a natural language based query, determining one or more intents for a chatbot, the one or more intents corresponding to possible contexts of the natural language based query and being associated with skills of the chatbot, generating one or more intent classification datasets, each intent classification dataset being associated with a probability that a natural language query corresponds to an intent of the one or more intents, generating one or more transformed datasets, each transformed dataset corresponding to a skill of the one or more skills, determining a first skill of the one or more skills based on the one or more transformed datasets, and processing the set of utterance data based on the determined first skill to resolve the natural language based query.

Description

[0001] Priority requirements

[0002] This application claims the benefit and priority of U.S. non-provisional patent application No. 17 / 578,170, filed January 18, 2022, and U.S. provisional patent application No. 63 / 139,693, filed January 20, 2021, both of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] This disclosure generally relates to chatbot systems, and more specifically to multi-factor model training for chatbot systems in natural language processing. Background Technology

[0004] Instant messaging and automated chat platforms are efficient solutions to modern customer service problems. Organizations can leverage these practices to provide timely and responsive service to their customers without investing valuable human capital in individual user inquiries. Modern automated chat platforms utilize parent "chatbots" to handle customer service requests. A parent chatbot can coordinate with multiple child chatbots, which are assigned more specialized tasks. However, selecting the right chatbot for a task is a difficult and potentially costly decision. Some chatbots are designed to handle only a specific range of inquiries, while others may be far less efficient at handling a single specialized task than others. Failure to select the most efficient chatbot to answer customer inquiries can lead to performance degradation, inefficient resource utilization, and customer frustration.

[0005] For example, some chatbots may include multiple "skill" categories that help the chatbot process and respond to queries. These queries can take the form of "discourses" in a natural language-based format, which simulates a text conversation or request. The discourses are processed to determine the most likely skills the chatbot should use to parse the discourses and respond to the queries. For example, the "order pizza" skill category might be chosen for the chatbot to respond to the natural language query "please order me a pizza".

[0006] Subtle differences in the context, cadence, spelling, tone, and / or settings of a customer's inquiry can lead to the selection of the wrong chatbot / skill for a particular task. When an organization executes hundreds or thousands of automated query responses daily, errors in chatbot selection can rapidly escalate. Simple methods of selecting chatbots (e.g., mapping words one-to-one to specific chatbots) may fail to account for proper contextual analysis or the complexities of the conversation.

[0007] To aid in skill selection, chatbots can employ machine learning models to process utterances and output the most likely skills to respond to them. Skills for answering queries can be selected based on contextual and lexical analysis of the queries provided to the organization. Therefore, one-factor selection machine learning techniques are used to train chatbots to select skills given input utterances. One-factor models directly map inputs to desired outputs, such as mapping received utterances to predicted skills for processing those utterances. While one-factor machine learning is more time-efficient in predicting skills based on input utterances, such models fail to consider the important context, rhythm, spelling, tone, or intent of the utterance. For example, the utterance “I ordered a pizza and it didn’t arrive” might be directed to the customer service chatbot skill based on the partial phrase “ordered a pizza.” Therefore, training machine learning models using less context-dependent methods (such as direct skill selection) does not provide the appropriate level of contextual learning required for efficient chatbot models. Thus, standard machine learning techniques used to directly determine the appropriate skill classification for parsing utterances are often insufficient. Summary of the Invention

[0008] Techniques for multi-factor modeling of natural language processing for training and utilizing chatbot systems are disclosed.

[0009] In one embodiment, a computer-implemented method includes: receiving, by a computing device, a set of utterance data corresponding to a natural language-based query interpretable by a chatbot; determining, by the computing device, one or more intents for the chatbot, wherein each of the one or more intents corresponds to a possible context of the natural language-based query and is associated with a skill of one or more skills of the chatbot; generating, by the computing device, one or more intent classification datasets based on the one or more intents using an intent classifier model, wherein each intent classification dataset is associated with a probability that the natural language query corresponds to an intent of one or more intents; generating, by the computing device, one or more transformed datasets based on the one or more intent classification datasets using a transform mask model, wherein each of the one or more transformed datasets corresponds to a skill of one or more skills; determining, by the computing device, a first skill of the one or more skills based on the one or more transformed datasets; and processing the set of utterance data based on the determined first skill to parse the natural language-based query.

[0010] In one embodiment, the set of speech data is received from a client device that interacts with an automated digital assistant communicating with the computer device.

[0011] In one embodiment, the transform mask model includes one or more transform values, each transform value corresponding to a ratio of at least one intention to at least one skill. In a further embodiment, the set of discourse data includes at least an indication of the ground-truth skill of the one or more skills, and the method further includes having the computing device compare the ground-truth skill with the first skill to generate one or more training loss values; and having the computing device modify the one or more transform values ​​based on the training loss values.

[0012] In one embodiment, the intent classifier model is a machine learning model configured to receive the set of utterance data as input and output the one or more intent classification datasets, the set of utterance data including at least an indication of the basic fact skills of the one or more skills, and the method further includes having the computing device compare the basic fact skills with the first skill to generate one or more training loss values; and having the computing device train the machine learning model using the one or more training loss values ​​by modifying the structure of the machine learning model based on the training loss values.

[0013] In one embodiment, the method further includes generating second skill data by the computing device using a skill classifier model, wherein the skill classifier model is a machine learning model configured to receive the set of utterance data as input and output predicted skills from the one or more skills; comparing the first skill and the second skill by the computing device to generate one or more training loss values, the one or more training loss values ​​corresponding to one or more differences between the first skill and the second skill; and training the machine learning model by the computing device using the one or more training loss values ​​by modifying the structure of the machine learning model based on the training loss values.

[0014] In one embodiment, the method further includes processing the set of discourse data based on a determined first skill to generate a natural language-based response to the natural language-based query; and sending the natural language-based response to a client device.

[0015] In one embodiment, the method further includes processing the utterance data by the computing device to generate refined utterance data in a format for input into the intent classifier model, the refined utterance data being input into the intent classifier model to result in the generation of the one or more intent classification datasets.

[0016] In one embodiment, each transformed dataset in the one or more transformed datasets includes a probability value corresponding to the likelihood that the discourse data corresponds to a skill among the one or more skills, and determining the first skill includes selecting the skill corresponding to the transformed dataset with the highest probability value in the set of one or more probability values.

[0017] In one embodiment, the first skill includes generating one or more average probability values ​​for each skill based on probability values ​​associated with the intent classification dataset, and selecting the skill associated with the highest average probability value among the one or more average probability values.

[0018] Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium and including instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0019] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more of the methods and / or one or more processes disclosed herein.

[0020] The techniques described above and below have been implemented in various ways and in various contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts are only a few of many implementations and contexts. Attached Figure Description

[0021] Figure 1 This is a simplified block diagram of a distributed environment incorporating exemplary embodiments.

[0022] Figure 2 This is a simplified block diagram of a parent robot (MB) system according to certain embodiments.

[0023] Figure 3 This is a simplified block diagram of a skill robot system according to certain embodiments.

[0024] Figure 4 This is a simplified block diagram of a computing system implementing a discourse data processing system according to certain embodiments.

[0025] Figure 5 This is a simplified block diagram of a computational system implementing a transformation model according to certain embodiments.

[0026] Figure 6A This is a simplified block diagram of a computational system implementing a transformation model according to certain embodiments.

[0027] Figure 6B This is a simplified block diagram of a computational system implementing a transformation model according to certain embodiments.

[0028] Figure 7A This is a simplified block diagram of an intent classifier configured as an artificial neural network according to certain embodiments.

[0029] Figure 7B This is a simplified block diagram of a skill classifier configured as an artificial neural network according to certain embodiments.

[0030] Figure 8 The illustrations depict processing flows for using a transformation model to select a contextual category in response to utterances, according to various embodiments.

[0031] Figure 9This is a simplified block diagram of example selection of skill robots based on intent-based classification techniques according to certain embodiments.

[0032] Figure 10 A simplified diagram of a distributed system for implementing various embodiments is depicted.

[0033] Figure 11 It is a simplified block diagram of one or more components of a system environment according to various embodiments, through which services provided by one or more components of the embodiment system are provided as cloud services.

[0034] Figure 12 The illustration shows an example computer system for implementing various embodiments. Detailed Implementation

[0035] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent, however, that the various embodiments can be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other embodiments or designs.

[0036] Some chatbot skills are designed to handle and respond to only a specific range of utterances, while others may only be able to parse specialized tasks that other skills cannot. For example, while some chatbot skills may be related to somewhat similar tasks (e.g., operating an automated system in a pizza shop), each task is suited to handling and responding to specific types of utterances (e.g., initiating a new pizza order, modifying an existing order, providing customer support, providing information related to transactions and business hours, etc.). Failure to select the most efficient skills to answer customer inquiries can lead to performance degradation, inefficient resource utilization, and customer frustration.

[0037] As mentioned above, inherent elements of natural language (such as the context, rhythm, spelling, tone, and / or settings of a customer's inquiry) can lead to the selection of the wrong chatbot / skill for a particular task. To help select the most likely skill, chatbots can employ machine learning models to process utterances and output the most likely skill to respond to them. Skills used to help answer inquiries can be selected based on contextual and lexical analysis of the inquiries provided to the digital assistant. For example, utterances are sent to a "skill classifier" to perform syntactic analysis (parse) using machine learning techniques to determine the specific skill to be used to parse the utterance. Single-factor modeling techniques for skill selection directly map the input (such as natural language utterances) to the predicted skill used to parse the utterance. Single-factor modeling is a useful technique because it allows for rapid skill prediction, which is preferred when using automated digital assistants to respond to customers. However, single-factor machine learning techniques that select skills based on utterance processing may fail to consider important context, rhythm, spelling, tone, or intent of the utterance. For example, the phrase "I ordered a pizza and it didn't arrive" might be directed to a customer service chatbot, but could be redirected to a pizza ordering chatbot based on the phrase "ordered a pizza".

[0038] Training chatbot models to select skills based on a direct mapping from utterances to possible skills may be insufficient to effectively handle the context and rhythm of utterances for sufficiently accurate skill selection. The inadequacy of single-factor skill selection is evident in both the training and prediction phases of chatbot systems. For example, training a chatbot system's machine learning model to directly select and determine skills to respond to utterances based on input utterances lacks sufficient contextual detail to build an appropriate classification model. Consequently, a poorly trained chatbot using single-factor modeling may continue to select inappropriate skills to parse utterances when interacting with real clients / customers.

[0039] This paper describes a technique for multifactor modeling in natural language processing to train and deploy chatbots / skills to process and respond to natural language-based queries. Multifactor modeling techniques improve the training and processing of input data (such as utterance data) to better predict the skills used to process utterances and train skill classifier models for utterance processing. Multifactor models can also be trained to provide more accurate output data for training single-factor models (such as skill classifiers) or for processing context-rich utterances to respond to client / customer inquiries.

[0040] The multi-factor modeling technique described in this paper utilizes the structured classification hierarchy of chatbot systems, including a set of skills for each chatbot and a set of intents for each skill. (The following text includes references.) Figures 1 to 3 The text describes skills and intentions. A multi-factor machine learning model is used to process natural language queries, outputting an intermediate context dataset representing the intent classification of the utterance. This intermediate intent classification can then be used to determine the skills used to process the utterance or to further refine the training of the skill classification for the chatbot system. Therefore, multi-factor modeling allows for the incorporation of context-determined intent data as a consideration in the final selection of chatbot skills for processing utterances. Various transformations can also be applied to the intermediate intent classification before selecting the final skills for processing the utterance. For example, a transformation “mask” can be applied to the intermediate intent classification data to map the intent classification to the corresponding skill classification. Thus, compared to single-factor modeling techniques (such as direct skill prediction based on utterance), the skill classification generated by multi-factor modeling will include more context-dependent processing of the original utterance.

[0041] The techniques described in this paper further improve the training and optimization of single-factor models that can be deployed for customer interaction. As mentioned above, single-factor models alone are insufficient to handle the low-granularity contextual speech and rhythm in natural language. Attempting to refine these single-factor models using basic supervised machine learning techniques can improve the accuracy of single-factor model skill prediction to some extent, but context-based training is not possible. Using multi-factor models to select more context-accurate skills based on a set of utterance data and comparing the context-predicted skills with those predicted by the single-factor model generates context-based loss data for retraining the single-factor model, which is impossible with supervised learning alone. Therefore, multi-factor modeling not only produces more accurate predictions when processing natural language queries, but also enables time-saving training of single-factor models to better predict skills given a set of input utterance data.

[0042] As described herein, “discourse data” can include data related to natural language-based queries provided by a client or customer for parsing by an automated chatbot system. For example, discourse could include a natural language-based query such as “I would like to order a pizza”.

[0043] As described in this article, "contextual classification" refers to chatbot "skills" associated with the categories and / or subroutines used by the chatbot to parse utterances. Contextual skill classifications can include instructions, configurations, or other functions for processing and responding to utterances in natural language-based formats. For example, a chatbot may include skills such as "pizza ordering skill." Each chatbot may include more than one "skill" for parsing and responding to utterances.

[0044] As described herein, a “context subclass” is a chatbot “intent” associated with a subclass and / or subroutine used by the chatbot to parse utterances through specific skills. Context intent classifications can include instructions, configurations, or other functions for processing and responding to utterances in natural language-based formats, depending on the skill to which the intent belongs. For example, a chatbot may include skills such as “pizza ordering skill,” and that skill may further include intents such as “Place new order from customer.” Each skill may include / correspond to more than one “intent” for parsing and responding to utterances, and each intent may correspond to a different subtask / context.

[0045] As described in this article, a “transformation model” is a set of transformed data representing the correspondence between one or more intents of a chatbot model and one or more skills. For example, a transformation model is a set of dynamically trained mapping data used to map a “context dataset” of intent scores output by a machine learning model to a “transformed” dataset of skill scores to select the skill to respond to a natural language-based query. The mapping data can be in matrix form, where the rows and columns of the matrix correspond to skills and intents, respectively, and vice versa.

[0046] Example chatbot system

[0047] A bot (also known as a skill, chatbot, conversational bot, or talkative bot) is a computer program capable of engaging in conversations with an end user. Bots typically respond to natural language messages (e.g., questions or comments) via messaging applications that use natural language messaging. Businesses can use one or more bot systems to communicate with end users through messaging applications. The messaging application (called a channel) is the end user's preferred messaging application, which is already installed and familiar to them. Therefore, end users do not need to download and install a new application to chat with the bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web applications / responsive mobile or web applications with chat functionality, or voice-based input (such as devices or applications with Siri, Cortana, Google Voice, or other voice input for interaction).

[0048] In some examples, the robot system is associated with a Uniform Resource Identifier (URI). A URI can identify the robot system using a string of characters. The URI serves as a webhook for one or more messaging application systems. A URI can include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The robot system is designed to receive messages from the messaging application system (e.g., Hypertext Transfer Protocol (HTTP) POST request messages). HTTP POST request messages relate to a URI from the messaging application system. In some embodiments, this message is different from an HTTP POST request message. For example, the robot system may receive messages from a Short Message Service (SMS). While the discussion herein may refer to the communication received by the robot system as a message, it should be understood that the message is an HTTP POST request message, an SMS message, or any other type of communication between the two systems.

[0049] End users can interact with robot systems through conversational interactions (sometimes called conversational user interface (UI)), just like human interactions. In some cases, the interaction may include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking how the robot can help. In other cases, the interaction may be transactional with, for example, a banking robot, such as transferring money from one account to another; informational with, for example, an HR robot, such as checking holiday balances; or with, for example, a retail robot, such as discussing returning purchased goods or seeking technical support.

[0050] In some embodiments, a robot system can intelligently handle end-user interactions without interaction with the robot system's administrator or developer. For example, an end-user can send one or more messages to the robot system to achieve a desired goal. Messages may include some form of content, such as text, emojis, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., using appropriate parameters for a Representational State Transition (REST) ​​call to an enterprise service) and generate a natural language response. The robot system may also prompt the end-user for additional input parameters or request additional information. In some embodiments, the robot system may also initiate communication with the end-user rather than passively responding to the end-user's utterances. Various techniques are described herein for identifying explicit calls to a robot system and determining the input of the called robot system. In some embodiments, explicit call analysis is performed by a parent robot based on the detected call name in the utterance. In response to the detection of the call name, the utterance is refined for input to the skill robot associated with the call name.

[0051] A conversation with a chatbot can follow a specific conversation flow that includes multiple states. This flow can define what will happen next based on input. In some embodiments, the chatbot system is implemented using a state machine that includes user-defined states (e.g., end-user intentions) and actions to be taken in or between states. The conversation can take different paths based on end-user input, which can affect the decisions the chatbot makes regarding the flow. For example, in each state, based on end-user input or utterances, the chatbot can determine the end-user's intention in order to determine the next appropriate action to take. As used herein and in the context of utterances, the term "intention" refers to the intention of the user providing the utterance. For example, a user might intend to involve the chatbot in a conversation about ordering pizza, so the user's intention can be expressed through the utterance "Order pizza." User intentions can relate to specific tasks that the user wants the chatbot to perform on their behalf. Therefore, utterances can be expressed as questions, commands, requests, etc., that reflect the user's intention. Intentions can include goals that the end-user wants to accomplish.

[0052] In the context of chatbot configuration, the term "intent" as used herein refers to configuration information used to map a user's utterances to specific tasks / actions or categories of tasks / actions that the chatbot can perform. To distinguish the intent of a utterance (i.e., the user's intent) from the intent of the chatbot, the latter is sometimes referred to as "bot intent" throughout this document. A bot intent can include one or more utterances associated with the intent. For example, the intent to order pizza can have various permutations of utterances expressing the expectation of placing an order for pizza. These associated utterances can be used to train the chatbot's intent classifier so that the intent classifier can subsequently determine whether an input utterance from the user matches the pizza-ordering intent. A bot intent is associated with one or more conversational flows used to initiate a conversation with the user in a certain state. For example, the first message for an pizza-ordering intent could be the question "What kind of pizza would you like?". In addition to associated utterances, bot intents can further include named entities related to the intent. For example, an pizza-ordering intent can include variables or parameters for performing the task of ordering pizza, such as topping 1, topping 2, pizza type, pizza size, number of pizzas, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0053] Figure 1This is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. Environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistant or chatbot systems. DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as... Figure 1 As shown, a user 104 of a specific business can use DABP 102 to create and deploy a digital assistant 106 for that business's users. For example, DABP 102 can be used by a bank to create one or more digital assistants for its customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, a restaurant owner (e.g., a pizzeria) can use DABP 102 to create and deploy a digital assistant that enables the restaurant's customers to order food (e.g., order pizza).

[0054] For the purposes of this disclosure, a "digital assistant" is an entity that assists a user of a digital assistant in performing various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. Digital assistants can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, and vehicles. Digital assistants are sometimes also referred to as chatbot systems. Therefore, for the purposes of this disclosure, the terms "digital assistant" and "chatbot system" are interchangeable.

[0055] A digital assistant (such as digital assistant 106 built using DABP 102) can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and receive a response 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and the response 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the task requested by the user and respond to the user with an appropriate response.

[0056] User input 110 is typically in the form of natural language and is referred to as a speech. User speech 110 can be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides the text as input to the digital assistant 106. In some embodiments, user speech 110 can be in the form of audio input or speech, such as when a user speaks or utters something as input to the digital assistant 106. Speech is typically in the form of the language spoken by user 108. For example, the speech is in English or some other language. When the speech is in speech form, the speech input is converted into a text form of speech in that particular language, and then the text speech is processed by the digital assistant 106. Various speech-to-text processing techniques are used to convert speech or audio input into text speech, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion is performed by the digital assistant 106 itself.

[0057] Speech (which may be text or speech) can be fragments, a sentence, multiple sentences, one or more words, one or more questions, combinations of the above types, etc. Digital Assistant 106 is configured to apply Natural Language Understanding (NLU) technology to speech to understand the meaning of user input. As part of the NLU processing for speech, Digital Assistant 106 is configured to perform processing for understanding the meaning of the speech, which involves identifying one or more intentions and one or more entities corresponding to the speech. After understanding the meaning of the speech, Digital Assistant 106 may perform one or more actions or operations in response to the understood meaning or intention. For the purposes of this disclosure, it is assumed that these speech are text speech directly provided by user 108 of Digital Assistant 106, or the result of converting input speech speech into text form. However, this is not intended to limit or constrain in any way.

[0058] For example, user 108 can request to order a pizza by providing a statement such as "I want to order a pizza." Upon receiving such a statement, digital assistant 106 is configured to understand the meaning of the statement and take an appropriate action. The appropriate action may involve responding to the user's input, for example, by requesting information about the type of pizza the user wishes to order, its size, and any toppings. The responses provided by digital assistant 106 can also be in natural language and are typically in the same language as the input statement. As part of generating these responses, digital assistant 106 can perform natural language generation (NLG). To enable the user to order a pizza via a conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the necessary information for the pizza order and then allow the pizza to be ordered at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0059] At the conceptual level, the digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as Natural Language Understanding (NLU), determining the action to be performed in response to the utterance, performing the action where appropriate, generating a response to be output to the user in response to the user's utterance, outputting the response to the user, etc. NLU processing may include performing syntactic analysis on the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better comprehensible form (e.g., logical form) or structure of the utterance. Generating a response may include using NLG techniques.

[0060] The NLU processing performed by the digital assistant (such as digital assistant 106) may include various NLP-related processes such as sentence parsing (e.g., tokenization, classification by inflectional form, identification of part-of-speech tags, identification of named entities in sentences, generation of dependency trees to represent sentence structure, segmentation of sentences into clauses, analysis of individual clauses, resolution of pronouns, execution of chunks, etc.). In some embodiments, the NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of the input utterance sentence may be identified by processing the sentence using a parser, part-of-speech tagger, and / or named entity recognizer. In one implementation, for the English language, a parser, part-of-speech tagger, and named entity recognizer provided by the Stanford Natural Language Processing (NLP) Group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0061] While the various examples provided in this disclosure illustrate utterances in the English language, this is merely illustrative. In some embodiments, the digital assistant 106 is also capable of processing utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems are implemented as pluggable units that can be invoked from the NLU core server using service calls. This makes NLU processing flexible and scalable for each language, including allowing different processing sequences. Language packs are provided for individual languages, wherein the language packs may register a list of subsystems that can be served from the NLU core server.

[0062] Digital assistants (such as...) can be enabled through various channels (such as, but not limited to, certain applications, social media platforms, various messaging services and applications, and other applications or channels) Figure 1The digital assistant 106 depicted is available or accessible to its user 108. A single digital assistant can be configured with several channels, allowing it to run on different services simultaneously and access them through different services.

[0063] Digital assistant or chatbot systems typically incorporate or are associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill-based bots) configured to interact with users and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc. For example, for... Figure 1 In the depicted embodiments, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the terms "skill" and "skills" are used synonymously with the terms "skill bot" and "skill bots," respectively.

[0064] Each skill associated with the digital assistant helps the user complete tasks through a conversation with the user, wherein the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may take the form of text or audio messages to the user and / or simple user interface elements (e.g., selection lists) presented to the user for selection.

[0065] There are various ways to associate or add skills or skill bots to a digital assistant. In some instances, a business can develop a skill bot and then add it to a digital assistant using DABP 102. In other instances, a skill bot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet another instance, DABP 102 provides an online digital store (called the "Skill Store") that offers multiple skills covering a wide variety of tasks. Skills offered through the Skill Store can also be exposed to various cloud services. To add a skill to a digital assistant created using DABP 102, a DABP 102 user can access the Skill Store via DABP 102, select the desired skill, and instruct the selected skill to be added to the digital assistant created using DABP 102. Skills from the Skills Store can be added to the digital assistant as is or with modifications (for example, a DABP 102 user can select and copy a specific skill bot from the Skills Store, customize or modify the selected skill bot, and then add the modified skill bot to the digital assistant created using the DABP 102).

[0066] Various architectures are used to implement digital assistant or chatbot systems. For example, in some embodiments, a parent / child (or sub-)bot paradigm or architecture is used to implement a digital assistant created and deployed using DABP 102. According to this paradigm, the digital assistant is implemented as a parent bot that interacts with one or more child bots that are skill-based bots. For example, in... Figure 1 In the depicted embodiments, digital assistant 106 includes a parent robot 114 and skill robots 116-1, 116-2, etc., which are child robots of parent robot 114. In some embodiments, digital assistant 106 itself is considered to act as the parent robot.

[0067] Digital assistants implemented using a parent-child robot architecture enable users to interact with multiple skills through a unified user interface (i.e., via the parent robot). When a user engages with the digital assistant, the parent robot receives the user input. The parent robot then performs processing to determine the meaning of the user's input. The parent robot then determines whether the task requested in the user's utterance can be handled by the parent robot itself; otherwise, it selects the appropriate skill robot to handle the user's request and routes the session to the selected skill robot. This allows users to interact with the digital assistant through a common, single interface while still having the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the parent robot can interface with skill robots with specific functionalities, such as a CRM robot for performing functions related to Customer Relationship Management (CRM), an ERP robot for performing functions related to Enterprise Resource Planning (ERP), an HCM robot for performing functions related to Human Capital Management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through the common parent robot interface, while multiple skill robots are available in the background to handle user requests.

[0068] In some embodiments, within a parent robot / child robot infrastructure, the parent robot is configured to know a list of available skill robots. The parent robot can access metadata identifying the various available skill robots, and for each skill robot, access to the skill robot's capabilities, including tasks that can be performed by the skill robot. Upon receiving a user request in the form of a utterance, the parent robot is configured to identify or predict from among the multiple available skill robots that a specific skill robot can best serve or handle the user request. The parent robot then routes the utterance (or a portion of the utterance) to that specific skill robot for further handling. Thus, control flows from the parent robot to the skill robot. The parent robot may support multiple input and output channels.

[0069] Although Figure 1 The embodiments shown illustrate a digital assistant 106 including a parent robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. A digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an implementation using a combination of software and hardware.

[0070] The DABP 102 provides the infrastructure, services, and features that enable DABP 102 users to create digital assistants (including one or more skill bots associated with the digital assistant). In some instances, skill bots can be created by cloning existing skill bots, for example, cloning skill bots provided by a skill store. As mentioned earlier, the DABP 102 provides a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP 102 users can clone skill bots from the skill store. The cloned skill bots can be modified or customized as needed. In some other instances, DABP 102 users create skill bots from scratch using the tools and services provided by the DABP 102. As mentioned earlier, the skill store or skill catalog provided by the DABP 102 can offer multiple skill bots for performing various tasks.

[0071] In some embodiments, creating or customizing a skill robot at a high level involves the following steps:

[0072] (1) Configure settings for new skill robots

[0073] (2) Configure one or more intentions for the skill robot

[0074] (3) Configure one or more entities for one or more intents.

[0075] (4) Training Skills Robot

[0076] (5) Create a dialogue flow for the skill robot

[0077] (6) Add custom parts to the skill robot as needed.

[0078] (7) Test and deploy skill robots

[0079] Each of the above steps is briefly described below.

[0080] (1) Configuring settings for new skill robots—Configure various settings for skill robots. For example, skill robot designers can specify one or more invocation names for the skill robot being created. These invocation names can then be used by the digital assistant user to explicitly invoke the skill robot. For example, the user can enter the invocation name in their speech to explicitly invoke the corresponding skill robot.

[0081] (2) Configuring one or more intents and associated example utterances for the skill robot—The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent categories or classifications that the skill robot is trained to infer in response to input utterances. Upon receiving a utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes an appropriate action to respond to the utterance based on the intent inferred for it. In some instances, the intents of the skill robot represent tasks that the skill robot can perform for a user of a digital assistant. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot could include “CheckBalance”, “TransferMoney”, “DepositCheck”, etc.

[0082] For each intent defined for a skill bot, the skill bot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are designed to represent the utterances a user would input to the skill bot in response to that intent. For example, for the CheckBalance intent, example utterances could include “What's my savings account balance?”, “How much is in my checking account?”, “How much money do I have in my account?”, etc. Thus, various permutations of typical user utterances are designated as example utterances for the intent.

[0083] These intentions and their associated example utterances are used as training data for training a skills-based robot. Various training techniques are employed. As a result of this training, a predictive model is generated, configured to take utterances as input and output the intentions inferred by the predictive model in response to those utterances. In some instances, the input utterances are provided to an intention analysis engine, which is configured to use the trained model to predict or infer the intentions of the input utterances. The skills-based robot can then take one or more actions based on the inferred intentions.

[0084] (3) Configuring Entities for One or More Intents of a Skill Robot – In some instances, additional context is needed for the skill robot to respond appropriately to user utterances. For example, there are cases where user input utterances are parsed into the same intent in the skill robot. For example, in the example above, the utterances “What's my savings account balance?” and “How much is in my checking account?” are both parsed into the same CheckBalance intent, but these utterances are different requests for different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill robot, an entity called AccountType (which defines values ​​called “checking” and “saving”) enables the skill robot to perform syntactic analysis on user requests and respond appropriately. In the example above, although these utterances are parsed into the same intent, the values ​​associated with the AccountType entity for the two utterances are different. This allows the skill robot to perform potentially different actions for the two utterances, even though they are parsed into the same intent. One or more entities can be specified for certain intents configured for the skill robot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe intent and enable skill-based robots to fulfill user requests.

[0085] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with various bots. Examples of built-in entities include, but are not limited to, entities relating to time, date, address, number, email address, duration, cycle time, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for banking skills, the AccountType entity is defined by the skill bot designer to enable various banking transactions by examining keywords entered by the user (such as current account, savings, and credit card, etc.).

[0086] (4) Training the Skill Robot—The skill robot is configured to receive user input in utterance form, process the received input through syntactic analysis or otherwise, and identify or select an intent associated with the received user input. As indicated above, this requires training the skill robot. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example utterances associated with those intents (collectively, training data), such that the skill robot can parse user input utterances into one of its configured intents. In some embodiments, the skill robot uses a predictive model trained with the training data and allows the skill robot to discern what the user says (or, in some cases, is attempting to say). DABP 102 provides a variety of different training techniques that skill robot designers can use to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion of the training data (e.g., 80%) is used to train the skill robot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as the trained skill robot) can be used to process and respond to the user's utterances. In some cases, a user's utterance is a question that requires only a single answer and no further conversation. To handle this, define a Q&A (Question and Answer) intent for the skill bot. This allows the skill bot to output a response to the user's request without having to update the dialog definition. Create the Q&A intent in a similar manner to regular intents. The dialogue flow used for Q&A intents may differ from that used for regular intents.

[0087] (5) Creating a Dialogue Flow for the Skilled Robot—The dialogue flow specified for the skilled robot describes how it reacts as it parses different intentions in response to received user input. The dialogue flow defines the actions or behaviors the skilled robot will take, such as how it responds to user utterances, prompts for user input, and returns data. The dialogue flow is like the process followed by the skilled robot. Figure 1 Skill bot designers specify the dialogue flow using a language such as Markdown. In some embodiments, a version of YAML called OBotML is used to specify the skill bot's dialogue flow. The skill bot's dialogue flow definition acts as a model of the conversation itself, a model that allows the skill bot designer to carefully orchestrate the interactions between the skill bot and the users it serves.

[0088] In some embodiments, the dialogue flow definition of a skill robot includes the following three parts:

[0089] (a) Context section

[0090] (b) Default Transformation Section

[0091] (c) State section

[0092] The context section—skill bot designers can define variables used in the session flow in the context section. Other variables named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, and user variables that enable the skill bot to recognize and save user preferences.

[0093] The Default Transition section—transitions for the skill bot can be defined either in the dialog flow state section or in the Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when no applicable transition is defined within the state or when the conditions required to trigger a state transition cannot be met. The Default Transition section can be used to define routes that allow the skill bot to handle unexpected user actions appropriately.

[0094] The state component—the dialogue flow and its associated operations—is defined as a sequence of temporary states that manage the logic within the dialogue flow. Each state node within the dialogue flow definition is named a component, which provides the functionality required at that point in the dialogue. Therefore, states are built around components. States contain component-specific properties and define transitions to other states that are triggered after a component's execution.

[0095] Use the state section to handle special situations. For example, sometimes you might want to offer users the option to temporarily leave the first skill they're engaged in to do something in a second skill within the digital assistant. For instance, if a user is busy with a conversation with the shopping skill (e.g., the user has made some purchase choices), the user might want to jump to a banking skill (e.g., the user might want to ensure he / she has enough money for the purchase) and then return to the shopping skill to complete the order. To address this, the action in the first skill can be configured to initiate an interaction with a different second skill within the same digital assistant and then return to the original stream.

[0096] (6) Adding Custom Parts to the Skill Robot—As described above, the states specified in the skill robot's dialogue flow are named with parts that provide the required functionality corresponding to those states. Parts enable the skill robot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured parts for performing a wide variety of functions. The skill robot designer can select one or more of these pre-configured parts and associate them with states in the skill robot's dialogue flow. The skill robot designer can also use the tools provided by DABP 102 to create custom parts or new parts and associate custom parts with one or more states in the skill robot's dialogue flow.

[0097] (7) Testing and Deploying Skilled Robots — DABP 102 provides several features that enable skilled robot designers to test the skilled robots they are developing. The skilled robots can then be deployed and included in digital assistants.

[0098] While the above description outlines how to create skill bots, similar techniques can be used to create digital assistants (or parent bots). At the parent bot or digital assistant level, built-in system intents are configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant (i.e., the parent bot) can handle without invoking the skill bots associated with it. Examples of system intents defined for the parent bot include: (1) Exit: applicable when the user signals that they want to exit the current session or context within the digital assistant; (2) Help: applicable when the user requests help or direction; and (3) UnresolvedIntent: applicable to user input that doesn't quite match the Exit and Help intents. The digital assistant also stores information about one or more skill bots associated with it. This information enables the parent bot to select the specific skill bot to handle utterances.

[0099] At the parent robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user's input utterance should be routed to a specific skill for handling, handled by the digital assistant or the parent robot itself according to built-in system intent, or handled as a different state within the current conversation flow.

[0100] In some embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If the invocation name is present in the user input, the invocation name is considered an explicit invocation of the skill bot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In some embodiments, if no specific invocation or explicit invocation exists, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and skill bot associated with the digital assistant. The score calculated for the skill bot or system intent indicates how likely the user input represents a task that the skill bot is configured to perform or represents a system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Any intent with a confidence score exceeding a threshold (e.g., 70%) is generally considered a candidate intent. If a specific skill bot is selected, the user's utterance is routed to that skill bot for further processing. If a system intent is selected, the parent bot itself performs one or more actions based on the selected system intent.

[0101] Figure 2 This is a simplified block diagram of a parent robot (MB) system 200 according to certain embodiments. The MB system 200 may be implemented in software only, in hardware only, or in a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, multiple intention subsystems (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data storage 250. Figure 2 The depicted MB system 200 is merely an example of the component arrangement in the parent robot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have a more... Figure 2 The systems or components shown may have more or fewer subsystems, may combine two or more subsystems, or may have different subsystem configurations or arrangements.

[0102] The preprocessing subsystem 210 receives the utterance “A” 202 from the user and processes it via a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation marks. For example, if the utterance 202 is provided as audio, the preprocessing subsystem 210 can use a speech-to-text converter (not shown) that inserts punctuation marks (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio into text.

[0103] The language detector 212 detects the language of discourse 202 based on the text of discourse 202. The way discourse 202 is processed depends on the language, because each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of discourse.

[0104] Language parser 214 performs syntactic analysis on discourse 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within discourse 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. Language parser 214 can also tokenize the linguistic units of discourse 202 (e.g., convert each word into a separate token) and categorize words by inflectional forms. Lexicons are the primary forms of a group of words as represented in a dictionary (e.g., "run" is a lexicon of run, runs, ran, running, etc.). Other types of preprocessing that language parser 214 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". Language parser 214 can also identify relationships between words in discourse 202. For example, in some embodiments, language parser 214 generates dependency trees that indicate which parts of the discourse (e.g., specific nouns) are direct objects, which parts of the discourse are prepositions, etc. The result of the processing performed by the language parser 214 forms the extracted information 205, which, together with the utterance 202 itself, is provided as input to the MIS 220.

[0105] As indicated above, utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, utterance 202 may be treated as a single unit, even if it comprises multiple sentences. However, in some embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210, to identify individual sentences among multiple sentences for use in multiple intent analysis and explicit call analysis. Generally, whether utterance 202 is processed at the level of a single sentence or as a single unit comprising multiple sentences, the results produced by MIS 220 and EIS 230 are substantially the same.

[0106] MIS 220 determines whether utterance 202 represents multiple intentions. Although MIS 220 can detect the presence of multiple intentions in utterance 202, the processing performed by MIS 220 does not involve determining whether the intention of utterance 202 matches any intention already configured for the robot. Instead, the processing to determine whether the intention of utterance 202 matches a robot intention can be performed by the intention classifier 242 of MB system 200 or the intention classifier of the skill robot (e.g., such as...). Figure 3 The process performed by MIS 220 assumes the existence of a bot that can handle utterance 202 (e.g., a specific skill bot or the parent bot itself). Therefore, the process performed by MIS 220 does not need to know which bots are in the chatbot system (e.g., the identity of skill bots registered with the parent bot) or what intents have been configured for a particular bot.

[0107] To determine that utterance 202 includes multiple intentions, MIS 220 applies one or more rules from a set of rules 252 in data storage 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include sentence patterns indicating the presence of multiple intentions. For example, a sentence pattern may include co-conjunctions connecting two parts of a sentence (e.g., conjunctions) where the two parts correspond to different intentions. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intentions. It should be noted that utterances with multiple intentions do not necessarily have different intentions (e.g., intentions involving different robots or different intentions within the same robot). Instead, utterances can have different instances of the same intention, such as “Place a pizza order using payment account X, then place pizza order using payment account Y”.

[0108] As part of determining that discourse 202 represents multiple intentions, MIS 220 also determines which parts of discourse 202 are associated with each intention. MIS 220 constructs new discourses for individual processing in relation to each intention expressed in a discourse containing multiple intentions, replacing the original discourse, for example, such as... Figure 2The original utterance 202 can be broken down into two or more separate utterances, one at a time, as depicted in the diagram. The MIS 220 uses the extracted information 205 and / or analysis of the utterance 202 itself to determine which of the two or more utterances should be processed first. For example, the MIS 220 may determine that utterance 202 contains markers indicating a specific intention that should be processed first. The newly formed utterance corresponding to that specific intention (e.g., one of utterances 206 or 208) will be sent first for further processing by the EIS 230. After the session triggered by the first utterance has ended (or been temporarily suspended), the next highest priority utterance (e.g., another utterance of utterances 206 or 208) can then be sent to the EIS 230 for processing.

[0109] EIS 230 determines whether a received utterance (e.g., utterance 206 or utterance 208) contains a call name for a skill bot. In some embodiments, each skill bot in the chatbot system is assigned a unique call name that distinguishes it from other skill bots in the chatbot system. A list of call names may be stored in data storage 250 as part of skill bot information 254. When a utterance contains a word that matches a call name, the utterance is considered an explicit call. If a bot is not explicitly called, the utterance received by EIS 230 is considered a non-explicit call utterance 234 and is fed into the parent bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to handle the utterance. In some instances, intent classifier 242 will determine that the parent bot should handle the non-explicit call utterance. In other instances, intent classifier 242 will determine the skill bot to route the utterance to for handling.

[0110] The explicit invocation functionality provided by EIS 230 offers several advantages. It reduces the amount of processing the parent robot must perform. For example, when explicit invocation is present, the parent robot may not need to perform any intent classification analysis (e.g., using intent classifier 242), or may have to perform a simplified intent classification analysis to select the skill robot. Therefore, explicit invocation analysis enables the selection of a specific skill robot without resorting to intent classification analysis.

[0111] Furthermore, there are situations where there is functional overlap between multiple skill robots. This can occur, for example, if two skill robots handle overlapping or very similar intentions. In such cases, it becomes difficult for the parent robot to identify which of the multiple skill robots to select based solely on intention classification analysis. In this scenario, explicit invocation ensures unambiguity regarding the specific skill robot to be used.

[0112] In addition to determining whether a utterance is an explicit invocation, EIS 230 is also responsible for determining whether any part of the utterance should be used as input to the explicitly invoked skill robot. Specifically, EIS 230 can determine whether a portion of the utterance is irrelevant to the invocation. EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. EIS 230 may send the irrelevant portion of the utterance to the invoked skill robot instead of the entire utterance received by EIS 230. In some instances, the input to the invoked skill robot is simply formed by removing any utterance portion associated with the invocation. For example, “I want to order pizza using Pizza Bot” can be shortened to “I want to order pizza” because “using Pizza Bot” is related to invoking the Pizza Bot but not to any processing to be performed by the Pizza Bot. In some instances, EIS 230 can reformat the portion to be sent to the invoked robot, for example, to form a complete sentence. Therefore, EIS 230 not only determines that an explicit call exists, but also, when an explicit call exists, determines what to send to the skill robot. In some instances, there may be no text to be input into the called robot. For example, if the utterance is "Pizza Bot," EIS 230 can determine that Pizza Bot is being called, but there is no text to be processed by Pizza Bot. In this scenario, EIS 230 can indicate to the skill robot caller 240 that there is no content to send.

[0113] Skill robot invoker 240 invokes skill robots in various ways. For example, skill robot invoker 240 may invoke a robot in response to receiving an instruction 235 indicating that a specific skill robot has been selected as a result of an explicit invocation. Instruction 235 may be sent by EIS 230 along with input for the explicitly invoked skill robot. In this scenario, skill robot invoker 240 relinquishes control of the session to the explicitly invoked skill robot. The explicitly invoked skill robot will determine the appropriate response to the input from EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to begin a new session in a specific state, where the initial state of the new session depends on the input sent from EIS 230.

[0114] Another way the skill robot invoker 240 can invoke a skill robot is through implicit invocation using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that a particular utterance represents a task configured to be performed by the skill robot. The intent classifier 242 is trained on different categories, one category per skill robot. For example, whenever a new skill robot is registered with a parent robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task the new skill robot can perform. The parameters produced as a result of this training (e.g., a set of parameter values ​​for a machine learning model) can be stored as part of the skill robot information 254.

[0115] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described further in detail herein. Training the machine learning model may involve taking at least a subset of example utterances associated with various skill robots as input to generate an inference as the output of the machine learning model about which robot is the correct robot for disposing of any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance is provided as basic fact information. The behavior of the machine learning model can then be adapted (e.g., via backpropagation) to minimize the discrepancy between the generated inference and the basic fact information.

[0116] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the parent robot, indicating the likelihood that the skill robot can handle a utterance (e.g., a non-explicit invocation utterance 234 received from EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score meets one or more conditions, the skill robot invoker 240 invokes the robot associated with that particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain margin. Imposing such a condition when the confidence scores of multiple skill robots all exceed the threshold confidence score value will enable routing to a particular skill robot.

[0117] After identifying a robot based on a confidence score assessment, the skill robot invoker 240 hands over processing to the identified robot. In the case of system intent, the identified robot is the parent robot. Otherwise, the identified robot is the skill robot. Further, the skill robot invoker 240 determines what to provide as input 247 to the identified robot. As described above, in the case of explicit invoking, input 247 can be based on a utterance portion unrelated to the invoking, or input 247 can be nothing (e.g., an empty string). In the case of implicit invoking, input 247 can be the entire utterance.

[0118] Data storage 250 includes one or more computing devices that store data used by various subsystems of the parent robot system 200. As explained above, data storage 250 includes rules 252 and skill robot information 254. Rule 252 includes, for example, rules for determining, by MIS 220, when a utterance represents multiple intents and how to split the utterance representing multiple intents. Rule 252 further includes rules for determining, by EIS 230, which parts of the utterance explicitly invoking the skill robot should be sent to the skill robot. Skill robot information 254 includes the invoking names of skill robots in the chatbot system, for example, a list of invoking names of all skill robots registered to a particular parent robot. Skill robot information 254 may also include information used by intent classifier 242 to determine the confidence score of each skill robot in the chatbot system, for example, parameters of a machine learning model.

[0119] Figure 3 This is a simplified block diagram of a skill robot system 300 according to certain embodiments. The skill robot system 300 is a computing system that can be implemented solely in software, solely in hardware, or a combination of hardware and software. In some embodiments, such as Figure 1 In the depicted embodiments, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0120] The skill robot system 300 includes a MIS 310, an intent classifier 320, and a session manager 330. The MIS 310 is similar to... Figure 2 MIS 220 provides similar functionality, including operablely using rules 352 in data storage 350 to determine: (1) whether the utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by MIS 310 for detecting multiple intents and for splitting utterances are the same as those applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to... Figure 1The information extracted from 205 can be generated using a language parser 214 or a language parser native to the skill robot system 300.

[0121] The intent classifier 320 can be combined with the above. Figure 4 The intent classifier 242 discussed in the embodiments is trained in a similar manner and is described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. For a specific skill robot, the machine learning model of the intent classifier 320 is trained using at least a subset of example utterances associated with that specific skill robot as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with the training utterance.

[0122] Phras 302 can be received directly from the user or provided through the parent robot. When phrasing 302 is provided through the parent robot, for example, as via... Figure 4 The results processed by MIS 220 and EIS 230 in the embodiments depicted can bypass MIS 310 to avoid repeating the processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a session after routing to a skill bot, MIS 310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance “D” 306 and utterance “E” 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0123] Intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the same skill robot system 300. As explained above, the skill robot may be configured with one or more intents, each intent including at least one example utterance associated with said intent and used to train the classifier. Figure 2In one embodiment, the intent classifier 242 of the parent robot system 200 is trained to determine confidence scores for each skill robot and confidence scores for system intents. Similarly, an intent classifier 320 can be trained to determine confidence scores for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and therefore more granular. The intent classifier 320 can access intent information 354. For each intent associated with the skill robot system 300, the intent information 354 includes a list of utterances that represent the intent and explain its meaning, and are generally associated with a task that can be performed by that intent. The intent information 354 may further include parameters generated as a result of training on this list of utterances.

[0124] The session manager 330 receives a specific intent indication 322 as the output of the intent classifier 320, which identifies the intent as the best match for the utterance input to the intent classifier 320. In some instances, the intent classifier 320 may not determine any match. For example, if the utterance relates to a system intent or the intent of a different skill robot, the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill robot system 300 may submit the utterance to a parent robot for processing, e.g., to route it to a different skill robot. However, if the intent classifier 320 successfully identifies an intent within a skill robot, the session manager 330 will initiate a session with the user.

[0125] A session initiated by session manager 330 is a session specific to an intent identified by intent classifier 320. For example, session manager 330 implements this using a state machine configured to execute a dialogue flow in response to an identified intent. The state machine may include a default starting state (e.g., when an intent is invoked without any additional input) and one or more additional states, each associated with an action to be performed by the skill robot (e.g., performing a purchase transaction) and / or a dialogue to be presented to the user (e.g., a question, a response). Thus, session manager 330 can determine an action / dialogue 335 upon receiving an indication 322 that an intent has been identified, and can determine additional actions or dialogues in response to subsequent utterances received during the session.

[0126] Data storage 350 includes one or more computing devices that store data used by various subsystems of the skill robot system 300. For example... Figure 3 As depicted, data storage 350 includes rules 352 and intent information 354. In some embodiments, data storage 350 may be integrated into the data storage of a parent robot or digital assistant, for example... Figure 2The data storage in the middle is 250.

[0127] Example Data Processing System

[0128] Figure 4 This is a simplified block diagram of a data processing system 400 according to certain embodiments. For example... Figure 4 The arrangement shown is an example embodiment of a data processing system for performing all or part of the methods described herein, but Figure 4 The system shown is not the only system capable of performing these methods. Data processing system 400 may be implemented solely as software, solely as hardware, or a combination of hardware and software. Data processing system 400 includes a communication portal 402. Communication portal 402 is any hardware, software, or instructions that allow communication between an external entity and data processing system 400. In various embodiments, communication portal 402 allows the exchange of messages between a user and a chatbot implemented by data processing system 400. For example, as part of interacting with the chatbot, a user may send an incoming utterance 406 to data processing system 400. Incoming utterance 406 is received and / or interpreted according to user communication instructions 404. User communication instructions 404 is any set of instructions or steps that allow the communication portal to receive incoming utterance 406 or send outgoing interaction 410. In various embodiments, chatbot routing instructions are implemented to send outgoing interaction 410 associated with the chatbot to the user.

[0129] Once the incoming utterance 406 is received by the communication portal, it is routed to the utterance transformer 412. The utterance transformer 412 is any entity in the data processing system 400 that can transform the incoming utterance 406 into utterance data 416. In various embodiments, the utterance transformer 412 includes a transformation instruction 414, which may contain instructions or steps for transforming the incoming utterance 406 into utterance data 416. The utterance transformer 412 transforms the incoming utterance 406, received in a natural language-based format, into input data that is received and parsed by a classifier model such as an intent classifier. For example, the incoming utterance 406 is utterance data in the same format received from a client / customer interacting with an automated digital assistant to parse utterances using a chatbot. The utterance transformer 412 utilizes the transformation instruction 414 to generate corresponding utterance data 416 in a format that can be input into the classifier model. This can be performed when the format of the incoming utterance 406 is unacceptable as input to the classifier model. In various embodiments, the utterance data 416 from the utterance transformer 412 is sent to the intent classifier 418. Intent classifier 418 may classify, quantify, or otherwise construct intent data 420 corresponding to the incoming utterance 406. In various embodiments, intent data 420 may correspond to one or more probabilities that a particular intent corresponds to the incoming utterance 406. The intent classifier is, for example, an artificial neural network employing nodes including node parameters or weights that are adjusted during a training sequence to better output one or more probabilities for the received future utterance data 416. For example, a training dataset with some “foundational facts” data is received at intent classifier 418, and the probabilities generated therefrom are compared with the “desired” intent specified by the foundational facts data to generate a loss that will be used to retrain the machine learning model.

[0130] Intent data 420 is sent from intent classifier 418 to transformation model 422. The transformation model is an entity that exists independently within data processing system 400, or is stored and implemented by a separate component such as data storage 438. Transformation model 422 transforms intent data 420 into skill data 424, which will be sent to skill store 426. In various embodiments, skill data 424 may correspond to one or more probabilities of a particular skill or sub-chatbot corresponding to an incoming utterance 406 based on intent data 420 input therein. In various embodiments, transformation model 422 performs one or more mathematical transformations on intent data 420 to create skill data 424. Skill data 424 is sent to skill store 426 after being transformed by transformation model 422. Skill store 426 is any entity within data processing system 400 that stores, implements, or otherwise manipulates skill data 424 to implement the methods described herein. Skill store 426 includes skill selection instructions 428. Skill selection instruction 426 is any set of step instructions that help select a skill or chatbot in response to incoming utterance 406. For example, skill selection instruction 428 may result in the selection of a chatbot and / or skill for interaction with the user in response to syntactic analysis of skill data 424.

[0131] In various embodiments, the discourse transformer 412 can send discourse data 416 directly to the skill store 426 without first passing the data through the intent classifier 418 and / or the transform model 422. In various further embodiments, the skill classifier 430 can acquire the discourse data 416 and create skill data according to instructions different from those of the intent classifier 418 and the transform model 422. Comparisons are used to retrain any of the intent classifier 418, the transform model 422, and / or the skill classifier 430. This represents a comparison between predicting skills based on univariate modeling techniques and predicting skills based on multivariate modeling techniques. The input of discourse data 416 to the intent classifier 418 and subsequently to the transform model 422 represents a multivariate modeling approach for predicting skills for processing discourse. Conversely, the direct input of discourse data 416 to the skill classifier 430 represents a univariate modeling approach for predicting skills for processing discourse. The difference between the skills predicted using these two methods can be compared to generate a loss, which can be used to retrain the skill classifier 430 to better predict skills in single-factor modeling scenarios (such as prediction time during client / customer interaction).

[0132] Skill repository 426 includes chatbot repository 432. Chatbot repository 432 is any repository or storage device containing data or instances of one or more chatbots implemented to handle interactions. Chatbot repository 432 may include a parent chatbot 434, which typically interacts with the user of data processing system 400. Chatbot repository 432 may further include child chatbots 436A-436N corresponding to skills within the chatbot ecosystem.

[0133] The data processing system 400 may further include a data storage 438. Data storage 438 is any storage device, memory, or medium that can store data related to the operation of the data processing system 400. For example, data storage 438 may include a model repository 446, which may store one or more transformation models for transforming intent data into skill data. Data storage 438 may further include rules 440. Rules 440 are any set of rules, steps, instructions, computer-readable code, or other logic that implement the methods described herein. Data storage 438 may further include training instructions 442. Training instructions 442 are any set of steps or instructions that result in training of transformation model 422 or any other transformation model used to transform intent data 420 into skill data 424. Data storage 438 may further include skill bot information 444. Skill bot information 444 may include any data corresponding to a chatbot, skill, or intent. For example, skill bot information 444 may include information related to sub-chatbots 436A-436N, the skills corresponding to those chatbots, and / or the intents associated with those skills and chatbots. Data storage 438 is coupled to any entity in the data processing system 400 discussed so far, including... Figure 4 The entities depicted in the text.

[0134] Example of multi-factor modeling implementation

[0135] Figure 5 This is a simplified block diagram of a computational system implementing a transformation model according to certain embodiments. Specifically, Figure 5 The diagram illustrates the input interaction flow of a data processing system that utilizes multi-factor modeling for natural language processing. In some embodiments, Figure 5 The computational system depicted is used as part of a prediction process to determine the skills to be used in response to a utterance. For example, the computational system may receive an incoming utterance 406 from a client / customer and select the skills to be used to process the incoming utterance 406 and / or generate a response to the incoming utterance 406. In other embodiments, Figure 5The computational system described in the diagram is used as part of the training process to determine the skills to be compared with basic fact skills in order to refine the depicted multi-factor model. For example, by Figure 5 The determined skill output by the computational system can be compared with the basic fact skills included in some training data, and the difference between the determined skill and the basic fact skills can be used to retrain the components of the computational system. Simultaneously, the determined intent output from the intent classifier 418 can be compared with the basic fact intents included in some training data, and the difference between the determined intent and the basic fact intents can be used to retrain the components of the intent classifier 418.

[0136] like Figure 5 The illustrated block diagram describes a process implemented by a computer device or system to generate skill data 424 from a given raw incoming utterance 406. The generated skill data 424 includes information that can be used to select skills, such as a set of skill scores based on the raw incoming utterance 406. For example, the incoming utterance 406 may include a query related to ordering pizza from a pizza shop chatbot, and the skill data 424 includes multiple skill “scores” generated as a result of a multi-factor modeling technique by an intent classifier 418 and a transformation model 422.

[0137] In embodiments involving a prediction process, the incoming utterance 406 includes data related to utterances generated by the client / customer entity, which should be processed by appropriate skills to respond to the client / customer. In this case, Figure 5 The goal of the described multi-factor modeling process is to use information included in the incoming utterance 406 to predict the skills required to respond to a natural language-based query also included in the incoming utterance 406. The predicted skills can then responsively take an action and / or generate a natural language-based outgoing response to the client / customer's query based on the incoming utterance 406.

[0138] In embodiments involving the training process, the incoming utterance 406 includes utterances and will be used for refinement. Figure 5 The basic facts and skills-related data depicted in the multi-factor model. In this context, Figure 5The goal of the described multi-factor modeling process is to attempt to predict the skill corresponding to the incoming utterance, which would otherwise be used to process the incoming utterance 406 during the prediction time. The predicted skill can be compared to the base-fact skill (the skill that should be predicted if the multi-factor model were to function perfectly) to determine the difference between the predicted skill and the base-fact skill. This difference can be used to refine elements of the multi-factor model, such as the intent classifier 418 and / or the transformation model 422. Simultaneously, the predicted intent from classifier 418 can be compared to the base-fact intent to determine the difference between the predicted intent and the base-fact intent. This difference can be used to refine elements of the multi-factor model, such as the intent classifier 418.

[0139] In some embodiments, during the prediction or training process, the highest score or highest average score of a skill in skill data 424 is used to select the predicted skill for processing incoming utterance 406. Incoming utterance 406 is first transformed in utterance transformer 412 to create utterance data 416 input to intent classifier 418. For example, a natural language query related to ordering pizza is broken down into new data, including individual words, word groups, contextual details of words, and the entire utterance. Intent classifier 418 may utilize a model (such as an artificial neural network machine learning model) to transform utterance data 416 into intent data 420 associated with one or more “scores” for each intent. For example, the generated one or more scores may be associated with the probability that incoming utterance 406 is associated with a specific intent in a superset of intents associated with one or more chatbot systems. See below for reference. Figure 7A An example of the intent classifier 418 utilizing an artificial neural network is further discussed.

[0140] The obtained intent data 420 is input to a transformation model 422 including a mask 500, which takes the intent data 420 as input to generate a set of transformed skill data 424 as output. In various embodiments, the mask 500 is a mapping dataset, such as a matrix, representing the correspondence between the dataset in the intent data 420 and the obtained skills. Therefore, the mask 500 is applied to the intent data 420 to generate skill data 424, which can be interpreted by skill selection instructions 428 to select predicted skills based on multi-factor modeling of the intent classifier 418 and the transformation model 422. In some embodiments, the mask 500 is a dynamically trained array of transformation values ​​that maps the intent scores of the intent data 420 to the skill scores of the skill data 424. The skill data 424 is utilized according to some instructions / standards to select skills for processing the incoming utterance 406. In some embodiments, the skill data 424 and / or the selected skills are used to retrain various models, such as the utterance transformer 412, the intent classifier 418, the transformation model 422, etc. (i.e., the training process). In other embodiments, skill data 424 is used to select skills for syntactic analysis and to respond to incoming utterances 406 in order to parse client / client queries (i.e., the prediction process).

[0141] More specifically, such as Figure 5 The process described involves receiving incoming utterances 406 from a user and inputting them into an utterance transformer 412. The incoming utterances 406 are in a raw format and may have one or more variables or properties. For example, the incoming utterances 406 may include parameters X corresponding to one or more utterances received from the user (e.g., queries in a natural language-based format), parameters Y(i) corresponding to multiple labels representing basic fact intentions in a superset, and parameters Y(s) corresponding to multiple labels representing basic fact skills in a superset. For example, parameters X may include one or more natural language-based queries (e.g., “I would like to order a pizza”), parameters Y(i) may include a set of intentions (e.g., pizza ordering intention, order editing intention, etc.) to be considered by a model (e.g., an input classifier), and parameters Y(s) may include a set of skills to be considered by a model (e.g., an input classifier). Figure 5 The multi-factor modeling technique presented in the text is used to predict one of the skills (e.g., the ordering skill, which can be to order pizza or edit previous pizza orders).

[0142] As part of the training or prediction process described herein, the incoming utterance data includes elements X, Y(i), and Y(s) as parameters for generating skill predictions. For example, one or more natural language queries X are natural language queries included in the incoming utterance 406, the intent superset Y(i) is the set of all intents that can be considered as part of the intent classifier 418, and the skill superset Y(s) is the set of all skills predicted based on the skill data 424 output by the multi-factor model. During the training process, X can be all natural language queries known to the basic fact skills. Parameters Y(i) and Y(s) can be predefined by the performed training process, and the predicted skill will be at least one of the skills superset Y(s); the difference between the basic fact intents and the predicted intent data 420 can be used to update the parameters of the intent classifier 418; the difference between the basic fact skills and the predicted skill data 424 (if different) can be used to update the parameters of the intent classifier 418 or the transformation model 422. During the prediction process, X can be any natural language query generated by the customer / client, and Y(i) and Y(s) can be selected based on the source of the utterance (e.g., a customer / client interacting with an automated digital assistant and chatbot implemented by a pizzeria may only be able to interact with the customer using a specific set of intents and skills).

[0143] Then, the incoming utterance 406 is processed by the utterance transformer 412 to transform the original interactive data into... Figure 5 The multi-factor model described herein can recognize or more easily read discourse data 416. For example, one or more natural language queries X can be transformed to eliminate any data other than the simple ASCII character data that constitutes the query. In another example, one or more natural language queries X can be transformed into a vectorized format, where each individual word of the query is a value in the input vector.

[0144] Then, the utterance data 416 can be sent to an intent classifier 418, which uses the utterance data 416 as input to generate intent data 420 as output. The intent data is represented as Y'(i), corresponding to a score calculated for a set of intent labels. In various embodiments, the intent classifier 418 utilizes a machine learning model, such as an artificial neural network, to acquire the utterance data 416 and output the intent data 420. In this case, Y'(i) can be a vector of values ​​or scores associated with the output nodes of the artificial neural network. In some embodiments, Y'(i) can be calculated as part of a formula using features of one or more natural language queries X. The equation used to determine Y'(i) can be:

[0145] Y'(i)=2*sigmoid(SF*-1.0*distance(W, transform(X))

[0146] Where Y'(i) is the intent data 420, which includes a set of vectorized scores for each intent in Y(i), "sigmoid()" is the sigmoid function that returns the computed sigmoid of the feature parameters, SF is a scaling factor previously determined, for example, through hyperparameter tuning or by a data scientist managing the chatbot model, distance() is a function that calculates the relative mathematical distance between two values, W is a transformation matrix that represents the centroid of each intent category and is learned during model training, and transform(X) is a transformation of one or more natural language-based queries from the discourse data 416 that can be compared with W.

[0147] The computed intent data 420 (which comprises a vectorized dataset of values ​​corresponding to each intent in Y(i)) is therefore represented by Y'(i). These values ​​can be used to determine skill data 424 via a transformation model as part of a multi-factor modeling process for determining the predictive skill for processing incoming utterances 406. In an embodiment implementing the prediction process, the output intent data 420 is then fed into a transformation model 422 to continue the prediction process.

[0148] In embodiments implementing the training process, intent data 420 can also be input into the transformation model to continue the skill prediction process described herein. Additionally, when the intent classifier 418 is a machine learning model, a supervised machine learning process can occur. For example, the incoming utterance 406 can further include a base fact intent corresponding to the desired intent selected by the intent classifier given the input utterance data 416. The difference between one or more base fact intents Y(i) and the intent data 420 represented by Y'(i) is calculated; this difference, referred to as Loss(i), represents the degree of error of the intent classifier in predicting the base fact intent. The vectorized value in Y'(i) can be compared with one or more base fact intents to determine whether the vectorized value accurately represents the weighted importance of the base fact intent. For example, the equation for calculating the intent loss is as follows:

[0149] Loss(i)=w0*cross_entropy_loss(logit_y'(i),y(i))+w l *mse_loss(y'(i),y(i))+w2*margin_loss(y'(i))+w3*threshold_loss(y'(i),y(i))

[0150] Here, Loss(i) is the total loss calculated based on the difference between the values ​​in y'(i) and the basic fact intention y(i), w0, w1, w2, and w3 are weight values ​​previously determined, for example, through hyperparameter tuning or by data scientists managing the chatbot model, cross_entropy_loss() is a function that returns the cross-entropy loss of the basic fact label y(i) and the logistic distribution of the intention data 418 represented by logit_y'(i), mse_loss() is a function that returns the mean squared error loss of the values ​​in the basic fact intention y(i) and y'(i), margin_loss() is a function that returns a margin-based loss classification for the values ​​in y'(i), and threshold_loss() is a function that returns a threshold-based loss of the values ​​in the basic fact intention y(i) and y'(i).

[0151] In some embodiments, the loss computed during the training of the intent classifier 418 can be used to retrain the intent classifier. For example, when the intent classifier 418 is a machine learning model, the loss value can be used to proportionally change the operating parameters of the intent classifier 418. In the case that the intent classifier 418 is an artificial neural network, the loss function can be used to change the weighting parameters of the nodes in the neural network to refine the model to generate more accurate intent data 420 during future processing. In some embodiments, the loss computed for the intent classifier 418 is further used as part of a general loss value, as described below.

[0152] Intent data 420, output from intent classifier 418, is input into transformation model 422 to generate skill data 424. The transformation model may include one or more masks 500, which are any data, functions, processes, or equations used to transform the intent data 420 to generate skill data 424. For example, such as... Figure 5 As depicted, mask 500 corresponds to a value matrix that maps one or more intent labels from intent data 420 to one or more skill labels. For example, assuming the chatbot system recognizes 7 intents and 3 skills, a 7×3 value mask matrix is ​​generated to map the intents to skills. The output of transformation model 422 using mask 500 is skill data 424. Skill data is represented as Y'(s). For example, the equation used to determine Y'(s) from Y'(i) is:

[0153] Y′(s)=(Y′(i)*M)

[0154] Where Y'(i) is the input intent data 420, and M is a mask matrix. In some embodiments, the mask matrix can be transformed into a format acceptable for mathematical operations with the vectorized format of Y'(i). For example, the mask can be flipped on one or more axes to be added to or multiplied with the vectorized format of Y'(i), thereby computing the vector sum or product of Y'(s). In this way, the mask matrix can be modified from a standard matrix format to map certain rows or columns of the matrix to rows or columns or Y'(i) to create a vectorized output Y'(s). Therefore, the output Y'(s) will be a skill probability vector corresponding to the probability that one or more natural language queries X correspond to any skill in a set of skills.

[0155] The skills corresponding to the vectorized format of Y'(s) can be selected as the most likely skills for processing discourse based on some instructions (such as skill selection instruction 428) to be applied to skill data 424. For example, the skill selection instruction can occur by selecting the corresponding score value in Y'(s), which is the maximum, minimum, highest sum, highest average, etc., of each vectorized score. Since some skills are associated with more than one intention, there are many ways to select skills or generate Y'(s). In some embodiments, Y'(s) is a vectorized version of skill data 424, which contains only the highest relative intention value for each corresponding skill. The following equation illustrates this embodiment:

[0156] y′(i)M[y1, y2, y3, y4, y5, y6, y7]

[0157] M = [1 1 0 0 0 0 0] [0 0 1 1 1 0 0] [0 0 0 0 0 1 1]]

[0160] Y'(s){s=0}=max(y'(i)*M[0])=max([y1*1y2*1 0 0 0 0 0])=s1

[0161] Y′(s){s1}=max(y′(i)*M[1])=max([0 0 y3*1, y4*1, y5*1 0 0])=s2

[0162] Y'(s){s=2}=max(y'(i)*M[2])=max([0 0 0 0 0 y6*1, y7*1])=s3

[0163] Y′(s) = [s1, s2, s3]

[0164] Where Y'(s) is the vectorized skill data 424, M is the mask matrix, s1-s3 are three scores corresponding to the possible skills in Y(s) that can be selected for processing the utterance, and y1-y7 are seven scores corresponding to the intentions in Y(i). As shown in the equation above, the intention score vector Y'(i) is multiplied by the mask matrix M to form a three-row vector, each row corresponding to skills s1-s3. Each vector row can include multiple values; since the mask matrix M contains only values ​​0 and 1, each value represents the probability that the intention corresponds to the incoming utterance 406, as output by the intention classifier 418. From these three vector rows, only the highest value in each vector is selected and placed in the final vector of values ​​Y'(s) corresponding to the highest value of each skill in these three row vectors.

[0165] As an example, in the three row vectors generated by matrix multiplication of Y'(i) and M, the highest value in each row could be y2, y4, and y6, with each vector corresponding to one of the three skills s1-s3. The resulting vector Y'(s) would then be [y2, y4, y6]. Using vector Y'(s), the “best” skill can then be predicted based on the values ​​in Y'(s). The predicted skill is the skill predicted to be the best skill for processing and responding to the incoming utterance 406. An example of an instruction for selecting a predicted skill could include selecting the maximum value in vector Y'(s). For example, the equation for selecting the best predicted skill for responding to the incoming utterance 406 is given by:

[0166] Y(best) = (max(Y′(s))

[0167] Here, Y(best) is the predicted skill selected based on the highest value chosen from Y'(s). It should be understood that many different combinations of selection and rules can be applied to predict the skill of responding to an incoming utterance. In some embodiments, Y'(s) is a vector comprising a value for each possible skill in Y(s), where the value is the average of each relevant intention value in Y'(i). The predicted skill will correspond to the highest average value of the skills in Y'(s). In some embodiments, Y'(s) is a vector comprising a value for each possible skill in Y(s), where the value is the sum of each relevant intention value in Y'(i). The predicted skill will correspond to the highest average value of the skills in Y'(s).

[0168] In some embodiments, given the input of skill data 424, a machine learning model (such as a skill classifier 430) can be used to select the skill vector Y'(s), instead of applying instructions to form Y'(s). See below for reference. Figure 6A An example of applying a machine learning model to form Y'(s) is described.

[0169] Similar to the training of the intent classifier 418 described above, during the training process, the difference between the basic fact skill Y(s) and the predicted skill Y'(s) is called Loss(s), representing the degree of error of the multi-factor modeling system in predicting the correct skill. The difference between the basic fact skill Y(s) associated with the incoming utterance 406 and the predicted skill Y'(s) is also called Loss(s), representing the degree of error of the transformation model in predicting the correct skill for a given action. The equation for calculating skill loss is as follows:

[0170] Loss(s)=w0_s*cross_entropy_loss(logit_y'(s),y(s))+w1_s*mse_loss(y'(s),y(s))+w2_s*margin_loss(y'(s))+w3_s*threshold_loss(y'(s),y(s))

[0171] Here, y(s) is the basic fact skill, and y'(s) is the predicted skill. This equation is similar to the Loss(i) equation above, except that it applies the difference between the predicted skill and the basic fact skill to determine Loss(s). In various embodiments, Loss(i) can be combined with Loss(s) to form the total loss for skill prediction, which considers the loss from both intention rating and skill prediction. Using the calculated Loss(i) and Loss(s), the total loss of the data processing system is calculated as follows:

[0172] Loss(total)=Loss(s)+lambda*Loss(i)

[0173] Here, lambda is a scalar representing the regularization rate to be applied to the computed intent loss. According to various embodiments, the data processing system can normalize the Loss(total) value to the training format and modify any aspect of the multi-factor model by inputting this value into the training model. For example, the Loss(total) computed during syntactic analysis of the training text data is normalized and used to modify transform model 422 and / or mask 500 to better map known intents to known skills.

[0174] The calculated loss, whether it is Loss(I), Loss(s), Loss(total), or any combination of these losses, can be used for retraining. Figure 5Elements of the multi-factor model depicted herein. In some embodiments, the intent classifier is a machine learning model including one or more operational parameters for transforming utterance data 416 into intent data 420. For example, the intent classifier may be an artificial neural network machine learning model including one or more nodes and node parameters for transforming inputs such as utterance data 416 into outputs such as intent data 420. References below Figure 7A This embodiment is described.

[0175] The parameters of the machine learning model used as part of the intent classifier 418 can be changed proportionally to the calculated loss as defined above. For example, the parameters of specific nodes in the artificial neural network can be reweighted proportionally to the calculated loss. Specifically, the parameters of certain nodes corresponding to basic factual intents / skills can be increased proportionally to the loss, and the parameters of certain nodes associated with predicted intents / skills can be decreased proportionally to the loss to refine the machine learning model.

[0176] In some embodiments, the value of mask 500 may be changed based on the calculated loss. For example, mask 500 may include values ​​between 0 and 1, and these values ​​may be changed based on the calculated loss. For example, the value in mask 500 corresponding to prediction skill may be reduced proportionally to the loss, or the value in mask 500 corresponding to basic fact skill may be increased proportionally to the loss.

[0177] Figure 6A This is a simplified block diagram of a computational system implementing a transformation model according to certain embodiments. Specifically, Figure 6A It shows Figure 5 The alternative arrangement of components of the data processing system depicted includes a skill classifier 430 as part of a multi-factor modeling system. The skill classifier 430 can be implemented as a single-factor model when receiving discourse data 416, or as part of a multi-factor model when receiving skill data 424 from the transformation model 422.

[0178] As mentioned above, univariate modeling is an efficient method for predicting skills in responding to incoming utterances, but it cannot predict skills with the same accuracy as multivariate models. Figure 6A The described discourse data can be used in a single-factor model (i.e., skill classifier 430) to generate prediction data 610 corresponding to the predicted skills used to process the incoming discourse 406. Because prediction data 610 is generated by skill classifier 430 using direct input from discourse data 416, it can be compared to… Figure 5 The multifactor modeling described in the diagram predicts skills more quickly. However, the unifactor modeling performed by the skill classifier 430 is less accurate than the contextual multifactor modeling performed by the intention classifier 418 and the transformation model 422.

[0179] During the training phase, both univariate and multivariate modeling processes were performed to retrain the skill classifier 430 to improve the univariate modeling process. For example, the incoming utterance data 406 could undergo the process described in the reference... Figure 5 The described multi-factor modeling. In some embodiments, such as Figure 6A As depicted, mask 600 is a single vector mask, rather than a matrix mask, that will be applied to intent data 420 to form skill data 424. For example, each element of the vectorized intent data can be multiplied by the corresponding element in vector mask 600, and the resulting product will be inserted into the vectorized skill data represented by Y'(I→S).

[0180] Once the value vector Y'(I→S) is computed, it is used as part of a multifactor modeling process to compute predicted skill. In some embodiments, a set of rules is applied to the vector Y'(I→S) to determine the predicted skill, similar to a reference... Figure 5 The embodiments discussed. In some embodiments, the vector Y'(I→S) is input to a skill classifier 430 to output a predicted skill. For example, the skill classifier 430 may include one or more skill prediction models, including one model for transforming the input utterance data 416 into predicted data 610 (one-factor modeling) and another model for transforming the skill value vector into predicted data 610 (as part of a multi-factor modeling process). Although the input utterance data 416 and the skill data 424 are different types of inputs, the predicted skill output as part of the predicted data 610 is the same, so the output of the one-factor modeling process can be compared with the output of the multi-factor modeling process.

[0181] As described above, the skills predicted by univariate and multivariate modeling of the same incoming utterance 406 can be used to refine the univariate model to improve accuracy while maintaining the speed of univariate prediction. Predicted skills can be compared to determine differences between the predicted skills of each individual process. These differences are used to identify the loss function or discrepancies used to retrain the univariate model (in this case, the skill classifier 430). In various embodiments, when multiple training iterations are applied to a reference... Figure 5 , Figure 6A and Figure 6B When describing a system, a cost function can be used. For example, multiple loss functions can be computed for each training iteration and used to determine the cost function. The loss function / cost function can then be used to retrain the model.

[0182] As described in this paper, the predicted data 610 generated through a one-factor modeling process is more accurate than the predicted data 610 generated when utterance data 416 is directly input into the skill classifier, due to the use of more context-accurate intent data and a trainable transformation model. However, one-factor modeling is more time-efficient because it can predict skills without the steps of multifactor modeling. Therefore, it is useful to use the output difference between multifactor and one-factor modeling to train one-factor models such as the skill classifier 430 to improve the one-factor modeling capabilities of chatbot systems.

[0183] As an example, the received utterance is "I ordered a pizza and it wasn't cooked so it was wasteful." The chatbot utilizes two skills: pizza ordering and customer service. A poorly trained univariate model (such as a skill classifier) ​​might select the pizza ordering skill based solely on the terms "ordered" and "pizza" in the utterance, when customer service is the correct skill to handle the utterance. Using multivariate modeling for more granular contextual analysis is more likely to select the correct skill. For example, customer service skills could include intents related to dissatisfied customer reviews. Such intents are highly relevant to the utterance because they contain words like "ordered," "pizza," "wasn't cooked," and "was terrible." The intents can also determine utterance-specific context, such as the fact that the utterance was written in the past tense and likely corresponds to customer service. In this example, multivariate modeling provides the correct predicted skill compared to the univariate modeling process. The differences between skill predictions can be used to retrain the skill classifier—for example, the parameters in skill classifier 430 that weight the correlation between the terms “order” and “pizza” and the pizza ordering skill can be reduced, while the parameters that weight the correlation between the terms “wasn’t” and “terrible” and the customer service skill can be increased.

[0184] Therefore, the gain / difference between predicted skills is used to retrain the skill classifier 430 to provide more context-accurate prediction data 610 when the input discourse data 416 is used as part of a univariate modeling process. Figure 6AThe depicted block diagram illustrates a hybrid contextual model that can efficiently train a skill classifier 430 and utilize it during the prediction phase. As described above, the prediction phase (or “deployment phase”) is a phase in which the chatbot using the modeling techniques described herein interacts with real clients / customers and parses real natural language queries sent to the chatbot system as part of communication with an automated digital assistant used by the client / customer. During this phase, the model is effectively “deployed” to handle and respond to the client / customer’s needs. While the model can be retrained using predictions during the training phase, the predictions used during the prediction phase will be used to respond to real-world clients / customers. The client experience is improved when queries are parsed quickly and accurately. Therefore, a context-well-trained single-factor model (such as the skill classifier 430) is a useful asset for the chatbot during the prediction phase. In some cases where the queries proposed by the client / customer are highly complex or the single-factor model is not sufficiently trained, a multi-factor model can be utilized to process the request and predict the skills to handle the query, at the cost of time efficiency, to improve contextual accuracy. For example, highly complex natural language queries can be received, and the discourse transformer may determine during the conversion to discourse data 416 that the language in the discourse may be too complex for the skill classifier to provide accurate predictions. In response, the discourse transformer can send the discourse data 416 to the intent classifier 418 to begin a multi-factor modeling process.

[0185] Figure 6B This is a simplified block diagram of a computational system implementing a transformation model according to certain embodiments. For example... Figure 6B As depicted, mask 620 is a more complex mask than mask 610, providing a more contextually accurate set of skill data 424 before predicting skills based on skill data 424. For example... Figure 6B The mask depicted may include values ​​that are not Boolean values ​​(i.e., 0 and 1). For example, mask 620 may include values ​​corresponding to values ​​modified through training, such as double-precision or floating-point numbers that frequently change based on the training input according to the method described herein. These values ​​of mask 620 can be finely tuned to influence a more accurate transformation from intent data 420 to skill data 424. As mentioned above, mask 620 contains values ​​that map intent scores in intent data 420 to corresponding skills in the skill set. In some cases, intents may not have the same weight as other intents when determining predicted skills. For example, the use of the "promotional offer" intent may be far less than that of the "pizza order" intent. As a result, the values ​​in mask 620 representing the correspondence between the promotional offer intent and the skill may be far less than the values ​​representing the correspondence between the pizza order intent and the skill.

[0186] Figure 7AThis is a simplified block diagram of an intent classifier configured as an artificial neural network according to certain embodiments. Specifically, Figure 7A An example embodiment of an intent classifier 418 configured to use an artificial neural network to take utterance vectors as input and output one or more intent scores is described. As described herein, utterance vector 700 is a form of utterance data 416 obtained from utterance transformer 412, and intent scores 740(A)-740(N) are a form of intent data 420 or context dataset output by intent classifier 418.

[0187] like Figure 7A The depicted utterance vector 700 comprises one or more sets of vector data 701(A)-701(N). The vector data may include any data obtained from utterance / natural language-based queries and presented in a format acceptable as input to the intent classifier 418. Vector data 701 may include, for example, single words, word groups, context, font, punctuation, environment, client, or any other data related to the utterance received from the client / client. The utterance vector 700 is mapped to a set of input nodes 710(A)-710(N) in the artificial neural network of the intent classifier 418. Each of these input nodes 710 is a node including node parameters / weights that determine the relative weight of each input given in the initial phase of the artificial neural network.

[0188] Input nodes 710(A)-710(N) are mapped to additional layers of the artificial neural network via one or more edges. For example, multiple edges emanating from input nodes 710(A)-710(N) lead to additional nodes in the hidden layer, where each additional node can have corresponding node parameters / weight values. For example, as... Figure 7A As depicted, input nodes 710(A)-710(N) are mapped to hidden layers 720(A) of an artificial neural network. It should be understood that an artificial neural network is a network that includes a one-to-one relationship between nodes in each successive layer or any subset of edges thereon.

[0189] The values ​​passed from input nodes 710(A)-710(N) are sequentially passed through multiple hidden layers 720(A)-720(N) to advance in the neural network. It should be understood that node parameters / weights, and in some cases the weights of edges between nodes, may be changed according to one or more machine learning training methods. Once a hidden layer 720(N) has processed the data from the previous hidden layer, the data is passed to one or more output nodes 730(A)-730(N). Output nodes 730(A)-730(N) may correspond to one or more final values ​​of possible context subclasses (e.g., the intent of a chatbot) generated as a result of the artificial neural network. Therefore, the artificial neural network of the intent classifier 418 can generate one or more probability scores associated with the probability that the input utterance vector 700 corresponds to a specific intent for processing and responding to the utterance. These probabilities are represented by intent scores 740(A)-740(N) obtained from output nodes 730(A)-730(N). See below for reference. Figure 9 Further explore the use of intent scores to select skills.

[0190] Figure 7B This is a simplified block diagram of a skill classifier configured as an artificial neural network according to certain embodiments. Specifically, Figure 7B An example embodiment of a skill classifier 430 configured to use an artificial neural network to take utterance vectors as input and output predicted skills as part of a single-factor model is described.

[0191] Figure 7B This is a simplified block diagram of a skill classifier 430 configured as an artificial neural network according to certain embodiments. Specifically, Figure 7B An example embodiment of a skill classifier 430 configured to utilize an artificial neural network to take utterance vectors as input and output predicted skills for processing utterances is depicted. For example, the skill classifier 430 may acquire utterance vector 700 in a manner similar to how the intent classifier 418 acquires utterance vectors. The skill classifier 430 may also include multiple input nodes 750(A)-750(N) and multiple hidden layers 760(A)-760(N) to advance within the neural network. It should be understood that node parameters / weights, and in some cases the weights of edges between nodes, may be changed according to one or more machine learning training methods. Finally, the hidden layers 760(N) can then send the values ​​included therein to a score selector 770. The score selector 770 may use one or more score selection criteria and the values ​​received from the hidden layers 760(N) to determine the predicted skill 780 to be output by the skill classifier. For example, the score selector 770 may receive one or more score values ​​associated with multiple possible skills from which the predicted skill 780 can be selected, and select the predicted skill 780 using criteria specifying the selection of the highest score.

[0192] Figure 8 The illustrations depict processing flows for using a transformation model to determine context classification in response to utterances, according to various embodiments. Figure 8 The described processing is implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software is stored on a non-transitory storage medium (e.g., a memory device). Figure 8 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 8 The individual processing steps, occurring in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps are performed in a different order, or some steps may be performed in parallel. In various embodiments, the steps detailed in process 800 are as described herein. Figures 1 to 7B The system and model implementation discussed.

[0193] Process 800 begins at 810, where a set of utterance data comprising one or more natural language queries is received. For example, the utterance data is received as part of incoming utterance 406 and at the communication portal 402 of the data processing system 400. After receiving incoming utterance 406, the data processing system processes the data to prepare it for input into one or more context models to predict utterances for skills used in processing and parsing the utterances. In some embodiments, the utterance data may indicate that the utterances are to be processed as part of a training phase of a chatbot model and / or a classifier model to refine the model for future prediction phases. In some embodiments, the utterance data may indicate that the utterances originate from an automated digital assistant utilized by a client or customer and should therefore be used to provide predictions of skills corresponding to the utterances to respond to client / customer queries during the prediction phase. In various embodiments, in response to reception, the utterances are transformed by a data transformer, such as utterance transformer 412, to transform the data into utterance vectors, such as utterance vector 700, for input into a classifier, such as intent classifier 418.

[0194] In step 820, one or more intentions are determined from the utterance data. These one or more intentions are possible intentions that can be used to process and generate responses to the utterance, or possible intentions associated with predictable skills that will be used to process and generate responses to the utterance. In some embodiments, the received utterance is sent to a specific part of the data processing system 400, or has associated metadata to determine a specific chatbot or area of ​​interest indicating a set of skills / intents for parsing the utterance. The metadata can be used to determine the intentions of the utterance data. For example, the utterance might be received from an automated digital assistant operating as part of a pizza delivery company's website portal, thus indicating that a chatbot trained to handle pizza orders and related inquiries should be used to process the utterance. This chatbot is associated with a set of skills and intentions, and these intentions will be determined when selecting the chatbot. In some embodiments, a superset of all possible intentions known to the data processing system 400 is selected in step 820.

[0195] In step 830, an intent classifier model is used to generate one or more intent classification datasets associated with the probabilities that the utterance data corresponds to one or more intents. Specifically, the intent set determined in step 820 is used to select a specific machine learning model (e.g., ...) corresponding to the determined intent. Figure 7A (The artificial neural network described in the text). For example, the set of intents determined in 820 will all correspond to pizza delivery utterances, and will also correspond to chatbots that utilize intent classifier models as part of a multi-factor model for parsing pizza delivery-based queries.

[0196] In an embodiment implementing an artificial neural network, the specific intent classifier model includes nodes that take vectorized utterance data obtained from the utterance as input. This vectorized utterance data is mapped to a set of input nodes (e.g., input nodes 710(A)-710(N)) to begin generating one or more output intent classification datasets. The intent classification dataset output from the intent classifier is a set of "scores," such as intent scores 740(A)-740(N), which correspond to a set of probabilities that the utterance corresponds to a specific intent among one or more intents determined in 820. This is similar to... Figure 4 The depicted process involves inputting utterance data 416 into an intent classifier 418 to output intent data 420. Therefore, the output context dataset obtained from the intent classifier model will indicate the likelihood that a particular intent most closely corresponds to the utterance to be parsed.

[0197] In 840, a masking model is used to generate one or more transformed datasets corresponding to one or more skills. Specifically, the masking model is applied to the output intent classification dataset generated in 830 to generate one or more transformed datasets based on the data in the masking model. This is similar to... Figure 4 The depiction describes inputting intent data 420 into a transformation model 422 to output skill data. The mask model can be, for example, a vectorized version of a mathematically applicable intent classification dataset. Mathematically applying such a mask matrix to the vectorized intent classification data will produce a vectorized transformed dataset. The resulting transformed dataset can correspond to a relative score or metric associated with one or more skill classifications corresponding to the intent determined in 820. For example, one or more transformed datasets can include a composite score for each skill based on the transformation of the intent classification dataset, which will be used to predict the skill to respond to the utterance using a skill selection instruction such as skill selection instruction 430. The mask to be applied can include mapping values ​​for mapping context values ​​corresponding to the intent to relevant transformed values ​​corresponding to the skill, such as mask 500.

[0198] In step 850, skills for processing the set of utterance data are determined based on one or more transformed datasets. The determined skills will be used to process and respond to the utterance data received in step 810. For example, a transformed dataset similar to skill data 424 is processed according to a set of skill selection instructions 428 or by a skill classifier 430 to determine predicted skills for processing the utterance data. In some embodiments, selection criteria corresponding to value comparisons are performed according to the skill selection instructions 428. In some embodiments, a machine learning model (e.g., a model operating as part of the skill classifier 430) can take the transformed dataset as input and output a set of skill scores that will be used to select predicted skills for processing and responding to the utterance data. See below for further details. Figure 9 An example of selecting a contextual skill category is given.

[0199] In 860, the identified skills are used to process the set of discourse data to parse one or more natural language queries. For example, the skills identified in 850 are used to process and generate responses and / or perform responsive actions based on the processing of the discourse data in order to parse natural language queries.

[0200] In various embodiments, process 800 is performed as part of the training phase of the chatbot model. For example, a loss function or value is computed during the training phase according to the methods described herein. In various embodiments, a total loss is calculated based on the computed skill label loss and intent label loss. In various further embodiments, the total loss is normalized and fed into a transformation model to modify the transformation model. In various further embodiments, the query text data is processed and transformed using the modified transformation model. In various embodiments, the determination of skill classification before and after modifying the transformation model is further compared and / or used to further train the transformation model.

[0201] In various embodiments, process 800 is performed as part of the prediction phase of a chatbot model. For example, when a chatbot is provided to a customer for predicting actions, the skill classifier may not have been properly trained using the methods described herein. In this case, the utterance data received at prediction time can be processed by an intent classifier and a transformation mask applied before skill selection to improve the prediction accuracy of skills used to respond to natural language-based queries. In some embodiments, if the contextual skill classification for processing the utterance is selected using direct input of the utterance data and the relative values ​​from the resulting transformed dataset do not meet or exceed a confidence threshold, the utterance is reprocessed at prediction time using process 800, which includes the intent classifier and mask transformation.

[0202] Figure 9 This is a simplified block diagram of example selection of skill robots based on intent-based classification techniques according to certain embodiments. Specifically, Figure 9 The output score for intent-based subclassification is described (e.g., based on...). Figure 7A The embodiments described herein and according to Figure 8 The process described in the text generates those processes to select the skills used to process natural language-based queries.

[0203] Intents 900(A)-900(G) can correspond to the context dataset, representing the probabilities obtained by matching intent with utterance, for example... Figure 7A The output intents depicted are scored from 740(A) to 740(N). Each intent corresponds to a different capability or processing context. For example, such as... Figure 9 Each intent 900(A)–900(G) depicted corresponds to a different intent task, such as ordering pizza, leaving a customer review of a pizza shop, etc. Each intent 900 corresponds to an intent score as shown in the comprehensive intent score set 910. The intent score is part of a contextual dataset representing the probability that a utterance corresponds to an intent. For example, the utterance being processed might be “I ordered a pizza and it never showed up”.

[0204] The intent score 910 is transformed according to the mask 920 to generate a transformed dataset for selecting skills to process discourse data. For example, the intent score 920 is depicted in matrix format, and the mask is depicted in matrix format; the mask is multiplied by the intent score 910 to generate a corresponding skill score. The manner in which the mask is applied and the corresponding skill is selected is determined by a score selector 930 subsystem as part of a skill selection instruction 428. For example, the score selector 930 may include an instruction to multiply the intent score 910 by the mask 920 to generate a skill score 940 for each skill. The score selector 930 may further indicate that the highest skill score 940 among the skill scores 940 will be used to select the skill 950 for processing discourse.

[0205] Skills 950(A)-950(C) are skills selected based on a skill score of 940 and according to some scoring selector criteria of 930. Each skill 950 is associated with a subset of intentions 900(A)-900(G). Figure 9 The highest score in the skill score of 940 corresponds to "Feedback Skill" 950 (B). This skill incorporates the intent of "Customer Service" 900 (E) and is the appropriate skill for handling and responding to the statement "I ordered a pizza and it never showed up".

[0206] Explanatory System

[0207] Figure 10 A simplified diagram of a distributed system 1000 is depicted. In the illustrated example, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 are configured to execute one or more applications.

[0208] In various examples, server 1012 is adapted to run one or more services or software applications implementing one or more embodiments described in this disclosure. In some examples, server 1012 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services are provided as web-based services or cloud services (such as under a Software as a Service (SaaS) model) to users of client computing devices 1002, 1004, 1006, and / or 1008. Users operating client computing devices 1002, 1004, 1006, and / or 1008 can then use one or more client applications to interact with server 1012 to utilize the services provided by these components.

[0209] exist Figure 10 In the depicted configuration, server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by server 1012. These components may include software components executed by one or more processors, hardware components, or a combination thereof. It should be understood that various different system configurations, different from distributed system 1000, are possible. Therefore, Figure 10 The example shown is an example of a distributed system for implementing the example system and is not intended to be restrictive.

[0210] Users can use client computing devices 1002, 1004, 1006, and / or 1008 to execute one or more applications, models, or chatbots, which can generate one or more events or models that can then be implemented or serviced according to the teachings of this disclosure. The client device can provide an interface that enables users of the client device to interact with it. The client device can also output information to the user via this interface. Although Figure 10 It describes only four client computing devices, but supports any number of client computing devices.

[0211] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft). Apple Or a UNIX-like operating system, Linux, or a Linux-like operating system (such as Google Chrome). TM OS), including various mobile operating systems (e.g., Microsoft Windows). Windows Android TM , Palm Portable handheld devices can include cellular phones, smartphones (e.g., ), tablet computers (e.g., Wearable devices include Google's personal digital assistants (PDAs), personal digital assistants (PDAs), and more. Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, internet-enabled gaming devices (e.g., with or without internet access). Microsoft gesture input devices Game console, Sony System, by The client device can run various applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and can use various communication protocols.

[0212] (Multiple) networks 1010 are any type of network familiar to those skilled in the art that can support data communication using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), and IPX (Internet Packet Switching). And so on. By way of example only, the (multiple) networks 1010 are Local Area Networks (LANs), Ethernet-based networks, Token Ring, Wide Area Networks (WANs), the Internet, Virtual Networks, Virtual Private Networks (VPNs), Intranets, Extranets, Public Switched Telephone Networks (PSTNs), Infrared Networks, and Wireless Networks (e.g., according to the IEEE 1002.11 protocol suite). (and / or any other wireless protocol operating on a network) and / or any combination of these networks and / or other networks.

[0213] Server 1012 comprises the following: one or more general-purpose computers, a dedicated server computer (including, by way of example, a PC (personal computer) server), Server 1012 may be a server, mid-range server, mainframe computer, rack server, server cluster, or any other suitable arrangement and / or combination thereof. Server 1012 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as one or more flexible pools of virtual storage devices virtualized to maintain the server. In various examples, server 1012 is adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0214] The computing system in server 1012 can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. Server 1012 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, etc. Servers, database servers, etc. Exemplary database servers include, but are not limited to, those that can be accessed from... Those database servers acquired through purchases by companies such as IBM.

[0215] In some implementations, server 1012 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. As an example, data feeds and / or event updates may include, but are not limited to, those provided in the original text. feed, The server 1012 may receive real-time updates from one or more third-party information sources and continuous data streams. These real-time updates may include real-time events related to sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. The server 1012 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0216] The distributed system 1000 may also include one or more data stores 1014, 1016. In some examples, these data stores are used to store data and other information. For example, one or more of data stores 1014, 1016 are used to store information (such as information related to chatbot performance or generated models) for use by the chatbot used by server 1012 in performing various functions according to various embodiments. Data stores 1014, 1016 may reside in various locations. For example, the data store used by server 1012 may be local to server 1012 or remote from server 1012 and communicate with server 1012 via a network-based or dedicated connection. Data stores 1014, 1016 are of different types. In some examples, the data store used by server 1012 is a database, such as a relational database, like one by Oracle. Databases provided by other vendors. One or more of these databases are adapted to respond to SQL-formatted commands to store, update, and retrieve data from the database.

[0217] In some examples, one or more of the data stores 1014 and 1016 can also be used by the application to store application data. The data stores used by the application are of different types, such as key-value stores, object stores, or general-purpose stores supported by the file system.

[0218] In some examples, the functionality described in this disclosure is provided as a service via a cloud environment. Figure 11 This is a simplified block diagram of a cloud-based system environment in which various services are provided as cloud services, based on certain examples. Figure 11 In the depicted example, cloud infrastructure system 1102 can provide one or more cloud services requested by a user using one or more client computing devices 1104, 1106, and 1108. Cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above with respect to server 612. The computers in cloud infrastructure system 1102 are organized as general-purpose computers, dedicated server computers, server groups, server clusters, or any other suitable arrangement and / or combination.

[0219] Multiple networks 1110 can facilitate data communication and exchange between clients 1104, 1106, and 1108 and cloud infrastructure system 1102. Multiple networks 1110 may include one or more networks. The networks may be of the same or different types. Multiple networks 1110 may support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0220] Figure 11 The example depicted is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other examples, cloud infrastructure system 1102 may have more than... Figure 11 The depicted components may have more or fewer components, may combine two or more components, or may have different component configurations or arrangements. For example, although... Figure 11 Three client computing devices are depicted, but in alternative examples, any number of client computing devices are supported.

[0221] The term cloud service is generally used to refer to services that become available to users on demand through a service provider's systems (e.g., cloud infrastructure system 1102) and via communication networks such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's systems differ from the customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. Therefore, customers can utilize cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's systems can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase the infrastructure resources to run the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle, based in Redwood Shores, California. It provides several cloud services, such as middleware services, database services, Java cloud services, and other services.

[0222] In some examples, cloud infrastructure system 1102 may provide one or more cloud services using different models, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 1102 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0223] The SaaS model enables applications or software to be delivered as a service to customers via communication networks such as the Internet, without the customer having to purchase the underlying application's hardware or software. For example, the SaaS model provides customers with access to on-demand applications hosted by a cloud infrastructure system. Oracle Examples of the SaaS services offered include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0224] The IaaS model is typically used to provide customers with infrastructure resources (such as servers, storage, hardware, and networking resources) as cloud services to offer elastic computing and storage capabilities. (Produced by Oracle) It provides a variety of IaaS services.

[0225] The PaaS model is typically used to provide a platform and environment resources as a service that enable customers to develop, run, and manage applications and services without requiring them to purchase, build, or maintain such resources. (Provided by Oracle) Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solutions services, and other services.

[0226] Cloud services are typically delivered in an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may subscribe to one or more services provided by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to provide the service requested in the customer's subscription order. For instance, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent) as described above and / or to provide services for a chatbot system as described herein. Cloud infrastructure system 1102 is configured to provide one or more cloud services.

[0227] Cloud infrastructure system 1102 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 1102 is owned by a third-party cloud service provider, and cloud services are provided to any general public customer, where the customer is an individual or enterprise. In some other examples, under a private cloud model, cloud infrastructure system 1102 operates within an organization (e.g., within a business organization), and services are provided to customers within the organization. For example, customers are various departments within the enterprise, such as human resources or payroll departments, or even individuals within the enterprise. In some other examples, under a community cloud model, cloud infrastructure system 1102 and the services provided are shared by several organizations in the relevant community. Various other models, such as hybrids of the models mentioned above, can also be used.

[0228] Client computing devices 1104, 1106, and 1108 are of different types (e.g. Figure 10 The depicted client computing devices (1002, 1004, 1006, and 1008) are capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 1102, such as requesting services provided by the cloud infrastructure system 1102. For example, a user can use the client device to request information or actions from a chatbot as described in this disclosure.

[0229] In some examples, the processing performed by cloud infrastructure system 1102 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis is performed by one or more processors, which may process data in parallel, perform simulations using the data, etc. For example, big data analysis is performed by cloud infrastructure system 1102 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0230] like Figure 11 As illustrated in the examples, cloud infrastructure system 1102 may include infrastructure resources 1130 used to facilitate the provisioning of various cloud services offered by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to service storage requested from an application is part of cloud infrastructure system 1102. In other examples, the storage virtual machine is part of a different system.

[0231] In some examples, to facilitate the efficient provisioning of these resources to support various cloud services provided by cloud infrastructure system 1102 to different customers, resources are bundled into resource groups or resource modules (also referred to as “pods”). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods are pre-provisioned for different types of cloud services. For example, a first pod is provisioned for database services, a second pod (which may include a different combination of resources than the pods in the first pod) is provisioned for Java services, and so on. For some services, the resources allocated for provisioning the service are shared among the services.

[0232] The cloud infrastructure system 1102 itself can internally use services 1132 shared by different components of the cloud infrastructure system 1102 and that facilitate the provision of services by the cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0233] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems are implemented in software or hardware, or a combination thereof. Figure 11 The depicted subsystem may include a user interface subsystem 1112 that enables users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include various interfaces such as web interface 1114, online store interface 1116 (where advertising and consumers can purchase cloud services provided by cloud infrastructure system 1102), and other interfaces 1118. For example, a customer may use a client device to request (service request 1134) one or more services provided by cloud infrastructure system 1102 using one or more of interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse cloud services provided by cloud infrastructure system 1102, and place a subscription order for one or more services provided by cloud infrastructure system 1102 that the customer wishes to subscribe to. A service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by cloud infrastructure system 1102. As part of the order, the customer may provide information identifying the service to be provided to a chatbot system and optionally provide one or more credentials for the chatbot system.

[0234] In some examples (such as) Figure 11 In the illustrated example, cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this process, OMS 1120 is configured to: create an account for a customer (if not already created); receive billing and / or billing information from the customer to be used to bill the customer for the requested services; verify customer information; place an order for the customer after verification; and schedule various workflows to prepare the order for delivery.

[0235] Once correctly verified, OMS 1120 can invoke the Order Provisioning Subsystem (OPS) 1124, which is configured as order provisioning resources (including processing resources, storage resources, and networking resources). Provisioning may include allocating resources for an order and configuring resources to facilitate the service requested by the customer's order. The manner in which resources are provisioned to an order and the type of resources provided may depend on the type of cloud service the customer has subscribed to. For example, according to a workflow, OPS 1124 is configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated to an order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users the service is to support, the duration of the requested service, etc. The allocated groups can then be customized for a specific requesting customer to provide the requested service.

[0236] In some examples, the setup phase processing described above is performed by cloud infrastructure system 1102 as part of the provisioning process. Cloud infrastructure system 1102 may generate application IDs and select storage virtual machines for applications from storage virtual machines provided by cloud infrastructure system 1102 itself or from storage virtual machines provided by other systems besides cloud infrastructure system 1102.

[0237] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting client to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) enabling the client to begin using and taking advantage of the benefits of the requested service may be sent to the client. In some examples, for the client requesting the service, the response may include a chatbot system ID generated by cloud infrastructure system 1102 and information identifying the chatbot system selected by cloud infrastructure system 1102 corresponding to the chatbot system ID.

[0238] Cloud infrastructure system 1102 can provide services to multiple customers. For each customer, cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1102 can also collect usage statistics about customers' use of subscribed services. For example, it collects statistics on storage usage, data transfer, number of users, system uptime, and system downtime. It uses this usage information to issue bills to customers, for example, on a monthly basis.

[0239] Cloud infrastructure system 1102 can provide services to multiple customers in parallel. Cloud infrastructure system 1102 can store information about these customers (potentially including proprietary information). In some examples, cloud infrastructure system 1102 includes an Identity Management Subsystem (IMS) 1128 configured to manage customer information and provide separation of the managed information so that information related to one customer cannot be accessed by another customer. IMS 1128 is configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0240] Figure 12 An example of computer system 1200 is illustrated. In some examples, computer system 1200 is used to implement any digital assistant or chatbot system within a distributed environment, as well as the various servers and computer systems described above. Figure 12 As shown, the computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 may include non-transitory computer-readable storage media, including storage medium 1222 and system memory 1210.

[0241] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. Although bus subsystem 1202 is schematically shown as a single bus, alternative examples of bus subsystems may utilize multiple buses. Bus subsystem 1202 is any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (which is implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0242] Processing subsystem 1204 controls the operation of computer system 1200 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Processors may include single-core or multi-core processors. The processing resources of computer system 1200 are organized into one or more processing units 1232, 1234, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 1204 may include one or more dedicated coprocessors such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of processing subsystem 1204 are implemented using custom circuitry such as ASICs or FPGAs.

[0243] In some examples, the processing units in processing subsystem 1204 can execute instructions stored in system memory 1210 or on computer-readable storage medium 1222. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed resides in system memory 1210 and / or on computer-readable storage medium 1222 (potentially including residing on one or more storage devices). With appropriate programming, processing subsystem 1204 can provide the various functions described above. In an instance where computer system 1200 is executing one or more virtual machines, one or more processing units are allocated to each virtual machine.

[0244] In some examples, a processing acceleration unit 1206 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 1204, thereby accelerating the overall processing performed by the computer system 1200.

[0245] I / O subsystem 1208 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 1200. Generally, the term input device is intended to include all possible types of devices and mechanisms for inputting information to computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft's [unclear - possibly a specific technology or feature], which enables users to control and interact with input devices. Motion sensor, Microsoft The 360 ​​game controller provides an interface for receiving input using gestures and spoken commands. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translating the eye gestures into the input device (such as Google). Google input Blink detector. Additionally, the user interface input device may include enabling the user to interact with a voice recognition system (e.g., ...) via voice commands. A voice recognition sensing device for interaction with navigators.

[0246] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointing sticks, game controllers and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0247] Generally, the term "output device" is intended to encompass all possible types of devices and mechanisms for outputting information from computer system 1200 to a user or other computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems are cathode ray tubes (CRTs), flat panel devices (such as those using liquid crystal displays (LCDs) or plasma displays), projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headsets, car navigation systems, plotters, voice output devices, and modems.

[0248] Storage subsystem 1218 provides a repository or data store for storing information and data used by computer system 1200. Storage subsystem 1218 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Storage subsystem 1218 may store software (e.g., programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 1204. The software is executed by one or more processing units of processing subsystem 1204. Storage subsystem 1218 may also provide certification according to the teachings of this disclosure.

[0249] The storage subsystem 1218 may include one or more non-transitory memory devices, which may include volatile memory devices and non-volatile memory devices. For example... Figure 12 As shown, the storage subsystem 1218 includes system memory 1210 and computer-readable storage medium 1222. System memory 1210 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines that help transfer information between elements within computer system 1200 during startup may typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by processing subsystem 1204. In some embodiments, system memory 1210 may include various types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0250] By using examples rather than restrictions, such as Figure 12 As depicted, system memory 1210 can load an executing application 1212 (which may include various applications such as a web browser, middleware application, relational database management system (RDBMS), etc.), program data 1214, and operating system 1216. By way of example, operating system 1216 may include various versions of Microsoft... Apple and / or Linux operating system, and various commercially available... Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS, etc.) and / or such as iOS, Telephone, OS OS OS operating system and other mobile operating systems.

[0251] Computer-readable storage medium 1222 can store programming and data structures that provide some example functionality. Computer-readable storage medium 1222 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 1200. Software (programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 1204 is stored in storage subsystem 1218. By way of example, computer-readable storage medium 1222 may include, for example, hard disk drives, disk drives, optical disc drives (such as CD ROMs, DVDs, etc.). Non-volatile memory such as disks or other optical media. Computer-readable storage medium 1222 may include, but is not limited to, [other types of storage media]. Drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 1222 may also include SSDs based on flash memory, such as flash memory-based solid-state drives (SSDs), enterprise-class flash memory drives, SSDs based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0252] In some examples, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that can be further connected to computer-readable storage medium 1222. Reader 1220 may receive data from and be configured to read data from storage devices such as disks, flash memory drives, etc.

[0253] In some examples, computer system 1200 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 1200 may provide support for executing one or more virtual machines. In some examples, computer system 1200 may execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine is allocated memory, computing (e.g., processor, core), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. Virtual machines typically run their own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 1200. Therefore, multiple operating systems may potentially be run simultaneously by computer system 1200.

[0254] The communication subsystem 1224 provides an interface to other computer systems and networks. The communication subsystem 1224 serves as an interface for receiving data from other systems and transmitting data from computer system 1200 to other systems. For example, the communication subsystem 1224 can enable computer system 1200 to establish a communication channel via the Internet to one or more client devices for receiving and sending information to client devices. For example, when computer system 1200 is used to implement... Figure 1 When the robot system 120 is described, the communication subsystem is used to communicate with a chatbot system selected for the application.

[0255] The communication subsystem 1224 may support both wired and / or wireless communication protocols. In some examples, the communication subsystem 1224 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies such as cellular telephone technology, 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 802.XX Home Standard, or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some examples, in addition to or as an alternative to a wireless interface, the communication subsystem 1224 may provide wired network connectivity (e.g., Ethernet).

[0256] The communication subsystem 1224 can receive and transmit data in various forms. In some examples, among other forms, the communication subsystem 1224 can also receive input communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. For example, the communication subsystem 1224 is configured to receive (or send) data feeds 1226 in real time from users of social media networks and / or other communication services, such as... feed, Updates, web feeds (such as rich site summary (RSS) feeds) and / or real-time updates from one or more third-party information sources.

[0257] In some examples, the communication subsystem 1224 is configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230, and is inherently continuous or unbounded without a definite end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0258] The communication subsystem 1224 can also be configured to transmit data from computer system 1200 to other computer systems or networks. Data can be transmitted in various forms, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases communicating with one or more streaming data source computers coupled to computer system 1200.

[0259] Computer system 1200 is one of many types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), wearable devices (e.g., Google...) Head-mounted displays, personal computers, workstations, mainframes, self-service kiosks, server racks, or any other data processing systems. Due to the constantly evolving nature of computers and networks, [the following is relevant:] Figure 12 The description of the computer system 1200 is intended only as a concrete example. It has a higher... Figure 12 Many other configurations with more or fewer components are possible for the system depicted. Based on this disclosure and the teachings provided herein, it should be understood that there are other ways and / or methods to implement the various examples.

[0260] While specific examples have been described, various modifications, alterations, alternative constructions, and equivalents are possible. The examples are not limited to operations in a particular data processing environment but are free to operate in multiple data processing environments. Furthermore, although certain examples have been described using specific series of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be restrictive. While some flowcharts describe operations as sequential processes, many operations are performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. Processes may have additional steps not included in the diagrams. Various features and aspects of the above examples are used individually or in combination.

[0261] Furthermore, while certain examples have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. Some examples are implemented solely with hardware, or solely with software, or using a combination thereof. The various processes described herein are implemented on the same or different processors in any combination.

[0262] When a device, system, component, or module is described as being configured to perform certain operations or functions, this configuration is accomplished, for example, by designing electronic circuitry to perform operations, by programming programmable electronic circuitry (such as a microprocessor) to perform operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0263] Specific details are set forth in this disclosure to provide a thorough understanding of the examples. However, the examples are practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides exemplary examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the foregoing description of the examples will provide those skilled in the art with enabling descriptions for implementing various examples, making various changes to the function and arrangement of the elements.

[0264] Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense. However, it will be apparent that additions, omissions, deletions, and other modifications and changes can be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be restrictive. Various modifications and equivalents are within the scope of the following claims.

[0265] In the foregoing description, various aspects of this disclosure have been described with reference to specific examples; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure are used individually or in combination. Furthermore, the examples can be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0266] In the foregoing description, the methods are described in a specific order for illustrative purposes. It should be understood that, in alternative examples, the methods may be performed in a different order than described. It should also be understood that the methods described above are executed by hardware components or embodied in a sequence of machine-executable instructions for causing a machine (such as a general-purpose or special-purpose processor or logic circuit programmed with said instructions) to perform the methods. These machine-executable instructions are stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, floppy disk, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0267] When a component is described as being configured to perform certain operations, such configuration is accomplished, for example, by designing electronic circuitry or other hardware for performing the operations, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) for performing the operations, or any combination thereof.

[0268] Although illustrative examples of this application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in other ways, and the appended claims are intended to be interpreted as including such variations, except where limited by the prior art.

Claims

1. A computer-implemented method, comprising: Receive a set of utterance data corresponding to natural language-based queries that can be interpreted by the chatbot; A set of intents is determined for the chatbot, wherein each intent in the set of intents corresponds to a possible intent of the natural language-based query and is associated with one of the chatbot's multiple skills; An intent classification dataset is generated using an intent classifier model based on intents included in the set of intents, wherein each intent classification dataset is associated with an intent score, which is related to the probability that the natural language-based query corresponds to one of the intents in the set of intents; A transformed dataset is generated by applying the mask data to the intent classification dataset using a transform mask model that utilizes the mask data. Each of the transformed datasets corresponds to one of the plurality of skills. The mask data is a matrix that includes a transform value that maps the intent score associated with each intent classification dataset to the corresponding skill among the plurality of skills. Based on the transformed dataset, a skill classifier model is used to determine the first skill among the plurality of skills; and The first skill is used to process the set of discourse data to parse the natural language-based query. The skill classifier model includes a machine learning model trained to predict skills from a plurality of skills based on received utterance data as input. At least one of the transformation values ​​in the matrix of the mask data is a non-Boolean number that can be changed during the retraining of the machine learning model.

2. The computer-implemented method as described in claim 1, wherein, The set of speech data is received from a client device that interacts with an automated digital assistant.

3. The computer-implemented method as described in claim 1, wherein, Each of the transformation values ​​corresponds to a ratio of at least one intention to at least one skill.

4. The computer-implemented method as described in claim 3, wherein: The set of discourse data includes at least the basic factual skill indications of the multiple skills; and The computer-implemented method further includes: The basic fact skill is compared with the first skill to generate one or more training loss values; and The one or more transformation values ​​are changed based on the training loss value.

5. The computer-implemented method as described in claim 1, wherein: The intent classifier model is a second machine learning model configured to receive the set of utterance data as input and output the intent classification dataset. The set of discourse data includes at least the basic factual skill indications of the multiple skills; and The computer-implemented method further includes: The basic fact skill is compared with the first skill to generate one or more training loss values; as well as The second machine learning model is trained using one or more training loss values ​​by modifying the structure of the second machine learning model based on the training loss values.

6. The computer-implemented method of claim 1, further comprising: The second skill is generated using the skill classifier model; The first skill and the second skill are compared to generate one or more training loss values, the one or more training loss values ​​corresponding to one or more differences between the first skill and the second skill; as well as The machine learning model is retrained using one or more training loss values ​​by modifying the structure of the machine learning model based on the training loss values.

7. The computer-implemented method of claim 1, further comprising: The set of discourse data is processed based on the determined first skill to generate a natural language-based response to the natural language-based query; as well as The natural language-based response is sent to the client device.

8. The computer-implemented method of claim 1, further comprising processing the set of utterance data to generate refined utterance data in a format for input into the intent classifier model, the refined utterance data being input into the intent classifier model to result in the generation of the intent classification dataset.

9. The computer-implemented method as described in claim 1, wherein: Each transformed dataset in the transformed dataset includes a probability value corresponding to the likelihood that the set of discourse data corresponds to one of the plurality of skills; and Determining the first skill involves selecting the skill corresponding to the transformed dataset with the highest probability value from a set of one or more probability values.

10. The computer-implemented method as claimed in claim 1, wherein, Determining the first skill includes generating one or more average probability values ​​for each skill based on probability values ​​associated with the intent classification dataset, and selecting the skill associated with the highest average probability value among the one or more average probability values.

11. A system including an integrated computing system, the integrated computing system comprising one or more processors and a non-transitory machine-readable storage medium thereon storing instructions, the instructions, when executed by the one or more processors, causing the one or more processors to perform a method comprising the following operations: Receive a set of utterance data corresponding to natural language-based queries that can be interpreted by the chatbot; Define a set of intents for the chatbot, wherein, Each of the set of intents corresponds to a possible intent of the natural language-based query and is associated with one of the chatbot's multiple skills; An intent classification dataset is generated using an intent classifier model based on intents included in the set of intents, wherein each intent classification dataset is associated with an intent score, which is related to the probability that the natural language-based query corresponds to one of the intents in the set of intents; A transformed dataset is generated by applying the mask data to the intent classification dataset using a transform mask model that utilizes the mask data. Each of the transformed datasets corresponds to one of the plurality of skills. The mask data is a matrix that includes a transform value that maps the intent score associated with each intent classification dataset to the corresponding skill among the plurality of skills. Based on the transformed dataset, a skill classifier model is used to determine the first skill among the plurality of skills; and The first skill is used to process the set of discourse data to parse the natural language-based query. The skill classifier model includes a machine learning model trained to predict skills from a plurality of skills based on received utterance data as input. At least one of the transformation values ​​in the matrix of the mask data is a non-Boolean number that can be changed during the retraining of the machine learning model.

12. The system of claim 11, wherein, The set of speech data is received from a client device that interacts with an automated digital assistant communicating with the system.

13. The system of claim 11, wherein, Each of the transformation values ​​corresponds to a ratio of at least one intention to at least one skill.

14. The system of claim 13, wherein: The set of discourse data includes at least the basic factual skill indications of the multiple skills; and The method further includes: The basic fact skill is compared with the first skill to generate one or more training loss values; and The one or more transformation values ​​are changed based on the training loss value.

15. The system of claim 11, wherein: The intent classifier model is a second machine learning model configured to receive the set of utterance data as input and output the intent classification dataset. The set of discourse data includes at least the basic factual skill indications of the multiple skills; and The method further includes: The basic fact skill is compared with the first skill to generate one or more training loss values; as well as The second machine learning model is retrained using one or more training loss values ​​by modifying the structure of the second machine learning model based on the training loss values.

16. The system of claim 11, wherein the method further comprises: The second skill is generated using the skill classifier model; The first skill and the second skill are compared to generate one or more training loss values, the one or more training loss values ​​corresponding to one or more differences between the first skill and the second skill; as well as The machine learning model is retrained using one or more training loss values ​​by modifying the structure of the machine learning model based on the training loss values.

17. A non-transitory machine-readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform a method comprising the following operations: Receive a set of utterance data corresponding to natural language-based queries that can be interpreted by the chatbot; Define a set of intents for the chatbot, wherein, Each of the set of intents corresponds to a possible intent of the natural language-based query and is associated with one of the chatbot's multiple skills; An intent classification dataset is generated using an intent classifier model based on intents included in the set of intents, wherein each intent classification dataset is associated with an intent score, which is related to the probability that the natural language-based query corresponds to one of the intents in the set of intents; A transformed dataset is generated by applying the mask data to the intent classification dataset using a transform mask model that utilizes the mask data. Each of the transformed datasets corresponds to one of the plurality of skills. The mask data is a matrix that includes a transform value that maps the intent score associated with each intent classification dataset to the corresponding skill among the plurality of skills. Based on the transformed dataset, a skill classifier model is used to determine the first skill among the plurality of skills; and The first skill is used to process the set of discourse data to parse the natural language-based query. The skill classifier model includes a machine learning model trained to predict skills from a plurality of skills based on received utterance data as input. At least one of the transformation values ​​in the matrix of the mask data is a non-Boolean number that can be changed during the retraining of the machine learning model.

18. The non-transitory machine-readable storage medium of claim 17, wherein, The method further includes: Based on the determined first skill, the set of discourse data is processed to generate a natural language-based response to the natural language-based query; and The natural language-based response is sent to the client device.

19. The non-transitory machine-readable storage medium of claim 17, wherein, The method further includes: The set of utterance data is processed to generate refined utterance data in a format for input into the intent classifier model, the refined utterance data being input into the intent classifier model to result in the generation of the intent classification dataset.

20. The non-transitory machine-readable storage medium of claim 17, wherein: Each transformed dataset in the transformed dataset includes a probability value corresponding to the likelihood that the discourse data corresponds to one of the plurality of skills; as well as Determining the first skill involves selecting the skill corresponding to the transformed dataset with the highest probability value from a set of one or more probability values.

Images