Method, memory, system for training a directed acyclic graph-based framework of models

Abstract

The present disclosure relates to a directed acyclic graph (DAG)-based framework for training models. The present disclosure relates to techniques for chatbots, and more specifically, techniques for building and training models using a DAG-based framework. In one particular aspect, a computer-implemented method is provided that includes generating, by a DAG-based framework, a first model and a second model, executing the first model for a chatbot in run-time and the second model in design-time, collecting attributes for intent classification associated with a set of utterances while the chatbot runs the first model and the second model, evaluating performance of the first model and the second model based on analysis of the attributes for intent classification using one or more metrics, determining whether the second model has improved performance compared to the first model, and based on the performance determination, executing the first model or the second model for the chatbot in run-time.

Description

[0001] This application is a divisional application of Chinese invention patent application filed on April 23, 2020, with application number 2020103263273 and invention title "Framework Based on Directed Acyclic Graph for Training Model". Technical Field

[0002] This disclosure generally relates to training chatbots, and more specifically, to techniques for building and training chatbot models using a framework based on directed acyclic graphs (DAGs). Background Technology

[0003] To achieve immediate responses, many users around the world utilize instant messaging or chat platforms. Organizations frequently use these platforms to engage in real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots or bots have begun to be developed to simulate conversations with end users, especially via the Internet. End users can interact with bots through messaging applications they have already installed and are using. Intelligent bots (typically powered by artificial intelligence (AI)) can communicate more intelligently and context-sensitively in real-time conversations, thus allowing for more natural conversations between bots and end users to improve the conversational experience. Instead of end users learning a fixed set of keywords or commands to know how to respond, intelligent bots can understand the end user's intent based on natural language utterances and respond accordingly. Summary of the Invention

[0004] Techniques (e.g., a method, system, or non-transitory computer-readable medium storing code or instructions executable by one or more processors) are provided for building and training models for chatbots using a DAG-based framework.

[0005] In various embodiments, a computer-implemented method is provided, the method comprising: generating a first model and a second model via a directed acyclic graph (DAG)-based framework of an integrated computing system, wherein the first model is a pipeline for performing a first set of tasks for performing one or more operations associated with a chatbot and the second model is a pipeline for performing the one or more operations associated with the chatbot; executing the first model for the chatbot at runtime and the second model for the chatbot at design time via the DAG-based framework of the integrated computing system; collecting one or more attributes for intent classification associated with a set of utterances via an event collector of the integrated computing system while the chatbot is running the first model and the second model; evaluating the performance of the first model and the second model based on the analysis of the one or more attributes for intent classification via an analytics engine of the integrated computing system using one or more metrics; determining, based on the evaluation, that the performance of the second model is improved compared to the performance of the first model via the analytics engine; and executing the second model for the chatbot at runtime via the DAG-based framework of the integrated computing system.

[0006] In some embodiments, the method further includes: graphically displaying the pipeline of the first model on a GUI; receiving user selections for one or more user-selectable tasks through the GUI; and graphically displaying the first set of tasks having the one or more user-selectable tasks in the pipeline on the GUI based on the user selections.

[0007] In some embodiments, the method further includes: graphically displaying the pipeline of the second model on a GUI; receiving user selections for one or more user-selectable tasks through the GUI; and graphically displaying, based on the user selections, the second set of tasks having the one or more user-selectable tasks in the pipeline on the GUI.

[0008] In some embodiments, the method further includes: receiving user input via one or more user options; and training the first model and the second model based on the user input, wherein the user input is a set of utterances that the user considers to trigger an intent.

[0009] In some embodiments, the first model, running during runtime, executes on a dataset to generate outputs used by the chatbot in downstream processes, wherein the downstream processes include providing dialogue or taking action based on the intent classification, and wherein the second model, running in the background during design time, executes on the same dataset to generate different outputs that are not used by the chatbot in the downstream processes.

[0010] In some embodiments, the first set of tasks differs from the second set of tasks, and the difference lies in the addition or reduction of at least one task, the replacement of at least one task, the order in which at least one task is processed, or a combination thereof.

[0011] In some embodiments, executing the first model and the second model includes obtaining a dataset containing the set of utterances from one or more channels or obtaining the dataset containing the set of utterances from a database, and parsing intents based on the set of utterances using the first model and the second model.

[0012] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods disclosed herein.

[0013] In various embodiments, a computer program product is provided, which is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0014] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods disclosed herein and / or some or all of the processes disclosed herein. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein and / or some or all of the processes disclosed herein.

[0015] The techniques described above and below can be implemented in various ways and in various contexts. Several example implementations and contexts are provided below with reference to the accompanying drawings, as described in more detail below. However, the following implementations and contexts are only a few of many implementations and contexts. Attached Figure Description

[0016] Figure 1 Simplified block diagrams of distributed environments according to various embodiments are depicted.

[0017] Figure 2 An integrated system comprising a robot system and a robot analysis system for monitoring, analyzing, visualizing and improving the performance of the robot system, according to various embodiments, is described.

[0018] Figure 3 Production lines according to various embodiments are described.

[0019] Figure 4 Ineffective pipelines according to various embodiments are described.

[0020] Figure 5 The name checking process for nodes is described according to various embodiments.

[0021] Figure 6 Complex pipelines according to various embodiments are described.

[0022] Figure 7 The illustrations depict the process flow for building, training, and implementing one or more models according to various embodiments.

[0023] Figure 8 A simplified diagram of a distributed system for implementing various embodiments is depicted.

[0024] Figure 9 It is a simplified block diagram of one or more components of a system environment according to various embodiments, through which services provided by one or more components of the embodiment system can be provided as cloud services.

[0025] Figure 10 An example computer system that can be used to implement various embodiments is illustrated. Detailed Implementation

[0026] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent, however, that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other embodiments or designs.

[0027] introduction

[0028] A digital assistant is an AI-driven interface that helps users complete various tasks in natural language conversations. For each digital assistant, a client can assemble one or more skills. Skills (also described in this document as chatbots, bots, or skill bots) are individual bots focused on specific types of tasks such as tracking inventory, submitting timecards, and creating expense reports. When an end user interacts with a digital assistant, the assistant evaluates the end user's input and routes the conversation to and from the appropriate chatbot. This can be done through platforms such as Facebook. ® Messenger, Skype Mobile ® Various channels, such as Messenger or Short Message Service (SMS), make digital assistants available to end users. Channels enable chat to be sent back and forth between end users and digital assistants and their various chatbots on various messaging platforms. Channels can also support user agent upgrades, event-initiated sessions, and testing.

[0029] Intents allow chatbots to understand what users want them to do. Intents comprise a sequence of typical user requests and statements, also known as utterances (e.g., getting account balance, making a purchase, etc.). As used herein, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. Intents can be created by providing a name describing a user action (e.g., ordering pizza) and compiling a set of real-life user statements or utterances typically associated with triggering the action. Because the chatbot's cognition stems from these intents, each intent can be created from a robust and varied dataset (one to two dozen utterances), allowing the chatbot to interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what a user wants when it receives messages such as "ignore this order" or "cancel delivery" (messages that mean the same thing but expressed differently). Intents and the utterances that belong to intents together constitute the training corpus for the chatbot. By training the model with the corpus, the client essentially turns the model into a reference tool for parsing end-user input into individual intents. The client can improve the chatbot's cognitive acumen through multiple rounds of intent testing and intent training.

[0030] However, building chatbots capable of determining end-user intent based on user utterances is a challenging task, partly due to the subtlety and ambiguity of natural language, as well as the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (the number of intents). Thus, improving chatbot performance and user experience may require training, monitoring, debugging, and retraining. In traditional systems, the provided training model is essentially the default model hard-coded for training and retraining the design system of the digital assistant or chatbot. For example, a first model might be provided that requires only a small training corpus, allowing matching rules to be used to develop entities, intents, and the training corpus. Once the training corpus has matured to the point where tests show very accurate intent parsing, a second model can be used, trained using machine learning based on word vectors and other text-based features, to add deeper dimensions to the chatbot's cognition. These default training models are often inflexible in terms of the training methodologies employed. Therefore, without more flexible methods for training, it can be difficult to identify the root causes of chatbot underperformance and determine how to improve it.

[0031] Therefore, different approaches are needed to address these issues. In various embodiments, frameworks based on Directed Acyclic Graphs (DAGs) (described herein as sparky frameworks or training tools) are provided to build training models for robotic systems. Essentially, all tasks and activities to be implemented in the model are arranged as a clear structure or pipeline of discrete processes occurring at a setpoint and with well-defined relationships to other tasks. If multiple tasks exist, each task has at least one definite upstream (previous) task or downstream (subsequent) task (although each task can easily have both). No task can create data that continues to reference itself (this avoids any instances of infinite loops). A model developed using the training tool can be implemented in the robotic system during runtime, while one or more other models, also developed using the same training tool, can be implemented simultaneously in the robotic system during design time (i.e., behind the scenes or transparent to the client). An analytics system integrated with the robotic system and the DAG-based framework can collect metrics during model usage and can be used to generate scores for each model, allowing for decisions about which model to use during runtime (e.g., replacing the original runtime model with a more robust or accurate model operating within the design time). For example, metrics can provide information for determining which pipeline for a task yields the most efficient model (e.g., considering accuracy, how many queries per second it can process, complexity, etc.). In some embodiments, clients can use training tools to edit / change the vectors and features of a model at a finer granularity compared to overall edits / changes to the model structure. In some embodiments, one or more graphical user interfaces (GUIs) of the analysis system and training tools can display information related to the DAG-based framework, such as available tasks or actions, training corpora, pre-configured task models or pipelines, and analyses including collected metrics. In some embodiments, the GUI can be used by clients to build or modify one or more training models using the DAG-based framework.

[0032] Robots and Analysis Systems

[0033] A bot (also known as a skill, chatbot, conversational bot, or talkative bot) is a computer program capable of engaging in conversations with an end user. Bots typically respond to natural language messages (e.g., questions or comments) via messaging applications that use natural language messaging. Businesses can use one or more bot systems to communicate with end users through messaging applications. The messaging application (which may be referred to as a channel) can be a preferred messaging application that the end user has already installed and is familiar with. Therefore, the end user does not need to download and install a new application to chat with the bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web applications, responsive mobile or web applications with chat capabilities, or voice-based input (such as devices or applications with interfaces using Siri, Cortana, Google Voice, or other voice input for interaction).

[0034] In some examples, a robot system may be associated with a Uniform Resource Identifier (URI). A URI can identify a robot system using a string of characters. A URI can be used as a webhook for one or more messaging application systems. For example, a URI may include a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The robot system may be designed to receive messages from a messaging application system (e.g., a Hypertext Transfer Protocol (HTTP) post call message). An HTTP post call message may relate to a URI from a messaging application system. In some embodiments, the message may be different from an HTTP post call message. For example, the robot system may receive a message from a Short Message Service (SMS). While the discussion herein may refer to the communication received by the robot system as a message, it should be understood that a message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0035] End users can interact with robot systems through conversational interactions (sometimes called conversational user interface (UI)), just like human interactions. In some cases, the interaction may include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking how the robot can help. In other cases, the interaction may be transactional with, for example, a banking robot, such as transferring money from one account to another; informational with, for example, an HR robot, such as checking holiday balances; or with, for example, a retail robot, such as discussing returning purchased goods or seeking technical support.

[0036] In some embodiments, a robot system can intelligently handle end-user interactions without interaction with the robot system's administrator or developer. For example, an end-user can send one or more messages to the robot system to achieve a desired goal. Messages may include some form of content, such as text, emojis, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., using appropriate parameters for a Representational State Transfer (REST) ​​call to an enterprise service) and generate a natural language response. The robot system may also prompt the end-user for additional input parameters or request additional information. In some embodiments, the robot system may also initiate communication with the end-user rather than passively responding to end-user utterances. Various techniques are described herein for identifying explicit calls to a robot system and determining inputs for the called robot system. In some embodiments, explicit call analysis is performed by the master robot based on the call name detected in the utterance. In response to the detection of the call name, the utterance can be refined for input to the skill robot associated with the call name.

[0037] A conversation with a chatbot can follow a specific conversation flow that includes multiple states. This flow can be defined based on input to determine what will happen next. In some embodiments, the chatbot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken within or between states. The conversation can take different paths based on end-user input, which can influence the chatbot's decisions regarding the flow. For example, in each state, based on end-user input or utterances, the chatbot can determine the end-user's intent in order to determine the appropriate next action to take. As used herein and in the context of utterances, the term "intent" refers to the intent of the user providing the utterance. For example, a user might intend to have a conversation with the chatbot to order pizza, so the user's intent can be expressed through the utterance "order pizza." User intents can relate to specific tasks that the user wants the chatbot to perform on their behalf. Therefore, utterances can be expressed as questions, commands, requests, etc., that reflect the user's intent. Intents can include goals that the end-user wants to accomplish.

[0038] In the context of chatbot configuration, the term "intent" as used herein refers to configuration information used to map a user's utterances to a specific task / action or a specific type of task / action that the chatbot can perform. To distinguish the intent of a utterance (i.e., the user's intent) from the intent of the chatbot, the chatbot's intent is sometimes referred to as "bot intent" in this document. A bot intent can include one or more utterances associated with the intent. For example, the intent to order pizza can have various permutations of utterances expressing the expectation of placing an order for pizza. These associated utterances can be used to train the chatbot's intent classifier so that the intent classifier can subsequently determine whether an input utterance from the user matches the pizza-ordering intent. A bot intent can be associated with one or more conversational flows used to initiate a conversation with the user in a certain state. For example, the first message for a pizza-ordering intent could be the question "What kind of pizza do you want?". In addition to associated utterances, a bot intent can further include named entities related to the intent. For example, a pizza-ordering intent can include variables or parameters for performing the task of ordering pizza, such as topping 1, topping 2, pizza type, pizza size, number of pizzas, etc. The values ​​of the entities are typically obtained through the conversation with the user.

[0039] Figure 1 This is a simplified block diagram of a distributed environment 100 incorporating an exemplary embodiment. The distributed environment 100 includes a Digital Assistant Builder Platform (DABP) 102 that enables enterprises to create and deploy digital assistants for their users. For the purposes of this disclosure, a "digital assistant" is an entity that assists a user of a digital assistant in performing various tasks through natural language conversation. Digital assistants can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. Digital assistants can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. DABP 102 can be used to create one or more digital assistants (or DAs) 106. DABP 102 can be used by multiple enterprises to create digital assistants for their users. For example, as Figure 1 As shown, a user 104 representing a specific business can use DABP 102 to create and deploy a digital assistant 106 for the user of that specific business. For example, a restaurant (e.g., a pizzeria) owner can use DABP 102 to create and deploy a digital assistant that enables the restaurant's customers to order food (e.g., order pizza).

[0040] Once the digital assistant 106 is deployed, the user 108 can use the digital assistant 106 to perform various tasks through natural language-based conversations with it. As part of the conversation, the user 108 can provide one or more user inputs 110 and receive a response 112 from the digital assistant 106. Through these conversations, the user can request one or more tasks to be performed by the digital assistant 106, and in response, the digital assistant 106 is configured to perform the user-requested tasks and respond to the user with an appropriate response.

[0041] User input 110 is in natural language and is referred to as speech. User speech can be in text form (e.g., when a user types something as input to digital assistant 106) or in audio or speech form (e.g., when a user speaks something as input to digital assistant 106). Speech is typically spoken language by user 108. When user input 110 is in speech form, the speech input is converted into text speech in that specific language, and then processed by digital assistant 106. Various speech-to-text processing techniques can be used to convert speech or audio input into text speech, which is then processed by digital assistant 106.

[0042] The text utterance input by user 108 or generated from voice input converted into text form can be a text fragment, sentence, multiple sentences, etc. Digital assistant 106 is configured to apply Natural Language Understanding (NLU) technology to the text utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, digital assistant 106 is configured to perform processing for understanding the meaning of the utterance, which involves identifying one or more intentions and one or more entities corresponding to the utterance. Once the meaning of the utterance is understood, digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intention.

[0043] For example, a user inputting 110 can request to order a pizza, such as "I want to order a pizza." The digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action, which may involve responding to the user by asking them to input the type of pizza they wish to order, the size of the pizza, any toppings, etc. The response 112 provided by the digital assistant 106 can also be in natural language form, which may involve natural language generation (NLG) processing performed by the digital assistant 106. Once the digital assistant 106 has obtained the necessary information from the user, it can then place the pizza order. The digital assistant 106 can end the conversation with the user by outputting a message indicating that the pizza has been ordered.

[0044] In some embodiments, utterances received as input by the digital assistant 106 undergo a series of processing steps or a pipeline of processing steps. These steps may include, for example, performing syntactic analysis on the utterances, understanding the meaning of the utterances, refining and reorganizing the utterances to develop a more understandable structure for the utterances, determining the actions to be performed in response to the utterances, performing the actions, generating responses to be output to the user in response to the user's utterances, and outputting the responses to the user, etc.

[0045] NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related processes such as sentence grammatical analysis (e.g., speech segmentation, classification by inflectional forms, identification of part-of-speech tags, identification of named entities in sentences, generation of dependency trees to represent sentence structure, segmentation of sentences into clauses, analysis of individual clauses, resolution of pronouns, execution of chunks, etc.). Digital assistant 106 may use an NLP engine and / or machine learning models (e.g., an intent classifier) ​​to map end-user utterances to specific intents (e.g., specific tasks / actions or specific types of tasks / actions that a chatbot can perform). For example, a machine learning-based NLP engine can learn to understand and classify natural language conversations from end-users and extract necessary information from the conversation to take precise actions, such as executing transactions or retrieving data from a backend recording system. In some embodiments, NLU processing, or portions thereof, is performed by digital assistant 106 itself. In other embodiments, digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of a sentence can be identified by processing the sentence using a parser, part-of-speech tagger, and / or named entity recognizer. In one implementation, for the English language, a parser, part-of-speech tagger, and named entity recognizer provided by the Stanford Natural Language Processing (NLP) group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0046] While the various examples provided in this disclosure illustrate utterances in the English language, this is merely illustrative. In some embodiments, the digital assistant 106 is also capable of processing utterances in languages ​​other than English. In some embodiments, the digital assistant 106 provides subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems can be implemented as pluggable units that can be invoked from the NLU core server using service calls. This makes NLU processing flexible and scalable for each language, including allowing different processing sequences. Language packs can be provided for individual speech, wherein the language packs can register a list of subsystems that can be served from the NLU core server, and can also utilize provided general subsystems if needed.

[0047] A digital assistant (such as digital assistant 106) can be made available to its users through various channels, including but not limited to certain applications, social media platforms, various messaging services and applications, and other applications or channels. A single digital assistant can be configured with several channels, allowing it to run on different services and be accessed through different services simultaneously.

[0048] Digital assistants contain or are associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill bots) designed to interact with users and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing widgets, etc. For example, for Figure 1 In the depicted embodiments, the digital assistant 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the terms "skill" and "multiple skills" are used synonymously with the terms "skilled robot" and "multiple skilled robots," respectively.

[0049] Each skill associated with the digital assistant helps the user complete tasks through a conversation with the user, wherein the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may take the form of text or audio messages to the user and / or simple user interface elements (e.g., a selection list) presented to the user for selection.

[0050] There are various ways to add skills or skill bots to a digital assistant. In some instances, a business can develop a skill bot and then add it to a digital assistant using DABP 102. In other instances, a skill bot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet another instance, DABP 102 provides an online digital store (referred to as the "Skill Store") that offers multiple skills covering a wide range of tasks. Skills offered through the Skill Store can showcase various cloud services. A user 104 of DABP 102 can access the Skill Store, select desired skills, and add the selected skills to a digital assistant created using DABP 102. Skills from the Skill Store can be added to the digital assistant as is or in a modified form (e.g., a user of DABP 102 can select and copy a specific skill bot offered by the Skill Store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0051] In some embodiments, the digital assistant created and deployed using DABP 102 is implemented using a master robot / secondary (or sub) robot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more secondary robots that are skill-based robots. For example, in Figure 1 In the depicted embodiments, the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are secondary robots of the master robot 114. In some embodiments, the digital assistant 106 itself acts as the master robot.

[0052] The digital assistant implemented according to the master-secondary robot architecture enables users to interact with multiple skills through a unified user interface. When a user engages with the digital assistant 106, user input is received by the master robot 114, which then processes the input to identify the user request and determines, based on the processing, whether the requested task can be handled by the master robot 114 itself. Otherwise, the master robot 114 selects the appropriate skill robot 116-1, 116-2, or 116-3 to handle the user request and routes the session to the selected skill robot 116-1, 116-2, or 116-3. This allows the user 108 to engage with and use several skill robots configured to perform specific tasks through a common, single interface. For example, for a digital assistant 106 developed for an enterprise, the main robot 114 of the digital assistant 106 can connect to interfaces with skill robots 116-1, 116-2, etc., which have specific functions, such as a CRM robot for performing functions related to customer relationship management (CRM), an ERP robot for performing functions related to enterprise resource planning (ERP), and an HCM robot for performing functions related to human capital management (HCM). In this way, the end user or customer 108 of the digital assistant 106 only needs to know how to access the digital assistant 106.

[0053] In a master robot / sub-robot infrastructure, the master robot is configured to know a list of skill robots. The master robot can access metadata identifying the various available skill robots and, for each skill robot, access its capabilities, including the tasks it can perform. Upon receiving a user request in utterance form, the master robot is configured to identify or predict from among the multiple available skill robots that can best serve or process the user request. The master robot then routes the utterance (or a portion of it) to the specific skill robot for further processing. Thus, control flows from the master robot to the skill robots. The master robot can support multiple input and output channels.

[0054] Although Figure 1The embodiments shown illustrate a digital assistant 106 including a master robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. A digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code and instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations using a combination of software and hardware.

[0055] The DABP 102 provides the infrastructure, along with various services and features, that enable the use of the DABP 102 to create digital assistants that include one or more skill robots associated with the digital assistant. For example, skill robots can be created by copying an existing skill robot, copying an existing skill robot and then modifying it, or they can be created from scratch using the tools and services provided by the DABP 102. In some embodiments, the DABP 102 provides a skill store or skill catalog that offers multiple skill robots for performing various tasks. Users of the DABP 102 can copy skill robots from the skill store and create new skill robots.

[0056] DABP 102 also enables users (e.g., skill robot designers) to create skill robots from scratch. In some embodiments, creating a skill robot at a high level involves the following steps:

[0057] (1) Configure settings for new skill robots

[0058] (2) Configure one or more intentions for the skill robot

[0059] (3) Configure entities for one or more intents.

[0060] (4) Training Skills Robot

[0061] (5) Create dialogue flow for skill robots

[0062] (6) Add custom parts to the skill robot

[0063] (7) Test and deploy skill robots

[0064] (1) Configuring New Skill Robot Settings—Skill robot designers can specify one or more invocation names for the skill robot being created. These invocation names can be used in utterances to explicitly identify and invoke the skill robot in the digital assistant. Skill robot designers can also specify example utterances for the skill robot. These example utterances represent utterances used for the skill robot. When user input is received, the digital assistant's intent analysis engine compares the user input with these example utterances to determine whether to invoke a specific skill robot.

[0065] (2) Configuring One or More Intents for a Skill Robot—A skill robot designer can configure one or more intents (also called robot intents) for the skill robot being created. These intents identify the tasks the skill robot can perform for the user of the digital assistant. Each intent is given a name. For example, for a skill robot configured to help users perform various banking transactions, the intents can be specified by the skill robot designer for the skill robot, such as “Check Balance,” “Transfer Money,” “Deposit Check,” etc. For each intent, the skill robot designer specifies a set of example statements that represent and illustrate the meaning of the intent and are typically associated with the task performed through said intent. For example, for the “Check Balance” intent, example statements could include “What is my savings account balance?”, “How much money is in my current account?”, “How much money is in my account?”, etc. Therefore, a typical arrangement of user requests and statements can be specified as example statements for the intents.

[0066] (3) Configuring Entities for One or More Intents of a Skilled Robot — In some instances, additional context may be required for the skilled robot to respond appropriately to user requests. For example, there may be cases where user input utterances are parsed into the same intent in the skilled robot. For example, in the example above, the utterances “What is my savings account balance?” and “How much money is in my current account?” are both parsed into the same balance query intent, but these utterances are different requests requesting different things. To clarify such requests, one or more entities are added to the intent. Using the banking skills example, an entity called AccountType (which defines values ​​called “current account” and “savings”) enables the skilled robot to perform syntactic analysis on user requests and respond appropriately. One or more entities can be specified for certain intents configured for the skilled robot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and enable the skilled robot to fulfill the user request. In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (2) custom entities that can be specified by the skilled robot designer. Built-in entities are general entities that can be used with various robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, cycle time, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for banking skills, the account type entity can be defined by the skill bot designer to enable various banking transactions by checking user input against keywords such as current account, savings, credit card, etc.

[0067] (4) Training the Skilled Robot—The skilled robot is configured to receive user input, perform syntactic analysis, or otherwise process the received input and identify or select an intent associated with the received user input. To enable this, the skilled robot must be trained. In some embodiments, the skilled robot is trained based on intents configured for it and example utterances associated with those intents (collectively, training data), such that the skilled robot can parse user input into one of its configured intents. In some embodiments, the skilled robot is represented by a model trained using the training data that allows the skilled robot to discern what the user says (or, in some cases, is attempting to say). DABP 102 provides a variety of different training techniques that can be used by skilled robot designers to train skilled robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof, as described in detail herein with respect to the DAG-based framework. In some embodiments, a portion of the training data (e.g., 80%) is used to train the skilled robot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the skilled robot can then be used to process and respond to user utterances. In some cases, a user's utterance may be a question that requires only a single answer and no further conversation. To handle this, a Q&A (question and answer) intent can be configured for a skill bot. This allows the skill bot to output a response to the user's request without updating the dialogue definition. Q&A intents are created in a similar manner to regular intents. However, the dialogue flow for Q&A intents differs from that for regular intents.

[0068] (5) Creating a Dialogue Flow for the Skilled Bot—The dialogue flow specified for the skilled bot describes how it reacts as it parses different intentions in response to received user input. The dialogue flow defines the actions or behaviors the skilled bot will take (e.g., how it responds to user utterances, prompts for user input, or returns data). The dialogue flow is like the process followed by the skilled bot. Figure 1 Skill bot designers specify the dialogue flow using languages ​​such as standard language. In some embodiments, a version of YAML called OBotML may be used to specify the dialogue flow for the skill bot. The dialogue flow definition for the skill bot acts as a model for the session itself, enabling the skill bot designer to orchestrate the interactions between the skill bot and the users it serves.

[0069] In some embodiments, the dialogue flow definition comprises three parts:

[0070] (a) Background section

[0071] (b) Default Transformation Section

[0072] (c) State section

[0073] Background Section—Skill bot designers can define variables used in the session flow in the background section. Other variables that can be named in the background section include, but are not limited to: variables for error handling, variables for built-in or custom entities, user variables that enable the skill bot to recognize and persist user preferences, etc.

[0074] The Default Transition section—Transitions for the skill bot can be defined either in the Dialogue Flow State section or in the Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when no applicable transition is defined within the state or when the conditions required to trigger a state transition cannot be met. The Default Transition section can be used to define routes that allow the skill bot to handle unexpected user actions appropriately.

[0075] The state component—the dialogue flow and its associated operations—is defined as a sequence of temporary states that manage the logic within the dialogue flow. Each state node within the dialogue flow definition names a component that provides the functionality required at that point in the dialogue. Therefore, states are built around components. States contain component-specific properties and define transitions to other states that are triggered after a component's execution.

[0076] Special scenarios can be handled using the state section. For example, you might sometimes want to offer users the option to temporarily engage with a first skill within the digital assistant while performing actions within a second skill. For instance, if a user is busy interacting with a shopping skill (e.g., the user has already made some purchase choices), the user might want to switch to a banking skill (e.g., the user might want to ensure they have enough money for the purchase) and then return to the shopping skill to complete the order. To address this, the action in the first skill can be configured to initiate an interaction with a different second skill within the same digital assistant and then return to the original flow.

[0077] (6) Adding Custom Parts to the Skill Robot—As described above, the states specified in the dialogue flow for the skill robot name parts that provide the required functionality corresponding to those states. Parts enable the skill robot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured parts for performing a wide range of functions. The skill robot designer can select one or more of these pre-configured parts and associate them with states in the dialogue flow for the skill robot. The skill robot designer can also use the tools provided by DABP 102 to create custom parts or new parts and associate custom parts with one or more states in the dialogue flow for the skill robot.

[0078] (7) Testing and Deploying Skilled Robots — DABP 102 provides several features that enable skilled robot designers to test the skilled robots they are developing. The skilled robots can then be deployed and included in digital assistants.

[0079] While the above description illustrates how to create a skill bot, similar techniques can be used to create a digital assistant (or master bot). At the master bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master bot) can handle without invoking the skill bots associated with the digital assistant. Examples of system intents defined for the master bot include: (1) Exit: applicable when the user signals that they want to exit the current session or background within the digital assistant; (2) Help: applicable when the user requests help or introductions; and (3) UnresolvedIntent: applicable to user input that doesn't quite match the Exit and Help intents. The digital assistant also stores information about the one or more skill bots associated with it.

[0080] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the conversation. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user's input should be routed to a specific skill for processing, processed by the digital assistant or master robot itself according to built-in system intent, or processed into a different state within the current conversation flow.

[0081] In some embodiments, as part of this process, the digital assistant determines whether the user input identifies a skill bot using its invocation name. If the invocation name is present in the user input, it is considered an explicit invocation of the skill bot corresponding to that name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In some embodiments, if no specific invocation is made, the digital assistant evaluates the received user input and calculates a confidence score for the system intent and skill bot associated with the digital assistant. The score calculated for the skill bot or system intent indicates how likely the user input represents a task that the skill bot is configured to perform or represents a system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input. In some embodiments, after one or more skill bots are identified as candidates, intents associated with those candidate skills are evaluated (according to an intent model for each skill) and a confidence score is applied to each intent. Any intent with a confidence score exceeding a threshold is typically considered a candidate stream. If a specific skill bot is selected, user input is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed based on the selected system intent.

[0082] Figure 2 A robot system 205 (as described in some embodiments) is depicted. Figure 1 The described system is an integrated system 200 comprising a digital assistant or robot system 106 and a robot analytics system 210 for monitoring, analyzing, visualizing, and improving the performance of the robot system. As illustrated, the robot system 205 may include a connector 215 and multiple robot engines 220 such as a dialogue engine 222, an intent modeler 224, an entity parser 226, and custom components 228. The robot system 205 may also include a database 230, a management API 235, a user interface 245, and a UI server 240. The robot analytics system 210 may include a collector 250, an enrichment engine 255, a database 260, and a REST server 265. The robot analytics system 210 may also include a user interface 270 and a UI server 275. The collector 250 of the robot analytics system 210 may collect events 290 occurring at the robot system 205. Feedback 285 from the robot analytics system 210 may be provided to the robot system 205 through the user interface 270 and the user interface 245.

[0083] Connector 215 can act as an interface between robot system 205 and one or more end users via one or more channels (such as channels 286 and 287). Each channel can be a messaging application, such as a messaging channel (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), a virtual personal assistant (e.g., Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), a native or hybrid extended mobile and web application extension / a responsive mobile application or web application with chat capabilities, or voice-based input (e.g., a device or application with an interface using Siri, Cortana, Google Voice, or other voice input for interaction). In some embodiments, connector 215 can normalize content from different channels, allowing robot system 205 to analyze content across different messaging application systems. Content normalization processing can include formatting content from each type of messaging application into a common format for processing. In some embodiments, robot system 205 can include one or more connectors for each channel. Intent modeler 228 can be used to determine end-user intents associated with end-user utterances. After normalization, the probability that the occurrence of a word indicates a certain intention can be determined. In some examples, basic probabilistic algorithms can be used to combine probabilities, as if the probabilities were independent.

[0084] Examples can also be provided to prevent the model from making incorrect assertions. For instance, specific subphrases or words that appear only for a particular intent may lead to incorrect assertions. Similarly, it is possible to prevent the model from synthesizing broad rules by training with similar sentences belonging to different intents.

[0085] Entity parser 224 can identify entities (e.g., objects) associated with end-user intents. For example, in addition to end-user intents (such as “order pizza”) identified by intent modeler 228, entity parser 224 can also parse entities associated with intents, such as pizza type and toppings.

[0086] The dialogue engine 226 can be used to handle conversations between an end user and a robotic system. For example, the dialogue engine 226 can respond to end user utterances based on end user intentions identified by the intention modeler 228 and entities associated with those intentions identified by the entity parser 224. In some embodiments, the dialogue engine 226 can use a state machine that includes user-defined states (e.g., end user intentions) and actions to be taken within or between states to handle conversations with the end user.

[0087] Custom component 222 may include custom modules for a specific robotic system. For example, a financial robot may include custom components that can be used, for example, to check balances, transfer funds, or pay bills.

[0088] Database 230 can be used to store data for the robot system, such as data for the classification model, session logs, etc. Management API 235 can be used by the robot system's administrator or developer to manage the robot system, such as retraining the classification model, editing intents, or otherwise modifying the robot system. Administrators or developers can use user interface 245 and UI server 240 to manage the robot system.

[0089] Various events 290 may be generated during the operation of the robot system 205. Event 290 may be generated based on one or more instructions included in the robot system. For example, event 290 may be generated when the robot system 205 has entered a specific state, wherein the specific state is defined by the robot system administrator or developer. When event 290 is generated, event 290 may be collected, stored, and analyzed by the robot analysis system 210. When event 290 is captured, additional information associated with event 290 may also be collected, wherein the additional information may indicate the current context in which event 290 was generated.

[0090] For example, session events can be generated by dialogue engine 226. Session events may include messages (referred to as msg_received) received by the robot system from the end-user device. msg_received may include one or more of the following parameters or variables: message content, the time the message was received by robot system 205, the language of the received message, device nature (e.g., version or name), operating system nature (e.g., version or name), geographic location nature (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, robot system ID, possessor ID, etc.), timestamps (e.g., timestamps created by the device, timestamps sent by the device, timestamps obtained by the collector), channels, etc.

[0091] Session events may also include messages (referred to as msg_sent) sent by the robot system 205 to the end-user device. msg_sent may include one or more of the following: the content of the message (e.g., the text or HTML of the message), the time the message was sent by the robot system, the language of the message, the creator of the message (e.g., the robot system or the end-user device), the device nature, the operating system nature, the browser nature (e.g., version or name), the application nature (e.g., version or name), the geographic location nature (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, robot system ID, possessor ID, etc.), the channel (e.g., Facebook or Webhook), etc.

[0092] The dialogue engine 226 can also generate dialogue state execution events. As described above, the dialogue engine 226 can use a state machine to determine the flow of the conversation with the end user. The state machine can include a set of states and transition rules between states. The dialogue engine 226 can execute the state machine for each end user session, and dialogue state execution events can be generated for each state that the dialogue engine 226 progressively passes through to process the end user's utterance. Attributes of dialogue state execution events can include, for example, state name, component name, next action, entity match, intent match, variable, user query statement, response statement, execution time, communication language, device nature, operating system nature, geographic location nature, identification information, timestamp, channel, etc. The state name is the name of the currently executed state or "error state". The component name can be the name of the robot component being executed for the current state. The next action can be the next action to be executed. The entity match can be the entity resolved in the current message. The intent match can be the intent resolved with a score value. The variable can be the variable value of the current state. The query statement can be a message sent by the end user. The response statement can be a message sent to the end user. The execution time can be a timestamp of the completed state execution. The language of communication can be the language of the messages exchanged. Device characteristics and / or operating system characteristics can be associated with the end user interacting with the robot system. Browser characteristics and / or application characteristics can be associated with the end user interacting with the robot system. Geographic location characteristics can be the location of the end user interacting with the robot system.

[0093] An intent resolution event may occur due to the execution of intent modeler 228. Intent modeler 228 may identify end-user intents from a set of intents based on end-user utterances using a trained or otherwise defined classification model. The results of intent classification may be captured as intent resolution event attributes, which may include, for example, the final intent classification result (e.g., the identified intents) and a confidence score associated with each corresponding intent in the set of intents.

[0094] Entity parser 224 can generate entity parser events. An entity is an object associated with an end-user intent. Entity definition rules can be determined when the robot system is created. For example, in addition to parsing end-user intents such as "order pizza," the robot system can also use entity parser 224 to parsing associated entities such as pizza size and toppings. Entity parser events can be captured during entity parsing. Examples of attributes associated with entity parser events may include entity name, applied rules, search terms, parsed state, query statement, entity type, execution time, communication language, device nature, operating system nature, browser nature, application nature, geolocation nature, identification information, timestamp, channel, etc. Entity name can be the name of the currently parsed entity. Applied rules can be, for example, those mentioned above, those mentioned below, or aggregated. Search terms can be from, to, destination, origin, etc. Parsed state can be the dialogue state for entity parsing. Query statement can be a message containing entity values. Entity type can be systematic or derived. Execution time can be the timestamp of entity parsing. Communication language can be the language of the conversational message. Device properties and / or operating system properties can be associated with the end user interacting with the robot system. Browser properties and / or application properties can be associated with the end user interacting with the robot system. Geographic location properties can be the location of the end user interacting with the robot system.

[0095] Custom component 222 can also generate events such as predefined events or custom events. Predefined events can be properties captured when the custom component is executed. Examples of predefined event properties can include: component name, event name, payload, execution time, communication language, device property, operating system property, browser property, application property, geolocation property, identification information, timestamp, channel, etc. The component name can be the name of the currently executed custom component. The event name can be called, call failed, answered, answer failed, etc. The payload can be the reason for failure (in the case of failure), stack trace, etc. The execution time can be a timestamp indicating when the event occurred. The communication language can be the language of the message being communicated. Device property and / or operating system property can be associated with the end user interacting with the robot system. Browser property and / or application property can be associated with the end user interacting with the robot system. The geolocation property can be the location of the end user interacting with the robot system.

[0096] Custom component 222 can also issue custom events during the execution of the custom component. Examples of attributes for a custom event may include component name, event name, custom payload, execution time, communication language, device nature, operating system nature, browser nature, application nature, geolocation nature, identification information, timestamp, channel, etc. The component name can be the name of the currently executed custom component. The event name can be a user-defined event name (e.g., Balance_Retrieved). The payload can be, for example, {"Amount": "$100", "Account": "Checking Deposit"}. The execution time can be a timestamp indicating when the event occurred. The communication language can be the language of the message being communicated. Device nature and / or operating system nature can be associated with the end user interacting with the robot system. Browser nature and / or application nature can be associated with the end user interacting with the robot system. The geolocation nature can be the location of the end user interacting with the robot system.

[0097] Error events and timeout events can also be generated by the robot system 205 during execution. Error events can be generated when an error occurs. Timeout events can be generated when an end-user session has been inactive for a period of time, which can be configured at the channel level.

[0098] While the robot system 205 is engaging in a conversation with the end user and generating corresponding events, the analysis system 210 can collect events 290 and other information. For example, collector 250 can collect events 290 and other information and send the collected information to a queue. In some embodiments, collector 250 can be configurable and programmed to collect the various events and / or event attributes described above as needed. For example, collector 250 can be configured to capture dialogue state attributes, intent resolution attributes, entity resolution attributes, and error and timeout attributes. In some embodiments, collector 250 can also be configured to collect information about events 280 generated by systems other than the robot system.

[0099] Enrichment engine 255 can validate and enrich the collected events and other information and write the collected events and other information into database 260. For example, based on the collected IP addresses, enrichment engine 255 can determine the location of the end user associated with the IP address. As another example, enrichment engine 255 can extract certain features from the collected information, such as determining the web browser or channel used by the end user. REST server 265 can analyze the enriched events and other information and generate various reports based on a certain aggregate metric 295. The reports can be displayed on user interface 270 via UI server 275 to the owner, manager, or developer of robot system 205. The owner, manager, or developer of robot system 205 can provide feedback 285 to robot system 205 for improvement.

[0100] DAG-based framework

[0101] The number of data processing workflows involved in developing machine learning systems and running models in production is increasing. Pipelines for modeling workflows (a set of stages executed one after another) typically run from ingesting and cleaning data through feature engineering and model selection in an interactive workbench environment, to training and experimentation (often with the option to share results), to deploying the trained model, to serving results such as prediction and classification. Managing the complexity of these pipelines becomes increasingly difficult, especially when users may attempt to work with real-time data and frequently update models. There are various traditional tools, libraries, and frameworks for machine learning, and many users have their own specific groups that prefer to work together, with tools, libraries, and frameworks integrating with data stores and platforms that run machine learning models in different ways. However, for scalability, ease of coding, and extensibility, machine learning pipelines—especially those used in natural language processing (e.g., with chatbots)—need mechanisms that are easy to define, build, and test.

[0102] The mechanism disclosed in this paper is a DAG-based framework (also referred to herein as a training framework or training tool) for programmatically writing, modeling, and monitoring robot-related workflows of robotic systems. The DAG-based framework can be implemented as... Figure 1 The described Digital Assistant Builder Platform (DABP) 102 is part of and is designed for performance use regarding Figure 2 The described robot analysis system 210 performs analysis and monitoring. Although an exemplary linguistic model is described herein, the DAG-based framework should be understood as applicable to handling any machine learning model. The training framework is a modular machine learning framework for rapidly developing and deploying machine learning algorithms for service applications such as Oracle Digital Assistant. In some embodiments, the training framework is built on a cluster computing framework such as Apache Spark, which distributes the machine learning algorithm across a cluster of computing nodes, allowing the model to scale during training and deployment. The training framework comes out of the box using a set of transformers and can build and add additional transformers over time. An example of a simple transformer would be a whitespace-based string splitting mechanism. The training framework can be implemented as a DAG of tasks for writing a workflow for training a chatbot. The training framework can execute tasks on an array of worker nodes while adhering to specified dependencies of the nodes defined in the workflow. The training framework may include command-line utilities that allow complex changes to the tasks. The training framework may also include a graphical user interface that makes it easy to visualize the pipeline running in production, monitor progress, and resolve issues as needed.

[0103] Essentially, all tasks and activities to be implemented in the model are arranged into a clear structure or pipeline of discrete processes that occur at a set point and have explicit relationships with other tasks. If multiple tasks exist, each task has at least one definite upstream (previous) task or downstream (subsequent) task (although each task can easily have both). No task can create data that continues to reference itself (this avoids any instances of infinite loops). Figure 3As shown, a DAG-based pipeline 300 can be defined as a JSON object containing an acyclic array of nodes 305 and connections 310 between the nodes. Pipeline 300 is a graph that tracks the trajectories of operations (OPs) applied to a dataset (DF) such as a Resilient Distributed Dataset (RDD) (e.g., a data table). The trajectories of operations connect directly from one node to another. This creates a sequence, where each node is linked from earlier to later in an appropriate sequence, and each node automatically recognizes the output from the previous node as input and the output of the current node as input for the next node. Each node within the sequence adds a new field to the DF, which is moved from one node to another as the workflow continues through pipeline 300. Furthermore, the pipeline is defined such that no cycles or loops are available. Once a transformation occurs, the pipeline cannot return to its earlier position. Due to the acyclic nature of the nodes and their connectivity, the cluster computing framework is able to automatically identify cyclic dependencies and reject pipelines during the pipeline detection phase. Figure 4 An invalid pipeline 400 with circular dependencies is shown.

[0104] In various embodiments, the DAG-based pipeline 300 includes: (i) one or more nodes 305 (also referred to herein as stages, transformers, or transformer stages); (ii) one or more connections 310; and (iii) optionally, one or more complex connections (see example...). Figure 6 Each node 305 has a stage definition, which includes: a "name", the class to be invoked via introspection such as Java reflection, the parameters required by the class, and one or more operations (OPs) to be performed on a given dataset. A class is a user-defined blueprint or prototype from which objects can be created. A class represents a set of properties or methods common to all objects of a type. Typically, a class declaration may include: modifiers (e.g., public or private access), a class name, a superclass (the name of the parent class), interfaces or operations (e.g., a list of interfaces implemented by the class), and the class body. Introspection, such as reflection, is a feature that allows an executable program to examine or "introspect" itself and manipulate the internal properties of the program. For example, a class can retrieve the names of all its members and display those names. The parameters required by the class refer to a list of variables (e.g., double-precision floating-point numbers, floating-point numbers, integers, etc.) in the method declaration. Connection 310 defines the nodes to be called and the order of operations to be performed. Complex connections provide multi-input nodes, pipelines as separate nodes between multiple pipelines, and ignore functions for stages that have been completed.

[0105] In pipeline 300, each node 305 adds a new field to the dataframe. An example of a node (feature extraction node) is shown below:

[0106] {

[0107] "params": {

[0108] "intentServer.sparkySettings.featureExtractor.location": "\ / u01\ / app\ / data\ / word_vectors\ / glove.6B.100d",

[0109] "intentServer.sparkySettings.featureExtractor.type": "word2vec"

[0110] },

[0111] "class": "oracle.cloud.bots.intent.model.spark.stages.featureExtraction.FeatureExtractionStages"

[0112] }

[0113] Each node 305 can implement a single method for a single input node. Because the pipeline 300 is built on a DAG-based framework, resilient and distributed computing is implicitly achieved. In some embodiments, this single method can be modified to further simplify the approach by not passing the framework session. In the example above, the framework session is passed, allowing the class to also register user-defined functions (UDFs) that the framework might require, such as `public PipelineStage getStage(SparkSession spark){}`.

[0114] The one or more nodes 305 implicitly include the following characteristics: (i) a single output field / stage that is the node name (complex objects can be stored in the pipeline); and (ii) different parts of the pipeline are not allowed to have the same name (new nodes 500 with different names need to be created to implement nodes with the same functionality as another node, such as...). Figure 5 (As shown).

[0115] An exemplary pipeline is shown below. The pipeline shows several nodes (JSONObject "nodes") and the order in which those nodes are executed (JSONObject "pipelines").

[0116] {

[0117] "nodes": {

[0118] "trim": {

[0119] "class": "oracle.cloud.bots.intent.model.spark.stages.preprocessing.TrimPreprocessorStage"

[0120] },

[0121] "start": {

[0122] },

[0123] "rempunctuation": {

[0124] "class": "oracle.cloud.bots.intent.model.spark.stages.preprocessing.RemovePunctuationPreprocessorStage"

[0125] },

[0126] "model": {

[0127] "params": {

[0128] "intentServer.sparkySettings.trainingModel.name": "org.apache.spark.ml.classification.LogisticRegression",

[0129] "family": "multinomial"

[0130] },

[0131] "class": "oracle.cloud.bots.intent.model.spark.stages.train.TrainStages"

[0132] },

[0133] "end": {

[0134] },

[0135] "featureextraction": {

[0136] "params": {

[0137] "intentServer.sparkySettings.featureExtractor.location": "\ / u01\ / app\ / data\ / word_vectors\ / glove.6B.100d",

[0138] "intentServer.sparkySettings.featureExtractor.type": "word2vec"

[0139] },

[0140] "class": "oracle.cloud.bots.intent.model.spark.stages.featureExtraction.FeatureExtractionStages"

[0141] },

[0142] "tokenization": {

[0143] "class": "oracle.cloud.bots.intent.model.spark.stages.preprocessing.TokenizePreprocessorStage"

[0144] }

[0145] },

[0146] "pipelines": [

[0147] "start",

[0148] "trim",

[0149] "rempunctuation",

[0150] "tokenization",

[0151] "featureextraction",

[0152] "model",

[0153] "end"

[0154] ],

[0155] "tenantid": "chatbot-tenant",

[0156] "botid": "83A1B880-5FE9-4E78-A755-C8EA09E1A0E7"

[0157] }

[0158] Figure 6 An example of a complex machine learning pipeline 600 with task calibration (which tasks are repetitive) and task dependency calibration is shown. The DAG structure helps detect recurring requirements and also identifies task dependency calibration.

[0159] In various embodiments, the cluster computing framework includes the following features: (i) model definition (e.g., configuration based on a domain-specific language (DSL); (ii) model performance; and (iii) the actual machine learning model. The advantage of this mechanism is that the model can still execute a self-contained model as long as the implementation of the model definition remains unchanged. The following is an example of a DSL configuration. The following model illustrates the different steps of the model (AVG_word2vec_LogisticRegression): lower → trim → remove_punctuation → tokenization → feature_extraction → model. In some embodiments, steps can be introduced into / removed from the pipeline. This delineates model development. For example, the cluster computing framework can perform multiple (e.g., 5) cross-validations by default and provides metrics.

[0160] {

[0161] "CV": {

[0162] "numFolds": 5

[0163] },

[0164] "nodes": {

[0165] "end": {},

[0166] "featureextraction": {

[0167] "class": "oracle.cloud.bots.intent.model.spark.stages.featureExtraction.FeatureExtractionStages",

[0168] "params": {

[0169] "featureextractor.languagelabel": "English",

[0170] "featureextractor.type": "word2vec"

[0171] }

[0172] },

[0173] "lower": {

[0174] "class": "oracle.cloud.bots.intent.model.spark.stages.preprocessing.LowercasePreprocessorStage"

[0175] },

[0176] "model": {

[0177] "class": "oracle.cloud.bots.intent.model.spark.stages.train.TrainStages",

[0178] "inference": "oracle.cloud.bots.intent.model.spark.model.inference.LogisticRegression",

[0179] "params": {

[0180] "stageParams": {

[0181] "regParam": [

[0182] 0.1, 0.01 ]

[0185] },

[0186] "trainer.name": "org.apache.spark.ml.classification.LogisticRegression"

[0187] }

[0188] },

[0189] "rempunctuation": {

[0190] "class": "oracle.cloud.bots.intent.model.spark.stages.preprocessing.RemovePunctuationPreprocessorStage"

[0191] },

[0192] "start": {},

[0193] "tokenization": {

[0194] "class": "oracle.cloud.bots.intent.model.spark.stages.preprocessing.TokenizePreprocessorStage"

[0195] },

[0196] "trim": {

[0197] "class": "oracle.cloud.bots.intent.model.spark.stages.preprocessing.TrimPreprocessorStage"

[0198] }

[0199] },

[0200] "pipelines": [

[0201] "start",

[0202] "lower",

[0203] "trim",

[0204] "rempunctuation",

[0205] "tokenization",

[0206] "featureextraction",

[0207] "model",

[0208] "end" ]

[0210] }

[0211] Regarding model performance, the cluster computing framework can use multiple cross-validation to report performance metrics. These metrics can depend on the data used during training. Examples of these metrics, which can be obtained using the model, are shown below. In some embodiments, these metrics can be exposed to clients. At a glance, it's clear that the model's accuracy is 1; where the f-score is 0.806.

[0212] {

[0213] "accuracy": 1,

[0214] "confusionMatrix": "4.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 2.0 ",

[0215] "f1scoreClass:Balances": 1,

[0216] "f1scoreClass:Send Money": 1,

[0217] "f1scoreClass:Track Spending": 1,

[0218] "modelAVGMetrics": [

[0219] 0.8062962962962963, 0.8062962962962963

[0221] ],

[0222] "precisionClass:Balances": 1,

[0223] "precisionClass:Send Money": 1,

[0224] "precisionClass:Track Spending": 1,

[0225] "recallClass:Balances": 1,

[0226] "recallClass:Send Money": 1,

[0227] "recallClass:Track Spending": 1,

[0228] "weightedF1Score": 0.9999999999999999,

[0229] "weightedFalsePositive": 0,

[0230] "weightedPrecision": 0.9999999999999999,

[0231] "weightedRecall": 0.99999999999999999

[0232] Advantageously, implementing a cluster computing framework to build a pipeline for robotic systems can reduce model size (in some instances, by a factor of ten), achieve improved accuracy, and improve f-score (a measure of model accuracy that considers both precision p and recall r of the test to calculate the score). Below, Tables 1 through 3 illustrate some exemplary model size, accuracy, and f-score metrics implemented using a Spark-like framework.

[0233] Table 1 - Model Size

[0234]

[0235] Table 2 - Model Accuracy

[0236]

[0237] Table 3 - Model F1 Score

[0238]

[0239] The dependency on the cluster computing framework during query time has been removed to achieve higher queries per second. Since the amount of memory required per model for AVG_word2vec_LogisticRegression is 10 times less than Tamao, performance becomes more stable as more bots are added. Utilizing the cluster computing framework, intent classification is no longer memory-limited but CPU-limited. In some embodiments, the AVG_word2vec_LogisticRegression model uses Glove 100d with 600 million weighted word vectors. With 110 utterances, the model achieves close to 91.87% accuracy. With 220 utterances, the model achieves close to 93.97% accuracy. With 330 utterances, the model achieves close to 95.17% accuracy. Achieving accuracy above 97% typically requires approximately 1984 utterances.

[0240] Changes in model testing

[0241] The ability to describe models as JSON objects within a cluster computing framework allows for: (i) testing multiple models; and (ii) testing multiple variations of a single model. In various embodiments, when a client executes a training method (e.g., activating a training button in the GUI), the cluster computing framework computes a model (known to perform optimally on multiple datasets) and makes the model available for client use. In some embodiments, a user can use the cluster computing framework to manually create, train, modify, retrain, and execute models. In other embodiments, the cluster computing framework automatically creates, trains, modifies, retrains, and executes models. In still other embodiments, a combination of user input and automated processes on the cluster computing framework creates, trains, modifies, retrains, and executes models. Instructions can be provided to the client that a "certain" model has been trained and that a robot running the model is available at runtime; further probing of the model can be performed in the background or at design time. In other words, (i) multiple models and (ii) multiple variations of a single model can continue to be trained and monitored to determine which models perform best based on one or more parameters configured to measure the performance of one or more models.

[0242] Customers can provide various parameters they want to adjust across their models; for example, some customers want high accuracy at the expense of throughput, while others want to optimize throughput by sacrificing some accuracy. Model deployment can be performed automatically, for example, based on the number of classification requests per second, scaling the cost of classification during peak times, and / or based on the highest accuracy. As shown below and described in this paper, the model and its accuracy have been captured according to the cluster computing framework in the model.

[0243] {

[0244] "accuracy": 1,

[0245] "confusionMatrix": "4.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 2.0 ",

[0246] "f1scoreClass:Balances": 1,

[0247] "f1scoreClass:Send Money": 1,

[0248] "f1scoreClass:Track Spending": 1,

[0249] "modelAVGMetrics": [

[0250] 0.8062962962962963, 0.8062962962962963

[0252] ],

[0253] "precisionClass:Balances": 1,

[0254] "precisionClass:Send Money": 1,

[0255] "precisionClass:Track Spending": 1,

[0256] "recallClass:Balances": 1,

[0257] "recallClass:Send Money": 1,

[0258] "recallClass:Track Spending": 1,

[0259] "weightedF1Score": 0.9999999999999999,

[0260] "weightedFalsePositive": 0,

[0261] "weightedPrecision": 0.9999999999999999,

[0262] "weightedRecall": 0.99999999999999999

[0263] }

[0264] Users can create new models for training by creating different getModel() methods (multiple models), each containing different learning algorithms. For example, the following definition constructs a logistic regression model.

[0265]

[0266] Another model design technique is to skip certain preprocessing steps to enable the creation of different variations of the model. For example, in the following code, multiple models are returned. These models experiment with different permutations and combinations of lowercase letters, remove punctuation marks, and trim nodes; thus producing eight (2) models to be tested. 3 ( ) models.

[0267] public List <jsonobject>getModelDef() {

[0268] if (this.modelDef == null) {

[0269] modelDef = new ArrayList <jsonobject>();

[0270] List <string>modelChanges = new ArrayList <string>();

[0271] modelChanges.add(SparkModelKeys.LOWERCASE_NODE);

[0272] modelChanges.add(SparkModelKeys.REMOVEPUNCTUATION_NODE);

[0273] modelChanges.add(SparkModelKeys.TRIM_NODE);

[0274] Set<List <string>>l = combinations(modelChanges);

[0275] CommonStageDefinitions cs = new CommonStageDefinitions();

[0276] for (List <string>lv : l) {

[0277] JSONObject node = new JSONObject();

[0278] node.put(SparkModelKeys.START_NODE, cs.getNoOp());

[0279] if(lv.contains(SparkModelKeys.LOWERCASE_NODE))

[0280] node.put(SparkModelKeys.LOWERCASE_NODE, cs.getLowercase());

[0281] if (lv.contains(SparkModelKeys.REMOVEPUNCTUATION_NODE))

[0282] node.put(SparkModelKeys.REMOVEPUNCTUATION_NODE,cs.getRemovePunctuation());

[0283] if (lv.contains(SparkModelKeys.TRIM_NODE))

[0284] node.put(SparkModelKeys.TRIM_NODE, cs.getTrim());

[0285] node.put(SparkModelKeys.TOKENIZATION_NODE, cs.getTokenization());

[0286] Language language = bot.getPredominantLanguage();

[0287] if (Language.zh != language) {

[0288] language = Language.en;

[0289] }

[0290] node.put(SparkModelKeys.FEATUREEXTRACTION_NODE,cs.getFeatureExtraction(language.getLabel()));

[0291] node.put(SparkModelKeys.MODEL_NODE, this.getModel());

[0292] node.put(SparkModelKeys.END_NODE, cs.getNoOp());

[0293] JSONObject tmodelDef = new JSONObject();

[0294] tmodelDef.put(SparkModelKeys.NODES, node);

[0295] tmodelDef.put(SparkModelKeys.PIPELINES, this.getPipeline());

[0296] tmodelDef.put(SparkModelKeys.CV_NODE, cs.getCV());

[0297] modelDef.add(tmodelDef);

[0298] }

[0299] }

[0300] return this.modelDef;

[0301] }

[0302] }

[0303] Techniques for creating and implementing models

[0304] Figure 7 This is a simplified flowchart 700, illustrating examples of processes for monitoring, analyzing, visualizing, and improving the performance of a robotic system according to certain embodiments. Figure 7 The described processing can be performed through the following: robot builder platform, cluster computing framework, and integrated system, as discussed in [the relevant section]. Figures 1 to 6 The described digital assistant builder platform, cluster computing framework, and integration system. Figure 7 The described process can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding system, in hardware, or in a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 7 The treatments presented and described below are intended to be illustrative rather than limiting. Although Figure 7 The individual processing steps, which occur in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel.

[0305] At step 705, one or more models are generated from the task pipeline using a DAG-based framework (e.g., a clustered computing framework) and trained on the one or more models to be implemented by the chatbot. For example, multiple natural language processing tasks may be implemented within the pipeline to generate models for predicting intent from utterances. In some embodiments, the one or more models are multiple models, each with a different execution method. In some embodiments, the one or more models are a single model with multiple permutations (variation exists between each version of the model). At least one of the one or more models is implemented for the chatbot during runtime, while the remaining models or variations of the models may run in the background during design time. The one or more models running during runtime execute on the dataset to generate outputs to be used by the chatbot in downstream processes (e.g., predicting user intent from utterances, allowing the chatbot to provide dialogue or take action based on the predicted user intent). In contrast, the one or more models running in the background during design time execute on the same dataset to generate outputs; however, these outputs are not used by the chatbot in downstream processes, and instead, they are used to analyze the performance of the one or more models running in the background compared to the one or more models running during runtime. In other words, one or more models that run during runtime are executed within the chatbot to perform or implement chatbot functionality, while one or more models that run in the background during design time are executed either within or outside the chatbot for the purpose of analyzing and comparing the models that run during runtime.

[0306] A pipeline is a collection of all the tasks a client wants to run in a chatbot, organized in a way that reflects the relationships and dependencies between all tasks. For example, a simple pipeline might consist of three tasks: A, B, and C. Through nodes and connections, the pipeline can provide: A must run successfully before B can run, but C can run at any time. The pipeline can provide: Task A times out after 5 minutes, and if B fails, B can be restarted up to 5 times. The pipeline can also specify: The workflow will run for each utterance. In this way, a DAG describes how the client wants the chatbot to implement its workflow. Tasks A, B, and C can be anything related to implementing one or more functions of the chatbot. For example, in a first pipeline, A might fetch data from a database, B might prepare data for C to analyze by trimming it, and C might use the trimmed data as input to train or execute an algorithm or machine learning model. Alternatively, in a second pipeline, A might fetch data from a database, B might prepare data for C to analyze by removing punctuation, and C might use the data with punctuation removed as input to train or execute an algorithm or machine learning model. The results of the first and second pipelines can be compared by analyzing the system to determine the best method for processing the data. Importantly, a DAG does not concern itself with what its constituent tasks do; the job of a DAG is to ensure that whatever the tasks do is at the right time, in the right order, or by properly handling any unexpected problems that occur.

[0307] In some embodiments, a DAG-based framework of the integrated computing system is used to generate a first model and a second model. In some instances, generation includes training the first and second models on a dataset prior to deployment and implementation by the chatbot. In some instances, the first and second models are trained based on intents configured for the chatbot (e.g., a skill bot) and example utterances associated with those intents (collectively, training data), such that a chatbot using the first and / or second models can parse user input into one of its configured intents. The first model is a first pipeline for performing one or more operations associated with the chatbot. These operations are executed sequentially on the first set of nodes, and each node automatically recognizes the output from the previous node as input and the output of the current node as input for the next node. Each node within the sequence adds a new field to a data table that is moved from one node to another as the workflow progresses through the first pipeline. The second model is a second pipeline for performing one or more operations associated with the same chatbot. The one or more operations are executed sequentially on the second set of nodes, and each node automatically recognizes the output from the previous node as input and the output of the current node as input for the next node. Each node in the sequence adds a new field to a data table that is moved from one node to another as the workflow progresses through the second pipeline.

[0308] The first set of tasks differs from the second set. The difference may lie in the addition or removal of at least one task, the replacement of at least one task, or the order in which at least one task is processed. For example, the first set of tasks may include data extraction, data trimming, punctuation removal, word segmentation, feature extraction, model training, and cross-validation. The second set of tasks may include data extraction, punctuation removal, word segmentation, feature extraction, model training, and cross-validation. Alternatively, the second set of tasks may include data extraction, data trimming, punctuation removal, word sense disambiguation, feature extraction, model training, and cross-validation. Alternatively, the second set of tasks may include data extraction, data trimming, punctuation removal, word sense disambiguation, feature extraction, model training, and cross-validation. Alternatively, the second set of tasks may include data extraction, data trimming, punctuation removal, word segmentation, feature extraction, model training, and cross-validation. As should be understood, the first set of tasks is performed on the chatbot during runtime, while the second set of tasks may run in the background during design time. The first set of tasks, running during runtime, is executed on the dataset to generate outputs to be used by the chatbot in downstream processes (e.g., predicting user intent from utterances, allowing the chatbot to offer suggestions or take actions based on the predicted user intent). In contrast, the second set of tasks, running in the background during design time, is executed on the same dataset to generate outputs; however, these outputs are not used by the chatbot in downstream processes, and instead, they are used to analyze the performance of the second set of tasks running in the background compared to the first set running during runtime—this is why the first and second sets of tasks differ. At step 710, the first model for the chatbot is executed during runtime using a directed acyclic graph (DAG)-based framework of the integrated computing system, and the second model for the chatbot is executed during design time using the DAG-based framework of the integrated computing system. Executing the first and second models may include acquiring content data from one or more channels (e.g., regarding...). Figure 2 As described, user content data is acquired at connector 215 via channel 286 or 287, or content data is acquired from a database (e.g., regarding...). Figure 2 As described, this involves retrieving content data such as session logs from database 230 at robot engine 220, and using the first and second models (e.g., regarding...). Figure 2 The model described herein (running as an intent modeler 228 in robot engine 220) parses the intent. Further execution includes generating events in response to the parsed intent (e.g., regarding...). Figure 2 The described robot event 290). The event may be generated based on one or more instructions included in the integrated computing system, requesting the collection of one or more attributes for classifying intents associated with a set of utterances. In an instance of the first model, further execution includes the chatbot generating dialogue (e.g., using, for example, regarding...). Figure 2 The described dialogue engine 226), parsing entities (e.g., using information such as about...) Figure 2 The entity parser described 224) and / or performs operations or actions in response to the parsing intent (e.g., initiating an order for pizza).

[0309] At step 715, the event collector of the integrated computing system (e.g., as per [reference]) Figure 2 The described event collector 250 is configured to capture one or more attributes for classifying intents associated with a set of utterances while the chatbot runs a first model and a second model. An attribute is any piece of information that enables the generation of metrics to measure the model's performance in predicting or classifying intents from the set of utterances. For example, attributes may include: the set of utterances, the final intent classification result (e.g., the identified intent), the parameters currently used for prediction or classification by one or more models (e.g., weights and biases), the hyperparameters currently used for prediction or classification by one or more models, the size of one or more models, the percentage of probability assigned to each intent by one or more models for prediction or classification, the average query time, the model processing time, or the classification time, and a confidence score associated with the probability of each corresponding intent in the set of intents being identified for the final intent classification.

[0310] As should be understood, the events and attributes collected by the event collector are not limited to intent categories associated with a set of utterances, which are provided as illustrative examples. Events generated by the chatbot system may include, for example, conversation events, dialogue state execution events, intent resolution events, entity resolution events, and events generated by custom components. The event collector can be configured to collect desired events or desired attributes associated with various events such as intent categories. When the chatbot resolves intents using a first model and a second model and generates corresponding events, and when the chatbot conducts a conversation with an end user using the first model and generates corresponding events, the integrated computing system can collect events and additional information. For example, the event collector can collect events and additional information and send the collected information to a queue. In some embodiments, the event collector can be configurable and can be programmed to collect the different events and / or event attributes described above as desired. For example, the event collector can be configured to capture dialogue state attributes, intent resolution attributes, entity resolution attributes, and error and timeout attributes. In some embodiments, the collector can also be configured to collect information about events generated by systems other than the integrated computing system.

[0311] At step 720, the analysis engine of the integrated computing system evaluates the performance of the first and second models based on the analysis of the one or more attributes used for intent classification using one or more metrics. For example, an event collector can collect events (including event attributes) and additional information and send the collected information to an evaluation queue. Enrichment engines (e.g., regarding...) Figure 2 The enrichment engine (255) described above can validate and enrich queuing events and other information, and write the collected events, attributes, enriched data, and additional information into a database. For example, based on collected IP addresses, the enrichment engine can determine the location of the end user associated with the IP address and append enriched data to the event or attribute. The analytics engine can then analyze the collected events, attributes, enriched data, and additional information and calculate one or more metrics indicating the model's usefulness and / or performance, such as interpretability, usefulness, accuracy, log loss, confusion matrix, area under the curve, f-score, mean absolute error, and mean squared error. Metrics can be calculated based on one or more events or attributes described above. Metrics can be calculated on a daily, weekly, monthly, or custom scale. Examples of basic metrics include (1) the number of unique, total, new, active, inactive, or returning end users; (2) total chats / sessions; (3) average, maximum, median, or minimum session duration; (4) the average time between two sessions for an end user; (5) sentiment (positive, negative, or neutral); (6) the number of end users, the number of sessions, or the number of unique end users; (7) average, maximum, median, or minimum message count, etc. Each metric can be filtered by the following: channel (e.g., Facebook or webhook), geography (e.g., country, state, city, or postal code), language (e.g., English or Spanish), device and its type (e.g., iPhone, Samsung, Motorola, LG, HP, or Dell), OS and its version (e.g., Windows, iOS, Android, Mac, or Linux), browser and its version (e.g., Firefox, Safari, Internet Explorer, or Chrome), application name and its version (e.g., in-app integrated chat), agent type (e.g., bot system or user device), etc. In some examples, custom events from custom parts can have custom reports developed by the robot developers.

[0312] The interpretability of data for machine learning models is one of those aspects that is crucial to the actual 'usefulness' of the data pipeline, and interpretability ensures that the model is consistent with the problem that the chatbot is trying to solve. In some instances, metrics used for interpretability and usefulness analysis can include: (i) the size of the training dataset, such as word2vec vectors, which may contain gender bias or have too small a corpus width; (ii) the context of the model, such as training data that only roughly represents the problem and fails to capture the full complexity of the real-life task; (iii) weights acquired after training that are direct proxies of feature importance and provide a very specific interpretation of the model's internals, for example, when building a text classifier, the most important features can be plotted and used to verify whether the model is overfitting to noise; (iv) gradients of the target concept computed by backward inference can be used to generate maps highlighting important regions in the input used to predict the target concept; and (v) a specific set of inputs, such as utterances, which can be modified and their impact on prediction can be monitored.

[0313] Accuracy, log loss, confusion matrix, area under the curve, F-score, mean absolute error, and mean squared error are those metrics that are crucial to the actual 'performance' of a data pipeline, and these ensure that the model solves the problem accurately and efficiently. Classification accuracy is a metric commonly used to evaluate model performance. Classification accuracy is the ratio of the number of correctly predicted cases to the total number of input samples. Log loss (Logarithmic Loss / Log Loss) works by penalizing misclassifications. Log loss works well for multi-class classification. When used with log loss, the classifier must assign probabilities to each class for all samples. Generally, minimizing the log loss provides higher accuracy for the classifier. The confusion matrix provides a matrix as output and describes the complete performance of the model. The confusion matrix forms the basis for other types of metrics. The area under the curve is one of the most widely used metrics for evaluation. The area under the curve can be used for binary classification problems. The area under the curve of a classifier is equal to the probability that the classifier will rank randomly selected positive examples higher than randomly selected negative examples. The F-score, or F1 score, is used to measure test accuracy. The F-score is the harmonic mean between precision and recall. The F-score ranges from [0, 1] and provides information about how precise the classifier is (how many instances it correctly classified) and how robust it is (how many instances it doesn't miss). Mean Absolute Error (MAO) is the average of the differences between the original and predicted values. MAO provides a measure of the distance between the prediction and the actual output. Mean Squared Error (MSE) is similar to MAO, except that it takes the average of the squares of the differences between the original and predicted values. The advantage of MSE is that gradients are easier to compute, while MAO requires complex linear programming tools to calculate gradients.

[0314] At step 725, the analysis engine of the analysis system determines, based on analysis, whether the performance of the second model is improved compared to the performance of the first model. In some instances, the analysis engine determines that the performance of the second model is improved compared to the performance of the first model. In other instances, the analysis engine determines that the performance of the second model is not improved compared to the performance of the first model. At 730, when the performance of the second model is improved relative to the first model, the directed acyclic graph-based framework of the integrated computing system executes the second model for the chatbot during runtime. In other words, the directed acyclic graph-based framework of the integrated computing system uses the second model instead of the first model to start during runtime because the second model performs better than the first model. In contrast, when the performance of the second model is not improved relative to the first model, the directed acyclic graph-based framework of the integrated computing system executes the first model for the chatbot during runtime. In other words, the directed acyclic graph-based framework of the integrated computing system continues to use the first model instead of the second model during runtime because the first model performs better than the second model. Although only two models are discussed regarding the process of flowchart 700, it should be understood that other models can be implemented and evaluated based on the description provided herein (e.g., one model can be run in runtime, while the total of two or more models is run in the background for evaluating model performance).

[0315] As described herein, an analytics system integrated with a robotic system and a DAG-based framework can collect metrics during model usage and can be used to generate scores for each model, enabling decisions about which model to use at runtime (e.g., replacing the original runtime model with a more robust or accurate model that operates at design time). For example, metrics can provide information for making decisions about which pipeline for a task yields the most efficient model (e.g., considering accuracy, how many queries per second it can process, complexity, etc.). In some embodiments, clients can use training tools to edit / change the vectors and features of a model at a finer granularity compared to overall edits / changes to the model structure. In some embodiments, one or more graphical user interfaces (GUIs) of the analytics system and training tools can display information related to the DAG-based framework, such as available tasks or actions, training corpora, pre-configured model or task pipelines, and analytics including collected metrics. In some embodiments, the GUI can be used by clients to build or modify one or more training models using the DAG-based framework.

[0316] In some embodiments, reports may be generated based on the analysis of attributes, and may include information indicating one or more utterances or messages from one or more end users whose intent cannot be identified (sometimes referred to as unresolved intents). For example, the bot system may calculate the probability that a message from an end user is associated with an intent. If the probability is less than a threshold, the message may not be associated with an intent. If the message is not associated with any intent, the bot system may not be able to continue the conversation. Instead, the bot system may have to ask one or more additional questions to identify the intent. By presenting information about messages in which the intent is unresolved, the analysis and / or reporting can enable users or Spark-like frameworks to reconfigure the bot system's model to correctly identify intents when receiving new messages similar to the message. For example, the report may present one or more potential intents based on probability, allowing the user to select an intent from said one or more potential intents, thereby allowing the message to be added to a training dataset used to train a classification model for identifying intents from messages.

[0317] In some embodiments, the robot analytics system can identify which parts of a session with the robot system are functioning well and which are not. The robot analytics system can use users or Spark-like frameworks to delve into the session history, tracking abandoned / completed intents and sessions, identifying the most popular / least popular paths taken by completed paths based on depth, time, or both, or using copies to identify the history of all abandoned sessions to troubleshoot the reasons for abandoned sessions (e.g., the number of states traversed, error conditions, etc.) or the cause of a pipeline failure. In some embodiments, the results generated by the robot analytics system can be filtered. Filtering can be based on channels, length, intent (abandoned / completed), etc.

[0318] Explanatory System

[0319] Figure 8 A simplified diagram of a distributed system 800 is depicted. In the illustrated example, the distributed system 800 includes one or more client computing devices 802, 804, 806, and 808 coupled to a server 812 via one or more communication networks 810. The client computing devices 802, 804, 806, and 808 can be configured to execute one or more applications.

[0320] In various examples, server 812 may be adapted to run one or more services or software applications implementing one or more embodiments described in this disclosure. In some examples, server 812 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided as web-based services or cloud services (such as under a Software as a Service (SaaS) model) to users of client computing devices 802, 804, 806, and / or 808. Users operating client computing devices 802, 804, 806, and / or 808 may then use one or more client applications to interact with server 812 to utilize the services provided by these components.

[0321] exist Figure 8 In the depicted configuration, server 812 may include one or more components 818, 820, and 822 that implement the functions performed by server 812. These components may include software components that can be executed by one or more processors, hardware components, or a combination thereof. It should be understood that various different system configurations, different from distributed system 800, are possible. Therefore, Figure 8 The example shown is an example of a distributed system for implementing the example system and is not intended to be restrictive.

[0322] Users can use client computing devices 802, 804, 806, and / or 808 to execute one or more applications, models, or chatbots, which can generate one or more events or models that can then be implemented or serviced according to the teachings of this disclosure. The client device can provide an interface that enables users of the client device to interact with it. The client device can also output information to the user through this interface. Although Figure 8 It describes only four client computing devices, but can support any number of client computing devices.

[0323] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Windows). ® Apple Macintosh ® UNIX ® Or UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome™ OS), including various mobile operating systems (e.g., Microsoft Windows Mobile). ® (iOS®, Windows Phone®, Android™, BlackBerry®, PalmOS®). Portable handheld devices can include cellular phones, smartphones (e.g., iPhone). ® ), tablet computers (e.g., iPad) ® Wearable devices include personal digital assistants (PDAs), etc. Google Glass is another example. ® Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices and internet-enabled gaming devices (e.g., with or without Kinect). ® Microsoft Xbox gesture input device ® Game consoles, Sony PlayStation® systems, various game systems provided by Nintendo®, and others. Client devices can be able to run a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, Short Message Service (SMS) applications), and can use various communication protocols.

[0324] One or more networks 810 can be any type of network familiar to those skilled in the art that supports data communication using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. By way of example only, one or more networks 810 can be a Local Area Network (LAN), an Ethernet-based network, a Token Ring, a Wide Area Network (WAN), the Internet, a Virtual Network, a Virtual Private Network (VPN), an Intranet, an Extranet, a Public Switched Telephone Network (PSTN), an Infrared Network, or a Wireless Network (e.g., according to the IEEE 1002.11 protocol suite, Bluetooth). ® (and / or any other wireless protocol operating network) and / or any combination of these networks and / or other networks.

[0325] Server 812 may consist of: one or more general-purpose computers, special-purpose server computers (including, by way of example, PC (personal computer) servers, UNIX servers, etc.) ® Servers (including servers, mid-range servers, mainframe computers, rack servers, etc.), server groups, server clusters, or any other suitable arrangement and / or combination. Server 812 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization (such as logical storage devices that can be virtualized into one or more flexible pools of virtual storage devices that maintain the server). In various examples, server 812 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0326] The computing system in server 812 can run one or more operating systems, including any of those discussed above and any commercially available server operating system. Server 812 can also run any of a variety of other server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, and Java applications. ® Servers, database servers, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM®, etc.

[0327] In some implementations, server 812 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client computing devices 802, 804, 806, and 808. As an example, data feeds and / or event updates may include, but are not limited to, Twitter. ® Feedback, Facebook ® The server 812 may receive real-time updates from one or more third-party information sources and continuous data streams. These real-time updates may include real-time events related to sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. The server 812 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client computing devices 802, 804, 806, and 808.

[0328] The distributed system 800 may also include one or more data repositories 814, 816. In some examples, these data repositories may be used to store data and other information. For example, one or more of data repositories 814, 816 may be used to store information (such as information related to chatbot performance or generated models) for use by the chatbot, which is used by server 812 in performing various functions according to various embodiments. Data repositories 814, 816 may reside in various locations. For example, the data repository used by server 812 may be local to server 812 or may be located remotely to server 812 and communicate with server 812 via a network-based connection or a dedicated connection. Data repositories 814, 816 may be of different types. In some examples, the data repository used by server 812 may be a database, such as a relational database, such as a database provided by Oracle Corporation® and other vendors. One or more of these databases may be adapted to perform data storage, updating, and retrieval from the database in response to commands in SQL format.

[0329] In some examples, one or more of data stores 814 and 816 can also be used by the application to store application data. The data store used by the application can be of different types, such as a key value store, an object store, or a general-purpose storage store supported by a file system.

[0330] In some examples, the functionality described in this disclosure can be provided as a service through a cloud environment. Figure 9 This is a simplified block diagram of a cloud-based system environment in which various services, based on certain examples, can be provided as cloud services. Figure 9 In the depicted example, cloud infrastructure system 902 can provide one or more cloud services that can be requested by users using one or more client computing devices 904, 906, and 908. Cloud infrastructure system 902 may include one or more computers and / or servers, which may include those described above with respect to server 812. The computers in cloud infrastructure system 902 may be organized as general-purpose computers, dedicated server computers, server clusters, server groups, or any other suitable arrangement and / or combination.

[0331] One or more networks 910 can facilitate data communication and exchange between clients 904, 906, and 908 and cloud infrastructure system 902. One or more networks 910 may include one or more networks. Networks can be the same or different types. One or more networks 910 may support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0332] Figure 9 The example depicted is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other examples, the cloud infrastructure system 902 can have more than... Figure 9 The depicted components may have more or fewer components, may combine two or more components, or may have different component configurations or settings. For example, although... Figure 9 Three client computing devices are depicted, but in alternative examples, any number of client computing devices can be supported.

[0333] The term cloud service is generally used to refer to services that become available to users on demand through a service provider's systems (e.g., cloud infrastructure systems 902) and via communication networks such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's systems differ from the customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. Therefore, customers can utilize cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's systems can host applications, and users can order and use the applications on demand via the Internet without having to purchase the infrastructure resources to run the applications. Cloud services are designed to provide convenient, scalable access to applications, resources, and services. Several vendors provide cloud services. For example, several cloud services are provided by Oracle Corporation® in Redwood Shores, California, such as middleware services, database services, Java Cloud services, and others.

[0334] In some examples, cloud infrastructure system 902 may use different models, such as Software as a Service (SaaS) model, Platform as a Service (PaaS) model, Infrastructure as a Service (IaaS) model, and other models (including hybrid service models), to provide one or more cloud services. Cloud infrastructure system 902 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0335] The SaaS model enables applications or software to be delivered as a service to customers via communication networks such as the Internet, without requiring customers to purchase the hardware or software used in the underlying applications. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by cloud infrastructure systems. Examples of SaaS services provided by Oracle Corporation® include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0336] The IaaS model is typically used to provide customers with infrastructure resources (such as servers, storage, hardware, and networking resources) as cloud services to offer elastic computing and storage capabilities. Various IaaS services are provided by Oracle Corporation®.

[0337] The PaaS model is typically used to provide a platform and environment of resources as a service that enables customers to develop, run, and manage applications and services without requiring them to purchase, build, or maintain such resources. Examples of PaaS services provided by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solutions, and other services.

[0338] Cloud services are typically delivered in an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may subscribe to one or more services provided by cloud infrastructure system 902 via a subscription order. Cloud infrastructure system 902 then performs processing to provide the service requested in the customer's subscription order. For instance, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent) as described above and / or to provide services for a chatbot system as described herein. Cloud infrastructure system 902 can be configured to provide one or more cloud services.

[0339] Cloud infrastructure system 902 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 902 can be owned by a third-party cloud service provider and the cloud services can be provided to any general public customer, which can be an individual or a business. In some other examples, under a private cloud model, cloud infrastructure system 902 can operate within an organization (e.g., within a business organization) and the services can be provided to customers within the organization. For example, customers can be various departments within the enterprise (such as human resources, payroll, etc.) or even individuals within the enterprise. In some other examples, under a community cloud model, cloud infrastructure system 902 and the services provided can be shared by several organizations in the relevant community. Various other models, such as hybrids of the models mentioned above, can also be used.

[0340] Client computing devices 904, 906, and 908 can be of different types (e.g. Figure 8 The client computing devices 802, 804, 806, and 808 depicted may be capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 902, such as requesting services provided by the cloud infrastructure system 902. For example, a user can use the client device to request information or actions from a chatbot as described in this disclosure.

[0341] In some examples, the processing performed by the cloud infrastructure system 902 used to provide the service may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, potentially processing data in parallel, using data to perform simulations, etc. For example, big data analysis may be performed by the cloud infrastructure system 902 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or constructed according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0342] like Figure 9 As depicted in the examples, cloud infrastructure system 902 may include infrastructure resources 930 used to facilitate the provision of various cloud services offered by cloud infrastructure system 902. Infrastructure resources 930 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to service storage requested from an application may be part of cloud infrastructure system 902. In other examples, the storage virtual machine may be part of a different system.

[0343] In some examples, to facilitate the efficient provisioning of these resources to support the various cloud services offered by the cloud infrastructure system 902 to different customers, resources can be bound to resource groups or resource modules (also known as "clusters (pods)"). Each resource module or cluster can include a pre-integrated and optimized combination of one or more types of resources. In some examples, different clusters can be pre-provisioned for different types of cloud services. For example, a first set of clusters can be provisioned for a database service, and a second set (which may include a different combination of resources than the clusters in the first set) can be provisioned for a Java service, and so on. For some services, resources allocated for provisioning the service can be shared between services.

[0344] The cloud infrastructure system 902 itself can internally use services 932 shared by different components of the cloud infrastructure system 902 and that facilitate the provision of services by the cloud infrastructure system 902. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0345] Cloud infrastructure systems 902 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. Figure 9 The depicted subsystem may include a user interface subsystem 912 that enables users or clients of the cloud infrastructure system 902 to interact with the cloud infrastructure system 902. The user interface subsystem 912 may include various interfaces, such as a web interface 914, an online store interface 916 (where advertisements are displayed and customers can purchase cloud services provided by the cloud infrastructure system 902), and other interfaces 918. For example, a customer may use a client device to request (service request 934) one or more services provided by the cloud infrastructure system 902 through one or more of interfaces 914, 916, and 918. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 902, and place a subscription order for one or more services provided by the cloud infrastructure system 902 that the customer wishes to subscribe to. A service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 902. As part of the order, the customer may provide information identifying the chatbot system to which the service will be provided and optionally provide one or more credentials for the chatbot system.

[0346] In some examples (such as) Figure 9 In the described example, cloud infrastructure system 902 may include an order management subsystem (OMS) 920 configured to process new orders. As part of this process, OMS 920 may be configured to: create accounts for customers (if not yet completed); receive invoices and / or accounting information from customers to be used to invoice customers for the requested services; verify customer information; after verification, book customer orders; and plan various workflows to prepare orders for supply.

[0347] Once properly verified, OMS 920 can invoke the Order Provisioning Subsystem (OPS) 924, configured as order provisioning resources (including processing, storage, and network resources). Provisioning may include allocating and configuring resources to facilitate the service requested by the customer order. The manner in which resources are provisioned to the order and the type of resources provided may depend on the type of cloud service the customer has subscribed to. For example, according to a workflow, OPS 924 may be configured to determine the specific cloud service being requested and identify the number of clusters that may already be pre-configured for that specific cloud service. The number of clusters allocated to the order may depend on the size / volume / tier / scope of the requested service. For example, the number of clusters to be allocated may be determined based on the number of users the service is to support, the duration of the requested service, etc. The allocated clusters can then be customized for the specific requesting customer to provide the requested service.

[0348] In some examples, the setup phase processing described above can be performed by the cloud infrastructure system 902 as part of the provisioning process. The cloud infrastructure system 902 can generate an application ID and select a storage virtual machine for the application from the storage virtual machines provided by the cloud infrastructure system 902 itself or from storage virtual machines provided by other systems besides the cloud infrastructure system 902.

[0349] Cloud infrastructure system 902 may send a response or notification 944 to the requesting client to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) enabling the client to begin using and taking advantage of the benefits of the requested service may be sent. In some examples, for the requesting client, the response may include a chatbot system ID generated by cloud infrastructure system 902 and information identifying the chatbot system selected by cloud infrastructure system 902 for the chatbot system corresponding to the chatbot system ID.

[0350] Cloud infrastructure system 902 can provide services to multiple customers. For each customer, cloud infrastructure system 902 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 902 can also collect usage statistics about customers' use of subscribed services. For example, it can collect statistics such as storage usage, data transfer volume, number of users, system uptime, and system downtime. This usage information can be used to bill customers. Billing can be done, for example, on a monthly basis.

[0351] Cloud infrastructure system 902 can provide services to multiple customers in parallel. Cloud infrastructure system 902 can store information about these customers, which may include proprietary information. In some examples, cloud infrastructure system 902 includes an Identity Management Subsystem (IMS) 928 configured to manage customer information and provide separation of the managed information so that information related to one customer cannot be accessed by another customer. IMS 928 can be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0352] Figure 10 An example of computer system 1000 is illustrated. In some examples, computer system 1000 can be used to implement any digital assistant or chatbot system within a distributed environment, as well as the various servers and computer systems described above. Figure 10 As shown, the computer system 1000 includes various subsystems, including a processing subsystem 1004 that communicates with multiple other subsystems via a bus subsystem 1002. These other subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 may include non-transitory computer-readable storage media, including storage medium 1022 and system memory 1010.

[0353] Bus subsystem 1002 provides a mechanism for enabling the various components and subsystems of computer system 1000 to communicate with each other as intended. Although bus subsystem 1002 is schematically shown as a single bus, alternative examples of bus subsystems may utilize multiple buses. Bus subsystem 1002 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (which may be implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0354] Processing subsystem 1004 controls the operation of computer system 1000 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Processors may include single-core or multi-core processors. The processing resources of computer system 1000 may be organized into one or more processing units 1032, 1034, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 1004 may include one or more dedicated coprocessors such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of processing subsystem 1004 may be implemented using custom circuitry such as ASICs or FPGAs.

[0355] In some examples, the processing units in processing subsystem 1004 can execute instructions stored in system memory 1010 or on computer-readable storage medium 1022. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in system memory 1010 and / or reside on computer-readable storage medium 1022 potentially included on one or more storage devices. With suitable programming, processing subsystem 1004 can provide the various functions described above. In an instance where computer system 1000 executes one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0356] In some examples, a processing acceleration unit 1006 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 1004, thereby accelerating the overall processing performed by the computer system 1000.

[0357] I / O subsystem 1008 may include devices and mechanisms for inputting information to and / or outputting information from or through computer system 1000. Generally, the term input device is intended to include all possible types of devices and mechanisms for inputting information to computer system 1000. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens incorporated into a display, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect, which enables users to control and interact with the input devices. ® Motion sensor, Microsoft Xbox ® The 360 ​​game controller provides an interface for receiving input via gestures and spoken commands. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translating the eye gestures into the input device (such as Google Glass). ® Google Glass input ® Blink detector. Additionally, user interface input devices may include those enabling users to interact with voice commands and voice recognition systems (e.g., Siri). ® A voice recognition sensing device for interaction with navigators.

[0358] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointing sticks, game controllers and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0359] Generally, the term "output device" is intended to encompass all possible types of devices and mechanisms for outputting information from computer system 1000 to a user or other computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tubes (CRTs), flat panel devices (such as those using liquid crystal displays (LCDs) or plasma displays), projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, sound output devices, and modems.

[0360] Storage subsystem 1018 provides a repository or data repository for storing information and data used by computer system 1000. Storage subsystem 1018 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Storage subsystem 1018 may store software (e.g., programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 1004. The software may be executed by one or more processing units of processing subsystem 1004. Storage subsystem 1018 may also be certified in accordance with the teachings of this disclosure.

[0361] The storage subsystem 1018 may include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. For example... Figure 10 As shown, the storage subsystem 1018 includes system memory 1010 and computer-readable storage medium 1022. System memory 1010 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines, such as those that help transfer information between elements within computer system 1000 during startup, may typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by processing subsystem 1004. In some embodiments, system memory 1010 may include various different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0362] By using examples rather than restrictions, such as Figure 10 As depicted, system memory 1010 can load running application programs 1012 (which may include various applications such as web browsers, middleware applications, relational database management systems (RDBMS), etc.), program data 1014, and operating system 1016. By way of example, operating system 1016 may include various versions of Microsoft Windows. ® Apple Macintosh ® and / or Linux operating system, various commercially available UNIX systems ® Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google Chrome) ® OS, etc.) and / or such as iOS, Windows ® Phone, Android ® OS, BlackBerry ® OS, Palm ® OS operating system and other mobile operating systems.

[0363] Computer-readable storage medium 1022 can store programming and data structures that provide some example functionality. Computer-readable storage medium 1022 can provide computer system 1000 with storage for computer-readable instructions, data structures, program modules, and other data. Software (programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 1004 can be stored in storage subsystem 1018. By way of example, computer-readable storage medium 1022 may include, for example, hard disk drives, disk drives, optical disc drives (such as CD ROMs, DVDs, Blu-ray discs). ® Non-volatile memory such as disks or other optical media. Computer-readable storage medium 1022 may include, but is not limited to, Zip... ® Drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 1022 may also include SSDs based on non-volatile memory such as flash memory-based solid-state drives (SSDs), enterprise-class flash memory drives, solid-state ROMs, etc.; SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, etc.; DRAM-based SSDs; magnetoresistive RAM (MRAM) SSDs; and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0364] In some examples, storage subsystem 1018 may also include a computer-readable storage medium reader 1020 that can be further connected to computer-readable storage medium 1022. Reader 1020 may receive data from and be configured to read data from storage devices such as disks, flash drives, etc.

[0365] In some examples, computer system 1000 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 1000 may provide support for executing one or more virtual machines. In some examples, computer system 1000 may execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing (e.g., processor, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. Virtual machines typically run their own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 1000. Therefore, multiple operating systems may potentially be run simultaneously by computer system 1000.

[0366] The communication subsystem 1024 provides interfaces to other computer systems and networks. The communication subsystem 1024 serves as an interface for receiving data from other systems and transmitting data from computer system 1000 to other systems. For example, the communication subsystem 1024 enables computer system 1000 to establish communication channels to one or more client devices via the Internet for receiving and sending information to client devices. For example, when computer system 1000 is used to implement... Figure 1 When the robot system 120 is described, the communication subsystem can be used to communicate with a chatbot system selected for the application.

[0367] The communication subsystem 1024 may support both wired and / or wireless communication protocols. In some examples, the communication subsystem 1024 may include radio frequency (RF) transceiver components, GPS receiver components, and / or other components for accessing wireless voice and / or data networks (e.g., advanced data network technologies such as cellular phone technologies such as 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 802.XX Home Standard, or other mobile communication technologies, or any combination thereof)). In some examples, in addition to or as an alternative to a wireless interface, the communication subsystem 1024 may provide wired network connectivity (e.g., Ethernet).

[0368] The communication subsystem 1024 can receive and transmit data in various forms. In some examples, among other forms, the communication subsystem 1024 can also receive input communications in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. For example, the communication subsystem 1024 can be configured to receive (or send) data feeds 1026 in real time from users of social media networks and / or other communication services, such as Twitter. ® Feedback, Facebook ® Updates, web feeds (such as rich site summary (RSS) feeds) and / or real-time updates from one or more third-party information sources.

[0369] In some examples, the communication subsystem 1024 may be configured to receive data in the form of an inherently continuous or unbounded continuous data stream that may not have an explicit end, said continuous data stream may include real-time event stream 1028 and / or event update 1030. Examples of applications that generate continuous data may include, for example, sensor data applications, financial reporting machines, network performance measurement tools (e.g., network detection and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0370] The communication subsystem 1024 can also be configured to transmit data from computer system 1000 to other computer systems or networks. The data can be transmitted in various forms, such as structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, to one or more databases that can communicate with one or more streaming data source computers coupled to computer system 1000.

[0371] Computer system 1000 can be of various types, including handheld portable devices (e.g., iPhone). ® Cellular phone, iPad ® Computing tablets, PDAs, and wearable devices (e.g., Google Glass) ® Head-mounted displays, personal computers, workstations, mainframes, self-service kiosks, server racks, or any other data processing systems. Due to the constantly evolving nature of computers and networks, [the following is relevant / important]. Figure 10 The description of the computer system 1000 is intended only as a concrete example. It has a higher... Figure 10 Many other configurations with more or fewer components are possible for the system depicted. Based on this disclosure and the teachings provided herein, it should be understood that there are other ways and / or methods to implement the various examples.

[0372] While specific examples have been described, various modifications, alterations, alternative constructions, and equivalents are possible. The examples are not limited to operations in a particular data processing environment but are free to operate in multiple data processing environments. Furthermore, although certain examples have been described using specific series of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be restrictive. While some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. Processes may have additional steps not included in the diagrams. Various features and aspects of the examples described above can be used individually or in combination.

[0373] Furthermore, while certain examples have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. Some examples may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein can be implemented on the same or different processors in any combination.

[0374] When a device, system, component, or module is described as being configured to perform certain operations or functions, this configuration can be accomplished, for example, by electronic circuitry designed to perform the operations, by programming programmable electronic circuitry (such as a microprocessor) to perform the operations (e.g., by executing computer instructions or code), or by programming a processor or core to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes can communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0375] Specific details are set forth in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description is merely illustrative and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the foregoing description of the examples will provide those skilled in the art with enabling descriptions for implementing the various examples. Various changes can be made to the function and arrangement of the elements.

[0376] Therefore, the specification and drawings should be viewed in an illustrative rather than restrictive sense. However, it will be apparent that additions, omissions, deletions, and other modifications and alterations may be made therein without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be restrictive. Various modifications and equivalents are within the scope of the following claims.

[0377] In the foregoing description, various aspects of this disclosure have been described with reference to specific examples; however, those skilled in the art will recognize that this disclosure is not limited thereto. Various features and aspects of the disclosure described above may be used individually or in combination. Furthermore, the examples may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0378] In the foregoing description, the methods are described in a specific order for illustrative purposes. It should be understood that, in alternative examples, the methods may be performed in a different order than described. It should also be understood that the methods described above may be executed by hardware components or may be embodied in a sequence of machine-executable instructions that can be used to cause a machine (such as a general-purpose or special-purpose processor or logic circuit programmed with said instructions) to execute the methods. These machine-executable instructions may be stored on one or more machine-readable media such as CD-ROMs or other types of optical discs, floppy disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the methods may be executed by a combination of hardware and software.

[0379] When a component is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuitry or other hardware for performing the operations, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) for performing the operations, or any combination thereof.

[0380] While illustrative examples of this application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in other ways, and the appended claims are intended to be construed as including such variations, except where limited by the prior art.< / string> < / string> < / string> < / string> < / jsonobject> < / jsonobject>

Claims

1. A method for a framework based on directed acyclic graphs, comprising: A first model and a second model are generated using a directed acyclic graph-based framework of an integrated computing system. The first model is a pipeline of a first set of tasks for performing one or more operations associated with the chatbot, and the second model is a pipeline of a second set of tasks for performing the one or more operations associated with the chatbot. The first set of tasks differs from the second set of tasks in that at least one task is added or removed, at least one task is replaced, the order in which at least one task is processed, or a combination thereof. The integrated computing system executes the first model for the chatbot during runtime and the second model for the chatbot during design time using the directed acyclic graph-based framework. While the chatbot is running the first model and the second model, one or more attributes for intent classification associated with a set of utterances are collected by the event collector of the integrated computing system; The integrated computing system's analytics engine uses one or more metrics to evaluate the performance of the first model and the second model based on the analysis of one or more attributes used for the intent classification. Based on the evaluation, the analysis engine determines that the performance of the second model is not improved compared to the performance of the first model; and The first model for the chatbot continues to execute during runtime through the directed acyclic graph-based framework of the integrated computing system.

2. The method of claim 1, further comprising: The pipeline of the first model is displayed graphically on a GUI; And receive user selections for one or more user-selectable tasks through the GUI; And based on the user selection, the first group of tasks in the pipeline, which has one or more user-selectable tasks, is displayed graphically on the GUI.

3. The method of claim 1, further comprising: The pipeline of the second model is displayed graphically on a GUI; And receive user selections for one or more user-selectable tasks through the GUI; And, based on the user selection, the second group of tasks is displayed graphically on the GUI in the pipeline, which has one or more user-selectable tasks.

4. The method of claim 1, further comprising: Receive user input through one or more user options; The first model and the second model are trained based on the user input, wherein the user input is a set of utterances that the user considers to trigger an intent.

5. The method of claim 1, wherein, The first model, running during runtime, executes on the dataset to generate outputs to be used by the chatbot in downstream processes, wherein the downstream processes include providing dialogue or taking action based on the intent classification, and wherein the second model, running in the background during design time, executes on the same dataset to generate different outputs that are not used by the chatbot in the downstream processes.

6. The method of claim 1, wherein, The execution of the first model and the second model includes obtaining a dataset containing the set of utterances from one or more channels or obtaining the dataset containing the set of utterances from a database, and parsing intent based on the set of utterances using the first model and the second model.

7. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions including, when executed by the one or more processors, causing the one or more processors to perform a process comprising: The first and second models are generated using a directed acyclic graph-based framework of the integrated computing system, wherein... The first model is a pipeline of a first set of tasks for performing one or more operations associated with the chatbot, and the second model is a pipeline of a second set of tasks for performing the one or more operations associated with the chatbot, wherein the first set of tasks differs from the second set of tasks in that at least one task is added or removed, at least one task is replaced, the order in which at least one task is processed, or a combination thereof. The integrated computing system executes the first model for the chatbot during runtime and the second model for the chatbot during design time using the directed acyclic graph-based framework. While the chatbot is running the first model and the second model, one or more attributes for intent classification associated with a set of utterances are collected by the event collector of the integrated computing system; The integrated computing system's analytics engine uses one or more metrics to evaluate the performance of the first model and the second model based on the analysis of one or more attributes used for the intent classification. The analysis engine determines, based on the evaluation, that the performance of the second model is not improved compared to the performance of the first model; and the first model for the chatbot continues to be executed during runtime via the directed acyclic graph-based framework of the integrated computing system.

8. The non-transitory computer-readable storage device as claimed in claim 7, wherein, The process further includes: graphically displaying the pipeline of the first model on a GUI; receiving user selections for one or more user-selectable tasks through the GUI; and graphically displaying the first group of tasks in the pipeline having the one or more user-selectable tasks based on the user selections on the GUI.

9. The non-transitory computer-readable storage device as claimed in claim 7, wherein, The process further includes: graphically displaying the pipeline of the second model on a GUI; receiving user selections for one or more user-selectable tasks through the GUI; and graphically displaying the second group of tasks in the pipeline having the one or more user-selectable tasks based on the user selections on the GUI.

10. The non-transitory computer-readable storage device as claimed in claim 7, wherein, The process further includes: receiving user input via one or more user options; and training the first model and the second model based on the user input, wherein the user input is a set of utterances that the user considers to trigger an intent.

11. The non-transitory computer-readable storage device as claimed in claim 7, wherein, The first model, running during runtime, executes on the dataset to generate outputs to be used by the chatbot in downstream processes, wherein the downstream processes include providing dialogue or taking action based on the intent classification, and wherein the second model, running in the background during design time, executes on the same dataset to generate different outputs that are not used by the chatbot in the downstream processes.

12. The non-transitory computer-readable storage device as claimed in claim 7, wherein, The execution of the first model and the second model includes obtaining a dataset containing the set of utterances from one or more channels or obtaining the dataset containing the set of utterances from a database, and parsing intent based on the set of utterances using the first model and the second model.

13. A system for a framework based on directed acyclic graphs, comprising: One or more processors; as well as A memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions including, when executed by the one or more processors, causing the one or more processors to perform processing including the following operations: A first model and a second model are generated using a directed acyclic graph-based framework of an integrated computing system. The first model is a pipeline of a first set of tasks for performing one or more operations associated with the chatbot, and the second model is a pipeline of a second set of tasks for performing the one or more operations associated with the chatbot. The first set of tasks differs from the second set of tasks in that at least one task is added or removed, at least one task is replaced, the order in which at least one task is processed, or a combination thereof. The integrated computing system executes the first model for the chatbot during runtime and the second model for the chatbot during design time using the directed acyclic graph-based framework. While the chatbot is running the first model and the second model, one or more attributes for intent classification associated with a set of utterances are collected by the event collector of the integrated computing system; The integrated computing system's analytics engine uses one or more metrics to evaluate the performance of the first model and the second model based on the analysis of one or more attributes used for the intent classification. Based on the evaluation, the analysis engine determines that the performance of the second model is not improved compared to the performance of the first model; and The first model for the chatbot continues to execute during runtime through the directed acyclic graph-based framework of the integrated computing system.

14. The system of claim 13, wherein, The process further includes: graphically displaying the pipeline of the first model on a GUI; receiving user selections for one or more user-selectable tasks through the GUI; and graphically displaying the first group of tasks in the pipeline having the one or more user-selectable tasks based on the user selections on the GUI.

15. The system of claim 13, wherein, The process further includes: graphically displaying the pipeline of the second model on a GUI; receiving user selections for one or more user-selectable tasks through the GUI; and graphically displaying the second group of tasks in the pipeline having the one or more user-selectable tasks based on the user selections on the GUI.

16. The system of claim 13, wherein, The process further includes: receiving user input via one or more user options; and training the first model and the second model based on the user input, wherein the user input is a set of utterances that the user considers to trigger an intent.

17. The system of claim 13, wherein, The first model, running during runtime, executes on the dataset to generate outputs to be used by the chatbot in downstream processes, wherein the downstream processes include providing dialogue or taking action based on the intent classification, and wherein the second model, running in the background during design time, executes on the same dataset to generate different outputs that are not used by the chatbot in the downstream processes.

18. The system of claim 13, wherein, The execution of the first model and the second model includes obtaining a dataset containing the set of utterances from one or more channels or obtaining the dataset containing the set of utterances from a database, and parsing intent based on the set of utterances using the first model and the second model.

Images