Objective function optimization in target-based hyperparameter tuning

Abstract

A technique for optimizing an objective function in target-based hyperparameter tuning is disclosed herein. In one embodiment, a computer-implemented method is provided, comprising: initializing a machine learning algorithm with a set of hyperparameter values; and obtaining a hyperparameter objective function, the hyperparameter objective function including a domain score for each domain calculated based on the number of instances in an evaluation dataset that are predicted correctly or incorrectly by the machine learning algorithm during a given trial. For each trial of the hyperparameter tuning process, the machine learning algorithm is trained to generate a machine learning model, and the machine learning model is run in different domains using the set of hyperparameter values. The machine learning model is evaluated for each domain, and at least one machine learning model is output when the machine learning model reaches convergence.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Nonprovisional Patent Application No. 18 / 197,224, filed May 15, 2023, which claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 405,981, filed September 13, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] Field The present disclosure relates generally to machine learning techniques, and more particularly to techniques for objective function optimization in target-based hyperparameter tuning. [Background technology]

[0003] background Artificial intelligence has many applications. For example, many users around the world engage with instant messaging or chat platforms for instant responses. Organizations often use these instant messaging or chat platforms to engage with customers (or end users) in live conversations. However, it can be very costly for organizations to employ service personnel to participate in live communications with customers or end users. Chatbots or bots are beginning to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with bots through messaging apps that the end users already have installed and use. Intelligent bots, typically equipped with artificial intelligence (AI), can communicate more intelligently and contextually in live conversations, thus enabling more natural conversations between bots and end users for an improved conversational experience. Instead of end users learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots may be able to understand the end user's intent based on user utterances in natural language and respond accordingly.

[0004] However, artificial intelligence-based solutions such as chatbots can be difficult to build because many automated solutions require specific knowledge in a domain and the application of certain techniques that may be within the capabilities of specialized developers. For example, as part of building such a chatbot, a developer may first understand the needs of the enterprise and end users. The developer may then perform an analysis, for example, selecting a dataset to be used in the analysis, preparing the input dataset for the analysis (e.g., cleansing the data before analysis, extracting, formatting, and / or transforming the data, data feature engineering, etc.), identifying an appropriate machine learning (ML) technique or model to perform the analysis, and making decisions regarding improving the technique or model to improve results / outcomes based on feedback. The task of identifying an appropriate model may involve developing multiple models, possibly in parallel, and iteratively testing and experimenting with these models before identifying a specific model (or models) to use. Furthermore, supervised learning-based solutions typically involve a training phase, followed by an application (i.e., inference) phase, and an iterative loop between the training and application phases. The developer may be responsible for carefully refining and monitoring these stages to achieve an optimal solution. For example, to train an ML technique or model, precise training data is needed to enable the algorithm to understand and learn certain patterns or features (e.g., for a chatbot, not just raw language processing, but intent extraction and careful syntactic parsing) that the ML technique or model will use to predict a desired outcome (e.g., inferring intent from an utterance). To ensure that the ML technique or model properly learns these patterns and features, the developer may be responsible for selecting, augmenting, and optimizing the training data and set of hyperparameters for the ML technique or model. Summary of the Invention [Problem to be solved by the invention]

[0005] overview The techniques disclosed herein relate generally to machine learning techniques. More specifically, and without limitation, the techniques disclosed herein relate to objective function optimization in target-based hyperparameter tuning.

[0006] In various embodiments, a computer-implemented method includes initializing a machine learning algorithm with a set of hyperparameter values; and accessing a hyperparameter objective function defined at least in part for a plurality of domains of a search space associated with the machine learning algorithm, the search space including a training dataset and an evaluation dataset, each domain including a subdivision of the search space having at least one training dataset and at least one evaluation dataset, the hyperparameter objective function including a domain score for each domain calculated based on a number of instances in the at least one evaluation dataset that are predicted correctly or incorrectly by the machine learning algorithm during a given trial, the computer-implemented method including, for each trial of the hyperparameter tuning process, initializing a machine learning algorithm for each domain using at least one training dataset associated with each domain and the set of hyperparameter values. wherein the training outputs a plurality of machine learning models, including a machine learning model for each domain; the computer-implemented method further includes, for each trial of the hyperparameter tuning process, evaluating the machine learning model for each domain using at least one evaluation dataset and a set of hyperparameter values ​​associated with each domain, the evaluating including generating a domain score for each domain; the computer-implemented method further includes: calculating, for each trial of the hyperparameter tuning process, an objective score for a current trial based on the domain score for each domain and a domain weight associated with each domain using a hyperparameter objective function; determining whether the machine learning models have reached convergence based on the objective score for the current trial; and providing at least one of the plurality of machine learning models in response to determining that the machine learning models have reached convergence.

[0007] In some embodiments, the hyperparameter objective function is formulated to normalize the instance-level improvement and instance-level regression over the total number of instances to obtain improvement scores and regression scores, and a domain score for each domain is calculated based on the improvement scores and regression scores.

[0008] In some embodiments, the improvement score is calculated based on a total count of (i) the number of instances in the at least one evaluation dataset that are correctly predicted by the machine learning model during a given trial, (ii) the number of instances in the at least one evaluation dataset that are incorrectly predicted by the baseline machine learning model, and (iii) the number of instances in the at least one evaluation dataset, and the regression score is calculated based on (i) the number of instances in the at least one evaluation dataset that are incorrectly predicted by the machine learning model during a given trial, (ii) the number of instances in the at least one evaluation dataset that are correctly predicted by the baseline machine learning model, and (iii) the number of instances in the at least one evaluation dataset.

[0009] In some embodiments, the hyperparameter objective function is formulated to exclude unstable instances, which are instances in at least one evaluation dataset that have been determined to produce different predicted outcomes from each other using the same machine learning model.

[0010] In some embodiments, filtering out unstable instances includes (i) subtracting a count of the unstable instances to be filtered out from a number of instances in at least one evaluation dataset that are correctly predicted by the machine learning model during a given trial, and (ii) subtracting a count of the unstable instances to be filtered out from a number of instances in at least one evaluation dataset that are incorrectly predicted by the machine learning model during a given trial.

[0011] In some embodiments, a hyperparameter objective function is formulated to include parameters for one or more of the domains of acceptable regression ratios m.

[0012] In some embodiments, the parameter is defined based on the regression score, and if the regression score is smaller than an acceptable regression ratio m, the regression score is set to zero; otherwise, the regression score is calculated based on a total count of (i) the number of instances in at least one evaluation dataset that are incorrectly predicted by the machine learning model during a given trial, (ii) the number of instances in at least one evaluation dataset that are correctly predicted by the baseline machine learning model, and (iii) the number of instances in the at least one evaluation dataset.

[0013] In various embodiments, a system is provided that includes one or more processors and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform some or all of one or more methods disclosed herein.

[0014] In various embodiments, one or more non-transitory computer-readable media are provided for storing instructions that, when executed by one or more processors, cause the system to perform some or all of one or more of the methods disclosed herein.

[0015] The techniques described above and below can be implemented in multiple ways and in multiple contexts. Some exemplary implementations and contexts are provided with reference to the accompanying drawings, as described in more detail below. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0016] [Figure 1]FIG. 1 is a simplified block diagram of a distributed environment incorporating an illustrative embodiment. [Figure 2] FIG. 1 is a simplified block diagram of a computing system implementing a masterbot according to one embodiment. [Figure 3] FIG. 1 is a simplified block diagram of a computing system implementing a skillbot according to one embodiment. [Figure 4A] FIG. 1 illustrates exemplary types of hyperparameters according to various embodiments. [Figure 4B] FIG. 1 illustrates exemplary types of metrics according to various embodiments. [Figure 4C] FIG. 1 illustrates an exemplary set of specifications associated with metrics according to various embodiments. [Figure 5] FIG. 1 illustrates a hyperparameter tuning system in accordance with various embodiments. [Figure 6] FIG. 1 is a flow diagram illustrating a training process performed by a hyperparameter tuning system according to various embodiments. [Figure 7] FIG. 1 is a flow diagram illustrating a validation process performed by an exemplary hyperparameter tuning system in accordance with various embodiments. [Figure 8] FIG. 1 is a simplified diagram of a tuning workflow according to various embodiments. [Figure 9] FIG. 1 is a flow diagram illustrating objective function optimization and tuning performed by a hyperparameter tuning system according to various embodiments. [Figure 10] FIG. 1 is a simplified diagram of a distributed system for implementing various embodiments. [Figure 11] FIG. 1 is a simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment system can be provided as cloud services, according to various embodiments. [Figure 12]FIG. 1 illustrates an exemplary computer system that can be used to implement various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0017] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of certain embodiments of the invention. It will be understood, however, that various embodiments may be practiced without these specific details. The drawings and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0018] introduction Artificial intelligence has many uses. For example, digital assistants are artificial intelligence-driven interfaces that use natural language conversation to help users accomplish various tasks. For each digital assistant, customers can assemble one or more skills. Skills (also described herein as chatbots, bots, or skillbots) are individual bots specialized for specific types of tasks, such as tracking inventory, submitting time cards, and creating expense reports. When an end user engages with a digital assistant, the digital assistant evaluates the end-user input and interacts with the appropriate chatbot to route the conversation. Digital assistants can be made available to end users through various channels, such as FACEBOOK® Messenger, SKYPE MOBILE® Messenger, or short message service (SMS). Channels route chats from the end user to the digital assistant and its various chatbots and back again over various messaging platforms. Channels may also support user-agent escalation, event-triggered conversations, and testing.

[0019] When creating models (e.g., machine learning models) used to perform various tasks for an application (e.g., a digital assistant), developers will be presented with choices for how to define the model architecture and implement the model's learning process. Often, developers do not immediately know which optimal model architecture or learning process is for a given model, and therefore they desire to be able to explore various possibilities. Typically, developers will ask a computing device or subsystem, such as a hyperparameter tuner, to perform this search and automatically select the optimal model architecture and learning process. Parameters that define the model architecture and learning process are referred to as hyperparameters, and the search process for searching for the ideal model architecture and learning process is referred to as hyperparameter tuning. Hyperparameters are used to improve model learning, and their values ​​are set before starting the model's learning process. Hyperparameters are generally tuned by defining a model and a learning process (including selecting an initial set of values ​​for the hyperparameters that define the architecture and learning process), defining ranges of possible values ​​for the hyperparameters, defining a method for sampling the hyperparameter values, defining an evaluation metric for evaluating the model, such as aggregate accuracy, running the model on training data to learn a set of model parameters, evaluating the performance of the learned model based on the evaluation metric, and adjusting the values ​​of the hyperparameters accordingly using the method for sampling hyperparameter values. Searching for the best hyperparameters can be tedious, so tuning algorithms such as grid search and random search are used to sample hyperparameter values.

[0020] A drawback of such standard hyperparameter tuning algorithms is that they ignore other important goals (e.g., regression error) during the optimization process. Additionally, standard hyperparameter tuning mechanisms only consider a single domain (each domain having a training dataset and an evaluation dataset) to train / evaluate a machine learning model. Some hyperparameter tuning algorithms consider multiple target domains while evaluating the model, but each of these domains may not have the same level of importance in training the machine learning model. Additionally, standard hyperparameter tuning algorithms fail to adequately address the problem of instance-level regression. Individual training / validation / test cases in training / validation / test datasets or other datasets are considered instances. Instance-level regression may mean that an instance that was correctly classified by a previous version of a model is incorrectly classified by a later version of the model. Conversely, instance-level improvement may mean that an instance that was incorrectly classified by a previous version of a model is correctly classified by a later version of the model.

[0021] Standard hyperparameter tuning algorithms may not be able to detect instance-level regression or improvement.For example, both the first version model and the second version model can correctly classify 600 instances in a dataset with a total of 1000 instances.However, 200 of the instances correctly classified by the first model are incorrectly classified by the second model.Similarly, 200 instances incorrectly classified by the first model are correctly classified by the second model.Therefore, there are 200 instance-level improvements and 200 instance-level regressions between the first model and the second model.

[0022] A standard hyperparameter tuning objective function may evaluate a model using an accuracy score (e.g., accuracy rate). In this example, both the first and second models would have an accuracy score of 60% (e.g., 600 correct instances / 1000 total instances), and the standard algorithm would not detect the instance-level regression. Undetected instance-level regression may mean that the two models behave differently after deployment. Instance-level regression may cause a loss of perceived accuracy for customers using the model because the new model behaves differently despite consistent accuracy scores. Furthermore, in some situations, instance-level regression may be caused by a data point (e.g., an example from the training set) that causes unstable prediction results (i.e., predictions that are not the same or similar across multiple runs). The unstable prediction results may occur because the data point is located near the decision boundary. The data point may cause a standard hyperparameter tuning model to chase random noise during later stages of the tuning process, which may cause instance-level regression.

[0023] Therefore, a different approach is needed to address these and other challenges. The present disclosure provides hyperparameter tuning systems and techniques that optimize an objective function while considering multiple metrics at once, i.e., performing multi-objective optimization. Each metric is assigned a weight that represents the level of importance of the metric to the performance of the machine learning model. The hyperparameter tuning systems and techniques also provide for tuning hyperparameters while considering multiple different domains of varying levels of importance. Specifically, each domain is assigned a weight that specifies the importance of the domain in training the machine learning model. Additionally, the hyperparameter tuning systems and techniques allow one or more constraints to be defined for the machine learning model being trained. The training infrastructure (also referred to herein as a hyperparameter tuning system) utilizes various automated techniques to automatically identify, set, and tune hyperparameters for training a model such that the trained model complies with and satisfies the constraints specified for the model.

[0024] Additionally, the objective function in the hyperparameter tuning system can be modified to address instance-level regression. Rather than calculating an accuracy score, the hyperparameter tuning system can calculate instance-level regression or improvement by cross-referencing sets of correctly or incorrectly classified instances generated by multiple different models. The objective function can be further modified to exclude unstable instances from the set of correctly or incorrectly classified instances. By excluding unstable instances, the objective function is optimized during tuning without chasing random noise caused by unstable instances. In addition, not all domains may be equally important to customers (e.g., based on business use). In this case, the objective function can be further modified to include an acceptable regression ratio for each domain. Typically, the lower the acceptable regression ratio, the more important the domain. For example, if the acceptable regression ratio is set to zero for a domain, this means that no regression is allowed for that domain. In this way, the hyperparameter tuning system of the present disclosure provides a single machine learning model that works across multiple different domains and multiple different metrics while minimizing instance-level regression.

[0025] In an exemplary embodiment, a computer-implemented method is provided that includes initializing a machine learning algorithm with a set of hyperparameter values ​​and accessing a hyperparameter objective function defined at least in part for multiple domains of a search space associated with the machine learning algorithm, the search space including a training dataset and an evaluation dataset, each domain including a subdivision of the search space having at least one training dataset and at least one evaluation dataset, and the hyperparameter objective function including a domain score for each domain calculated based on the number of instances in the at least one evaluation dataset that are predicted correctly or incorrectly by the machine learning algorithm during a given trial. For each trial of the hyperparameter tuning process, the computer-implemented method includes: training a machine learning algorithm for each domain using at least one training dataset and a set of hyperparameter values ​​associated with each domain (the training outputs a plurality of machine learning models, including a machine learning model for each domain); evaluating the machine learning algorithm for each domain using at least one evaluation dataset and a set of hyperparameter values ​​associated with each domain (the evaluation includes generating a domain score for each domain); calculating an objective score for the current trial based on the domain score for each domain and the domain weight associated with each domain using a hyperparameter objective function; and determining whether the machine learning model has reached convergence based on the objective score for the current trial. In response to determining that the machine learning model has reached convergence, providing at least one of the plurality of machine learning models.

[0026] Bot System A bot (also referred to as a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. Bots can generally respond to natural language messages (e.g., questions or comments) through messaging applications that use natural language messages. Businesses can use one or more bots to communicate with end users through messaging applications. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), and mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat capabilities or voice-based input (such as Siri, Cortana, Google Voice, or devices or apps with interfaces that use other voice input for interaction).

[0027] In some examples, a bot may be associated with a Uniform Resource Identifier (URI). A URI may identify a bot using a string of characters. A URI may be used as a webhook for one or more messaging application systems. A URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). A bot may be designed to receive a message (e.g., a Hypertext Transfer Protocol (HTTP) POST call message) from a messaging application system. The HTTP POST call message may be directed from the messaging application system to a URI. In some examples, the message may be different from an HTTP POST call message. For example, a bot may receive a message via Short Message Service (SMS). While the discussion herein refers to a communication received by a bot as a message, it should be understood that the message may be an HTTP POST call message, an SMS message, or any other type of communication between two systems.

[0028] End users interact with bots through conversational interactions (sometimes referred to as conversational user interfaces (UIs)), just as end users interact with other people. In some cases, a conversational interaction may include the end user saying "Hello" to the bot, and the bot responding "Hi" and asking the end user how it can help. End users also interact with bots through other types of interactions, such as transactional interactions (e.g., with a banking bot trained to transfer money from at least one account to another), informational interactions (e.g., with an HR bot trained to check at least how much vacation time a user has remaining), and / or retail interactions (e.g., with a retail bot trained to discuss returning a purchased item or request technical support).

[0029] In other examples, a bot can intelligently handle end-user interactions without intervention by the bot's administrator or developer. For example, an end user may send one or more messages to a bot to achieve a desired goal. The messages may include content such as text, emojis, audio, images, videos, or other methods of conveying a message. In some examples, the bot can automatically convert the content into a standardized format and generate a natural language response. The bot can also automatically prompt the end user for additional input parameters or request other additional information. In some examples, the bot can also initiate communication with the end user rather than passively responding to end-user utterances.

[0030] A conversation with a bot may follow a specific conversational flow that includes multiple states. The flow can specify what happens next based on input. In some examples, a bot can run using a state machine that includes user-defined states (e.g., end-user intent) and actions to take in the states or from state to state. The conversation can take different paths based on end-user input, which can influence the decisions the bot makes for the flow. For example, in each state, based on the end-user input or utterance, the bot can determine the end-user's intent to determine the appropriate action to take next. As used herein and in the context of utterances, the term "intent" refers to the intent of the user who provided the utterance. For example, a user may intend to engage a bot in a conversation to order a pizza, where the user's intent is expressed through the utterance "order pizza." The user's intent can be directed to a specific task the user wants the bot to perform on their behalf. Thus, an utterance reflecting a user's intent can be phrased as a question, a command, a request, etc.

[0031] In the context of bot configuration, the term “intent” is also used herein to refer to configuration information for mapping a user's utterance to a specific task / action or category of task / action that the bot can perform. To distinguish between an utterance intent (e.g., a user's intent) and a bot's intent, the latter may be referred to herein as a “bot intent.” A bot intent may include a set of one or more utterances associated with the intent. For example, an intent to order a pizza may have various permutations of an utterance expressing a desire to place a pizza order. These associated utterances can be used to train the bot's intent classifier, enabling the intent classifier to subsequently determine whether an input utterance from a user matches the intent to order a pizza. A bot intent may be associated with one or more dialog flows to initiate a conversation with a user in a state. For example, the first message of an intent to order a pizza may be the question, “What kind of pizza would you like?” In addition to the associated utterance, a bot intent may further include a named entity associated with the intent. For example, an intent to order a pizza may include variables or parameters used to perform the task of ordering a pizza (e.g., topping 1, topping 2, pizza type, pizza size, quantity of pizza, etc.) The values ​​of the entities are typically obtained through conversation with the user.

[0032] FIG. 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to one embodiment. The environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users 104 of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as shown in FIG. 1, a user 104 representing a particular business can use the DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, the DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform can be used by multiple businesses to create digital assistants. As another example, a restaurant (e.g., a pizza shop) owner may use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0033] For purposes of this disclosure, a "digital assistant" is a tool that helps a user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital tool that runs using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or executed in various physical systems or devices, such as a computer, a mobile phone, a watch, an appliance, a vehicle, etc. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0034] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and get responses 112 back from the digital assistant 106. A conversation can include one or more of the inputs 110 and responses 112. Through these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested task and respond to the user with an appropriate response.

[0035] User input 110 is generally in the form of natural language and is referred to as speech. User utterance 110 can be in text form, such as when a user types in a sentence, a question, a text fragment, or even a single word and provides it as input to the digital assistant 106. In some examples, user utterance 110 can be in the form of acoustic input or utterance, such as when a user says or speaks something that is provided to the digital assistant 106 as input. The utterance is typically in the language that the user speaks. For example, the utterance may be in English or some other language. When the utterance is in the form of an utterance, the utterance input is converted into text form speech in that particular language, and the text speech is then processed by the digital assistant 106. Various utterance-to-text processing techniques may be used to convert the utterance or acoustic input into text speech that is then processed by the digital assistant 106. In some examples, the utterance-to-text conversion may be performed by the digital assistant 106 itself.

[0036] The utterance, which may be a text utterance or a spoken utterance, can be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing of the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, including identifying one or more intents and one or more entities that correspond to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 can perform one or more actions or behaviors in response to the understood meaning or intent. For purposes of this disclosure, it is assumed that the utterance is either provided directly by a user of the digital assistant 106 or is a text utterance that is the result of converting an input spoken utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0037] For example, user input can request that a pizza be ordered by providing an utterance such as, "I want to order a pizza." Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action may include, for example, responding to the user with a question requesting user input regarding the type of pizza the user desires to order, the size of the pizza, any toppings on the pizza, etc. The responses provided by digital assistant 106 may also be in natural language format and typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 may perform natural language generation (NLG). For a user ordering a pizza, through a conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the information required for the pizza order so that the pizza is ordered at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0038] At a conceptual level, the digital assistant 106 performs various processes in response to utterances received from a user. In some examples, this processing involves a sequence or pipeline of processing steps, including, for example, understanding the meaning of the input utterance, determining an action to be performed in response to the utterance, causing the action to be performed, if appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing can include analyzing the received input utterance to understand the structure and meaning of the utterance, refining and reshaping the utterance to develop a better understandable form (e.g., logical form) or structure of the utterance, etc. Generating a response may include using NLG techniques.

[0039] NLU processing performed by a digital assistant such as digital assistant 106 can include various NLP-related tasks such as sentence analysis (e.g., tokenization, lemmatization, identifying utterance part tags for sentences, identifying named entities within sentences, generating dependency trees to represent sentence structure, dividing sentences into clauses, analyzing individual clauses, resolving anaphora, performing chunking, etc.). In some examples, NLU processing is performed by digital assistant 106 itself. In some other examples, digital assistant 106 may use other resources to perform portions of NLU processing. For example, the syntax and structure of input spoken sentences can be identified by processing the sentences using a parser, utterance part tagger, and / or NER. In one embodiment, for the English language, a parser, utterance part tagger, and named entity recognizer, such as those provided by the Stanford NLP Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0040] While various examples provided in this disclosure show speech in the English language, this is intended as an example only. In some examples, the digital assistant 106 is also capable of handling speech in languages ​​other than English. The digital assistant 106 can provide subsystems (e.g., components that perform NLU functions) configured to perform processing for multiple different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from an NLU core server. This makes NLU processing flexible and extensible for each language, including allowing for different orders of processing. Language packs can be provided for individual languages, and the language packs can register a list of subsystems that can be serviced by the NLU core server.

[0041] A digital assistant, such as the digital assistant 106 shown in FIG. 1, can be available or accessible to its user 108 through a variety of different channels, including, but not limited to, through an application, through a social media platform, through various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for it, such that the digital assistant can be powered by and accessed on multiple different services simultaneously.

[0042] A digital assistant or chatbot system generally encompasses or is associated with one or more skills. In one embodiment, these skills are individual chatbots (referred to as skillbots) that are configured to interact with a user and perform specific types of tasks, such as tracking inventory, submitting a timecard, creating an expense report, ordering food, checking a bank account, making a reservation, purchasing a widget, etc. For example, in the embodiment shown in FIG. 1 , the digital assistant or chatbot system 106 includes skills 116-1, 116-2, 116-3, etc. For purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skillbot” and “skillbots,” respectively.

[0043] Each skill associated with a digital assistant helps a user of the digital assistant complete a task through a conversation with the user, where the conversation may include a combination of text or audio input provided by the user and responses provided by a skill bot. These responses may be in the form of text or audio messages to the user and / or may use simple user interface elements (e.g., selection lists) presented to the user for the user to make a selection.

[0044] There are various ways in which skills or skill bots can be associated with or added to a digital assistant. In some cases, skill bots can be developed by a company and then added to a digital assistant using DABP 102. In some cases, skill bots can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In still other cases, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills targeted at a wide range of tasks. Skills offered through the skill store may also expose various cloud services. To add a skill to a digital assistant being created using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and indicate that the selected skill should be added to the digital assistant being created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in a modified form (e.g., a user of DABP102 may select and clone a particular skillbot provided by the skill store, make customizations or modifications to the selected skillbot, and then add the modified skillbot to a digital assistant created using DABP102).

[0045] A variety of different architectures may be used to implement a digital assistant or chatbot system. For example, in one embodiment, a digital assistant created and deployed using DABP 102 may be implemented using a masterbot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a masterbot that interacts with one or more child bots, which are skillbots. For example, in the embodiment shown in FIG. 1, the digital assistant 106 includes a masterbot 114 and skillbots 116-1, 116-2, e.g., that are child bots of the masterbot 114. In one example, the digital assistant 106 itself may act as the masterbot.

[0046] A digital assistant implemented according to a master-child bot architecture allows a user of the digital assistant to interact with multiple skills through a unified user interface, i.e., via a master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself. If not, the master bot selects a suitable skill bot to handle the user request and routes the conversation to the selected skill bot. This allows a user to interact with the digital assistant through a common, single interface while still providing the ability to use several skill bots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the digital assistant's master bot may interface with skill bots having specific functions, such as a customer relationship management (CRM) bot for performing functions related to customer relationship management, an enterprise resource planning (ERP) bot for performing functions related to enterprise resource planning, a human resources management (HCM) bot for performing functions related to human capital management, etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master bot interface, and behind the scenes, multiple skill bots are provided to handle user requests.

[0047] In one example, in a masterbot / childbot infrastructure, the masterbot is configured to recognize an available list of skillbots. The masterbot has access to various available skillbots and, for each skillbot, metadata identifying the skillbot's capabilities, including tasks that can be performed by the skillbot. Upon receiving a user request in the form of an utterance, the masterbot is configured to identify or predict a specific skillbot from multiple available skillbots that can best service or address the user request. The masterbot then routes the utterance (or a portion of the utterance) to that specific skillbot for further handling. Thus, control flows from the masterbot to the skillbot. The masterbot can support multiple input and output channels. In one example, routing may be performed with the assistance of processing performed by one or more available skillbots. For example, as described below, a skillbot can be trained to infer the intent of an utterance and determine whether the inferred intent matches an intent for which the skillbot is configured. Thus, routing performed by the masterbot may involve the skillbot communicating to the masterbot an indication of whether the skillbot is configured with the appropriate intent to address the utterance.

[0048] 1 illustrates a digital assistant 106 with a masterbot 114 and skillbots 116-1, 116-2, and 116-3, but this is not intended to be limiting. A digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an embodiment using a combination of software and hardware.

[0049] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants, including one or more skillbots associated with the digital assistant. In some cases, a skillbot can be created by cloning an existing skillbot, for example, by cloning a skillbot provided by a skill store. As indicated above, DABP 102 provides a skill store or skill catalog that provides multiple skillbots for performing various tasks. A user of DABP 102 can clone a skillbot from the skill store. Modifications or customizations can be made to the cloned skillbot as needed. In some other cases, a user of DABP 102 created a skillbot from scratch using tools and services provided by DABP 102. As indicated above, a skill store or skill catalog provided by DABP 102 can provide multiple skillbots for performing various tasks.

[0050] In one example, at a high level, creating or customizing a skillbot involves the following steps: (1) Configure the settings for the new skill bot. (2) Configure one or more intents for the skillbot. (3) Constitute one or more entities of one or more intentions. (4) Train the skill bot. (5) Create a dialog flow for the skill bot. (6) Add custom components to your skill bot as needed. (7) Test and deploy the skill bot. Each of the above steps is briefly described below.

[0051] (1) Configure Settings for a New Skillbot—Various settings can be configured for a skillbot. For example, a skillbot designer can specify one or more invocation names for the skillbot being created. These invocation names can then be used by users of the digital assistant to explicitly invoke the skillbot. For example, a user can enter the invocation name in the user's utterance to explicitly invoke the corresponding skillbot.

[0052] (2) Configure one or more intents and associated example utterances for the skillbot—A skillbot designer specifies one or more intents (also referred to as bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes that the skillbot is trained to infer for input utterances. Upon receiving an utterance, the trained skillbot infers the intent of the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes an appropriate action to respond to the utterance based on the inferred intent for that utterance. In some cases, the intents of the skillbot represent tasks that the skillbot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot being trained for banking, the intents specified for the skillbot may include “CheckBalance,” “TransferMoney,” “DepositCheck,” etc.

[0053] For each intent defined for a skillbot, the skillbot designer may also provide one or more example utterances that express and illustrate the intent. These example utterances are intended to represent utterances a user might input into the skillbot for that intent. For example, for a CheckBalance intent, example utterances may include, "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account," etc. Thus, various permutations of typical user utterances may be specified as example utterances for the intent.

[0054] The intents and their associated example utterances are used as training data to train the skill bot. A variety of different training techniques may be used. This training results in a predictive model being generated that is configured to receive an utterance as input and output an intent inferred by the predictive model for the utterance. In some cases, the input utterance is provided to an intent analysis engine that is configured to predict or infer the intent of the input utterance using the trained model. The skill bot can then take one or more actions based on the inferred intent.

[0055] (3) Configuring One or More Intent Entities for a Skill Bot—In some cases, additional context may be required for a skill bot to respond appropriately to a user utterance. For example, there may be situations in which a user input utterance resolves to the same intent within a skill bot. For example, in the above example, "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests asking about different things. To disambiguate such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "savings" can enable the skill bot to parse the user request and respond appropriately. In the above example, the utterances resolve to the same intent, but the values ​​associated with the AccountType entity are different for the two utterances. This allows the skill bot to perform potentially different actions for the two utterances, even though they resolve to the same intent. One or more entities may be specified for an intent configured for a skill bot. Thus, entities are used to add context to the intent itself: they help to more fully describe the intent and enable the skill bot to complete the user request.

[0056] In one example, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by a skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, repeating time period, currency, phone number, URL, etc. Custom entities are used for more customized purposes. For example, for a banking skill, an AccountType entity can be defined by the skill bot designer that enables various banking transactions by checking user input for keywords such as checking, savings, and credit cards.

[0057] (4) Train the Skillbot—The skillbot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select an intent associated with the received user input. As indicated above, the skillbot must be trained for this. In one embodiment, the skillbot is trained based on intents configured for the skillbot and example utterances associated with the intents (collectively, training data) so that the skillbot can resolve user input utterances into one of its configured intents. In one example, the skillbot is trained using the training data and uses a predictive model that enables the skillbot to determine what a user says (or in some cases, what they intend to say). DABP 102 provides a variety of different training techniques that can be used by the skillbot designer to train the skillbot, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In one example, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as a trained skill bot) can then be used to address and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for a skill bot. This enables the skill bot to output a response to a user request without having to update the dialog definition. A Q&A intent is created similarly to a regular intent. The dialog flow of a Q&A intent may differ from that of a regular intent.

[0058] (5) Create a Dialog Flow for the Skill Bot—The dialog flow specified for a skill bot describes how the skill bot reacts as its different intents are resolved in response to incoming user input. Dialog flow defines the actions or behaviors the skill bot performs, such as how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. Dialog flow is like a flowchart that the skill bot follows. Skill bot designers specify the dialog flow using a language such as Markdown. In one embodiment, a version of YAML called OBotML may be used to specify the dialog flow for a skill bot. The dialog flow definition for a skill bot acts as a model of the conversation itself, i.e., allowing the skill bot designer to orchestrate the interaction between the skill bot and the user it serves.

[0059] In one example, a skill bot's dialog flow definition contains the following three sections: (a) Context Section (b) Default transition section (c) Status section Context Section - The context section allows the skill bot designer to define variables that will be used within the conversation flow. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to recognize and assert user preferences, etc.

[0060] Default Transition Section - Transitions for a skill bot can be defined in the dialog flow state section or the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there are no applicable transitions defined in a state or when the conditions required to trigger a state transition cannot be met. The default transition section can be used to define routing that allows the skill bot to gracefully handle unexpected user behavior.

[0061] State Section - The dialog flow and its associated behavior are defined as a series of transition states that govern the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality needed at that point in the dialog. Thus, states are built around components. States encompass component-specific characteristics and define transitions to other states that are triggered after the component is executed.

[0062] The state section can be used to address special case scenarios. For example, there may be times within a digital assistant when you want to temporarily exit a first skill that the user is engaged in and provide the user with the option to do something in a second skill. For example, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), the user may want to jump to a banking skill (e.g., the user may want to ensure they have enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, you can configure an action within a first skill to initiate an interaction with a second, different skill within the same digital assistant and then return to the original flow.

[0063] (6) Adding Custom Components to a Skillbot—As described above, states specified in a skillbot's dialog flow nominate components that provide the required functionality for that state. Components enable a skillbot to perform its functions. In one embodiment, DABP 102 provides a set of pre-configured components for performing a wide range of functions. A skillbot designer can select one or more of these pre-configured components and associate them with states in the skillbot's dialog flow. A skillbot designer can also create custom or new components using tools provided by DABP 102 and associate the custom components with one or more states in the skillbot's dialog flow.

[0064] (7) Testing and Deploying Skillbots - DABP102 provides several features that allow skillbot designers to test the skillbots they are developing, which can then be deployed and included within a digital assistant.

[0065] While the above description describes how to create a skillbot, similar techniques may also be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without invoking a skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit, which applies when a user signals a desire to exit the current conversation or context within the digital assistant; (2) Help, which applies when a user seeks help or orientation; and (3) Unresolved Intent, which applies to user input that does not match well with the exit and help intents. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information allows the masterbot to select a specific skillbot to handle an utterance.

[0066] At the MasterBot or Digital Assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and associated conversation. The digital assistant makes this determination using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance should be routed to a specific skill for action, should be actioned by the digital assistant or MasterBot itself via a built-in system intent, or should be actioned as a different state within the current conversation flow.

[0067] In one embodiment, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skill bot corresponding to the invocation name. In such a scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further handling. In the absence of a specific or explicit invocation, in one embodiment, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and the skill bot associated with the digital assistant. The calculated score for the skill bot or system intent represents how likely the user input represents the task the skill bot is configured to perform or represents the system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further handling of the user input utterance. In one embodiment, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model of each skill), and a confidence score is determined for each intent. Generally, any intent with a confidence score above a threshold (e.g., 70%) is treated as a candidate intent. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0068] FIG. 2 is a simplified block diagram of a MasterBot (MB) system 200 according to one embodiment. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skillbot invoker 240, and a data store 250. The MB system 200 shown in FIG. 2 is only one example of an arrangement of components within a MasterBot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some other embodiments, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0069] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in a variety of modalities, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the audio to text using an utterance-to-text converter (not shown) that inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text.

[0070] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. The manner in which the utterance 202 is addressed depends on the language, as each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of the utterance.

[0071] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (e.g., VB), etc. The language parser 214 can also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and lemmatize the words. A lemma is the primary form of a set of words as represented in a dictionary (e.g., "run" is a lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking of compound expressions, such as combining "credit" and "card" into the single expression "credit card." The language parser 214 can also identify relationships between words within the utterance 202. For example, in some embodiments, language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., particular nouns) are direct objects, which parts of the utterance are prepositions, etc. The results of the processing performed by language parser 214 form extracted information 205, which, along with utterance 202 itself, is provided as input to MIS 220.

[0072] As indicated above, utterance 202 can include more than one sentence. For purposes of multiple-intent detection and explicit invocation, utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in some embodiments, preprocessing, such as by preprocessing subsystem 210, can be performed to identify single sentences among multiple sentences for multiple-intent analysis and explicit invocation analysis. In general, the results produced by MIS 220 and EIS 230 are substantially the same whether utterance 202 is processed at the individual sentence level or as a single unit containing multiple sentences.

[0073] The MIS 220 determines whether the utterance 202 expresses multiple intents. While the MIS 220 can detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not involve determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the processing to determine whether the intent of the utterance 202 matches a bot intent can be performed by the intent classifier 242 of the MB system 200 or by an intent classifier of a skill bot (e.g., as shown in FIG. 3 ). The processing performed by the MIS 220 assumes that there is a bot (e.g., a particular skill bot or the master bot itself) that can address the utterance 202. Thus, the processing performed by the MIS 220 does not require knowledge of which bots are in the chatbot system (e.g., the identities of the skill bots registered with the master bot) or which intents are configured for a particular bot.

[0074] To determine that utterance 202 contains multiple intents, MIS 220 applies one or more rules from a set of rules 252 in data store 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include a sentence pattern that indicates the presence of multiple intents. For example, the sentence pattern may include a coordinating conjunction joining two parts of a sentence (e.g., a conjunction), where both parts correspond to separate intents. If utterance 202 matches the sentence pattern, it can be inferred that utterance 202 expresses multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents targeted at different bots or different intents within the same bot). Instead, the utterance may have separate instances of the same intent (e.g., "Place a pizza order using payment account X, then place a pizza order using payment account Y").

[0075] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines which portions of utterance 202 are associated with each intent. For each intent representing an utterance that encompasses multiple intents, MIS 220 constructs a new utterance in place of the original utterance for separate processing, such as, for example, utterance “B” 206 and utterance “C” 208 as shown in FIG. 2 . Thus, original utterance 202 can be split into two or more separate utterances that are addressed at one time. MIS 220 determines which of the two or more utterances should be addressed first using extracted information 205 and / or from an analysis of utterance 202 itself. For example, MIS 220 may determine that utterance 202 includes a marker word indicating that a particular intent should be addressed first. The newly formed utterance corresponding to this particular intent (e.g., one of utterance 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (e.g., or has been temporarily put on hold), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) can then be sent to EIS 230 for processing.

[0076] The EIS 230 determines whether an utterance it receives (e.g., utterance 206 or utterance 208) includes a skillbot's invocation name. In one embodiment, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes the skillbot from other skillbots in the chatbot system. A list of invocation names may be maintained in the data store 250 as part of the skillbot information 254. When an utterance includes words that match the invocation name, the utterance is considered to be an explicit invocation. If the bot is not explicitly invoked, the utterance received by the EIS 230 is considered an implicit invocation utterance 234 and is input to the masterbot's intent classifier (e.g., intent classifier 242) to determine which bot to use to address the utterance. In some cases, the intent classifier 242 determines that the masterbot should address the implicit invocation utterance. In other cases, the intent classifier 242 determines which skillbot to route the utterance to for handling.

[0077] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that a masterbot needs to perform. For example, when there is an explicit call, the masterbot may not need to perform any intent classification analysis (e.g., using intent classifier 242) or a reduced intent classification analysis to select a skillbot. Thus, the explicit call analysis can enable the selection of a particular skillbot without using intent classification analysis.

[0078] There may also be situations where there is overlap in functionality among multiple skill bots. This can occur, for example, when the intents addressed by two skill bots overlap or are very close to each other. In such situations, it may be difficult for the master bot to identify which of the multiple skill bots to select based solely on intent classification analysis. In such scenarios, explicit invocation eliminates ambiguity about the specific skill bot to be used.

[0079] In addition to determining that an utterance is an explicit invocation, EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the skill bot being explicitly invoked. In particular, EIS 230 can determine that a portion of the utterance is not associated with an invocation. EIS 230 can make this determination through analysis of the utterance and / or analysis of extracted information 205. EIS 230 can send the portion of the utterance that is not associated with an invocation to the skill bot being invoked instead of sending the entire sentence received by EIS 230. In some cases, the input to the skill bot being invoked is formed simply by removing any portion of the utterance that is associated with an invocation. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using Pizza Bot" is related to the invocation of Pizza Bot but not to any processing that will be performed by Pizza Bot. In some cases, the EIS 230 can reformat the portion to be sent to the bot being invoked, for example, to form a complete sentence. Thus, the EIS 230 determines not only that there is an explicit invocation, but also what to send to the skill bot when there is an explicit invocation. In some cases, there may not be any text to be entered into the bot being invoked. For example, if the utterance was "Pizza Bot," the EIS 230 may determine that the Pizza Bot is being invoked, but there is no text to be processed by the Pizza Bot. In such a scenario, the EIS 230 can instruct the skill bot invoker 240 that there is nothing to send.

[0080] The skillbot invoker 240 invokes a skillbot in various manners. For example, the skillbot invoker 240 can invoke a bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invocation. The indication 235 can be sent by the EIS 230 along with input for the skillbot being explicitly invoked. In this scenario, the skillbot invoker 240 hands over control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response can be to perform a particular action or to start a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0081] Another manner in which the skillbot invoker 240 can invoke a skillbot is through implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 is trained for multiple different classes, one class for each skillbot. For example, whenever a new skillbot is registered with the masterbot, a list of example utterances associated with the new skillbot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. Parameters (e.g., a set of values ​​for the parameters of a machine learning model) produced as a result of this training can be stored as part of the skillbot information 254.

[0082] In one embodiment, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training the machine learning model may involve inputting at least a subset of utterances from example utterances associated with various skill bots and generating, as the output of the machine learning model, a guess as to which bot is the correct bot to address any particular training utterance. For each training utterance, an indication of the correct bot to use for the training utterance may be provided as ground truth information. The behavior of the machine learning model may then be adapted (e.g., through backpropagation) to minimize the difference between the guesses generated and the ground truth information.

[0083] In one embodiment, the intent classifier 242 determines a confidence score for each skill bot registered with the master bot, indicating the skill bot's likelihood of handling the utterance (e.g., an implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skill bot invoker 240 invokes the bot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either the identity of the system intent or the identity of a particular skill bot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain win margin. Imposing such a condition enables routing to a particular skill bot when the confidence scores of multiple skill bots each exceed the threshold confidence score value.

[0084] After a bot is identified based on the evaluation of the confidence score, the skillbot invoker 240 hands over processing to the identified bot. In the case of system intent, the identified bot is a masterbot. Otherwise, the identified bot is a skillbot. Furthermore, the skillbot invoker 240 determines what to provide as input 247 to the identified bot. As indicated above, in the case of explicit invocation, the input 247 can be based on a portion of the utterance that is not associated with the invocation, or the input 247 can be none (e.g., an empty string). In the case of implicit invocation, the input 247 can be the entire utterance.

[0085] The data store 250 includes one or more computing devices that store data used by various subsystems of the masterbot system 200. As described above, the data store 250 includes rules 252 and skillbot information 254. The rules 252 include, for example, rules for determining, by the MIS 220, when an utterance expresses multiple intents and how to split an utterance expressing multiple intents. The rules 252 further include rules for determining, by the EIS 230, which portions of an utterance that explicitly invokes a skillbot to send to the skillbot. The skillbot information 254 includes, for example, invocation names of skillbots in the chatbot system, such as a list of invocation names of all skillbots registered with a particular masterbot. The skillbot information 254 can also include information used by the intent classifier 242 to determine a confidence score for each skillbot in the chatbot system, such as, for example, parameters of a machine learning model.

[0086] 3 is a simplified block diagram of a Skillbot system 300 according to one embodiment. Skillbot system 300 is a computing system that can run solely in software, solely in hardware, or a combination of hardware and software. In one embodiment, such as the embodiment shown in FIG. 1, Skillbot system 300 can be used to run one or more Skillbots within a digital assistant.

[0087] Skillbot system 300 includes MIS 310, intent classifier 320, and conversation manager 330. MIS 310 is similar to MIS 220 of FIG. 2 and provides similar functionality, including being operable to use rules 352 in data store 350 to (1) determine whether an utterance expresses multiple intents, and if so, (2) determine how to split the utterance into separate utterances for each of the multiple intents. In one embodiment, the rules applied by MIS 310 to detect multiple intents and split the utterance are the same as those applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. Extracted information 304 is similar to extracted information 205 of FIG. 1 and can be generated using language parser 214 or a language parser local to Skillbot system 300.

[0088] The intent classifier 320 can be trained similarly to the intent classifier 242 described above in connection with the embodiment of FIG. 2 and as described in more detail herein. For example, in one embodiment, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of example utterances associated with that particular skill bot as training utterances. The ground truth for each training utterance is the particular bot intent associated with the training utterance.

[0089] The utterance 302 can be received directly from a user or can be provided through a masterbot. For example, if the utterance 302 is provided through a masterbot as a result of processing through the MIS 220 and the EIS 230 in the embodiment shown in FIG. 2 , the MIS 310 can be bypassed to avoid repeating processing already performed by the MIS 220. On the other hand, if the utterance 302 is received directly from a user, for example, during a conversation that occurs after routing to a skillbot, the MIS 310 can process the utterance 302 to determine whether the utterance 302 represents multiple intents. If so, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, such as, for example, utterance “D” 306 and utterance “E” 308. If the utterance 302 does not represent multiple intents, the MIS 310 forwards the utterance 302 to the intent classifier 320 for intent classification without splitting the utterance 302.

[0090] The intent classifier 320 is configured to match an incoming utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As explained above, a skillbot can be configured with one or more intents, each including at least one example utterance associated with the intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the masterbot system 200 is trained to determine a confidence score for each individual skillbot and a confidence score for the system intent. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skillbot system 300. While the classification performed by the intent classifier 242 is at the bot level, the classification performed by the intent classifier 320 is at the intent level and is therefore more granular. The intent classifier 320 can have access to intent information 354. For each intent associated with the skillbot system 300, the intent information 354 includes a list of utterances that represent the intent, indicate its meaning, and are typically associated with tasks that can be performed by the intent. The intent information 354 can further include parameters that result from training on this list of utterances.

[0091] The conversation manager 330 receives as an output of the intent classifier 320 an indication 322 of the particular intent identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some cases, the intent classifier 320 is unable to determine any match. For example, if the utterance is targeted to the system intent or the intent of a different skill bot, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value. When this occurs, the skill bot system 300 may refer the utterance to the master bot for action, for example, to route it to a different skill bot. However, if the intent classifier 320 is successful in identifying the intent within the skill bot, the conversation manager 330 begins a conversation with the user.

[0092] The conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute the dialog flow of the identified intent. The state machine may include a default starting state (e.g., the intent is invoked without any additional input) and one or more additional states, each associated with an action to be performed by the skill bot (e.g., completing a purchase transaction) and / or a dialog to be presented to the user (e.g., question, response). Thus, the conversation manager 330 can determine an action / dialog 335 upon receiving an instruction 322 identifying the intent, and can determine the additional action or dialog in response to subsequent utterances received during the conversation.

[0093] Data store 350 includes one or more computing devices that store data used by various subsystems of skillbot system 300. As shown in Figure 3, data store 350 includes rules 352 and intent information 354. In an embodiment, data store 350 can be integrated with a masterbot or digital assistant data store, such as, for example, data store 250 of Figure 2.

[0094] Target-Based Hyperparameter Tuning As mentioned above, artificial intelligence-based solutions can use one or more machine learning models to perform various functions. For example, a chatbot can use a machine learning model configured to take utterances as input and infer or predict the intent of each utterance. The intent inferred by the model for the utterance can then be used by the chatbot to determine how to respond to the utterance. Implementing a machine learning model (also referred to as a model) typically involves two stages: (1) a training stage, in which training data is run on one or more algorithms to create a trained model, and (2) an inference stage, in which the trained model is used to make predictions based on new data. A training infrastructure is generally provided to perform the training stage to train the model. The training infrastructure may be provided by a tool or application or by software used to perform the training. The training infrastructure is configured to run training data against one or more algorithms to train or learn the algorithms and create the model. The training infrastructure typically provides control over hyperparameters that govern this training process. The set of values ​​for the hyperparameters determines the network structure for the algorithm (e.g., number of input layers, number of hidden layers, activation function, etc.), how the algorithm is trained (e.g., learning rate, number of epochs, etc.), and any other hyperparameters (e.g., data augmentation setup, batch balancing setup, caching setup, etc.). The set of values ​​for the hyperparameters is determined by the training infrastructure using a hyperparameter tuning process or algorithm.

[0095] Hyperparameter tuning can involve optimizing the performance of a model over multiple domains by trying different values ​​for different hyperparameters. At the heart of hyperparameter tuning is a tuning objective function whose value can be optimized (e.g., maximized or minimized) during hyperparameter tuning. For hyperparameter tuning optimized over N domains, e.g., D={D_0,D_1,...,D_N}, the objective function can be defined as follows:

[0096] F_{tuning}=w_0*f(D_0)+w_1*f(D_i)+...+w_N*f(D_N) where f(D_i) (i=0,1,...,N) is the domain score calculated for each domain, and w_0, w_1,..., w_N are the domain weights to be applied to each domain score. The domain score is a measure of how well the model is performing (e.g., model accuracy or F1) for a given domain D_0, D_i, D_N, etc. The domain score can be defined as an evaluation score or target-based score of the trained model, and a baseline performance is set to understand the improvement or regression of the trained model against the baseline. For example:

[0097] Evaluation scores of trained models, such as accuracy of intent classification, bilingual evaluation understudy (BLEU) scores for machine translation, and F1 scores for image classification.

[0098] · Target-based scores – improvements and / or regressions compared to the baseline core for the domain.

[0099] Each hyperparameter tuning can run T trials, where a trial can have an objective value F_{tuning}_t calculated for any trial t, t = 0, 1, ..., T. The goal of hyperparameter tuning can be to find a trial with a set of hyperparameter assignments that maximizes or minimizes the objective function value, e.g., H_best = argmax_{t}F_{tuning}_t(t = 1, ..., T). Domain weights in the objective function, e.g., W = {w_0, w_1, ..., w_N}, can indicate the importance of each domain; domains with higher weights can receive more attention and are therefore more likely to perform better. Because the objective score is scaled by the weight value, a domain score with a larger domain weight will have a greater influence on the final value of the objective function than the same domain score with a smaller domain weight.

[0100] FIG. 4A illustrates exemplary types of hyperparameters according to various embodiments. Hyperparameters may include the number of layers in the model, the type of learning algorithm used to train the model, the learning rate, the number of training epochs, the number of hidden units in each layer, and the width of each hidden layer, i.e., the width of the units in each hidden layer. In some cases, a user using the training infrastructure manually sets the values ​​of the hyperparameters. However, this can be a very difficult task requiring very deep knowledge of the training process. As described next with reference to FIG. 5, a hyperparameter tuning system according to various embodiments is provided that is configured to automatically perform objective optimization, i.e., optimizing a function of one or more metrics (e.g., a loss function). As shown in FIG. 4B, the metrics may include a stability metric, a regression error metric, a confidence score metric, a model size metric, a training or execution time metric, an accuracy metric, or any combination thereof.

[0101] The stability metric ensures that the training process is stable, i.e., predictions made by the model do not fundamentally change when small changes are made to the training data (e.g., when one training example is added or removed). The regression error metric minimizes the number of regressions of the model. That is, regression error corresponds to the model incorrectly classifying some inputs, while previous versions of the model correctly classified those inputs. The confidence score metric ensures that the model predicts certain examples with high confidence (i.e., the model not only makes correct predictions, but does so with high confidence). The model size metric corresponds to the size of the trained model within a user-defined threshold, such as 130 megabytes, while the training time metric corresponds to the amount of time used to train the model. The accuracy metric ensures that the trained model achieves a certain user-defined level of accuracy, such as 125% accuracy, for a certain validation dataset. In other words, when a machine learning model is trained on a specific training dataset, a certain target accuracy is achieved for the specific validation dataset. It will be appreciated that training and validation datasets can be selected to span the full range of chatbot use cases, i.e., datasets ranging from very small to very large datasets across a variety of applications.

[0102] The hyperparameter tuning system of the present disclosure is configured to train a machine learning model for multiple domains (e.g., each domain consisting of a training dataset and an evaluation dataset) and evaluate the performance of the machine learning model with respect to one or more metrics. According to some embodiments, each of the datasets used to train and evaluate the machine learning model is assigned a domain weight that indicates the importance of the dataset in training and evaluating the machine learning model. In other words, the weight assigned to a dataset corresponds to the level of influence the dataset has on the training and evaluation of the machine learning model.

[0103] Additionally, the hyperparameter tuning system can be configured to assign a weight to each metric utilized in the objective optimization. Specifically, the weight assigned to a metric indicates the importance of the metric to the performance of the machine learning model. As described in more detail below, the assignment of weights to metrics and various domains can be performed according to one or more policies governing the hyperparameter tuning system.

[0104] In some cases, the hyperparameter tuning system also allows one or more constraints to be specified in the training of the machine learning model. Constraints can be requirements imposed on the machine learning model, i.e., qualities or characteristics that a user desires the trained model to achieve. Constraints can relate to the training process itself. Constraints can be specified before model training begins. Thus, for a given set of constraints, the training infrastructure utilizes various automated techniques to automatically identify, set, and tune hyperparameter values ​​for training the machine learning model such that the trained model conforms to and satisfies the set of constraints.

[0105] In some embodiments, the hyperparameter tuning system enables validating machine learning models against a wide range of validation / test datasets. The hyperparameter tuning system associates specific target values ​​with one or more metrics used to evaluate the performance of the machine learning model. A hyperparameter tuning objective function (e.g., a loss function) is constructed based on the target values ​​for the one or more metrics. In certain cases, as described in more detail below with reference to FIG. 5, the hyperparameter tuning system utilizes an asymmetric loss mechanism to assign weights to different domains and / or metrics in evaluating the machine learning model (i.e., not meeting the target is heavily penalized compared to reward cases for meeting or exceeding the target).

[0106] Hyperparameter Tuning System 5, a hyperparameter tuning system according to various embodiments is shown. The hyperparameter tuning system 500 includes a dataset weight assignment unit 510, a metric selection and weight assignment unit 520, a constraint establishment unit 530, and a hyperparameter tuner 550. The hyperparameter tuner 550 includes an optimizer 551 (also referred to herein as a tuning unit) and a set of hyperparameters 555.

[0107] The hyperparameter tuning system 500 is configured to train a machine learning model (e.g., a model associated with a chatbot such as those described in connection with FIGS. 1-3 ) on multiple datasets (i.e., training datasets) and evaluate the performance of the machine learning model based on one or more metrics. The dataset weight assignment unit 510 captures one or more domains, e.g., Domain 1 505A, Domain 2 505B, and Domain K 505C. Each domain represents a knowledge base that a skill bot can use for natural language conversation about a particular topic (e.g., a PizzaBot can use a first domain to discuss ordering pizza, a FlightBookingBot can use a second domain to help users book flights, a FinancialBot can use a third domain to answer financial questions, and an InsuranceBot can use a fourth domain to provide insurance quotes), and assigns a weight to each domain according to one or more policies 515. The weights assigned to the domains correspond to the importance of the domains in training the machine learning model. Assigning different weights to different domains allows the domains to have an appropriate influence (i.e., according to their respective weights) on the training of the machine learning model.

[0108] As an example, one of the policies 515 may require the machine learning model to achieve a low regression relative to baseline performance. Accordingly, the dataset weight assignment unit 510 assigns a higher weight to a regression dataset (e.g., dataset 1 505A) than to another type of domain. As another example, domain 1 505A may correspond to a domain obtained from a first client of the hyperparameter tuning system 500, and domain 2 505B may correspond to a domain obtained from a second client different from the first client. The training data included in domain 1 and domain 2 may correspond to different types of user utterances (provided by the respective clients) in terms of context. Assuming that one of the policies 515 indicates that the first client is more important than the second client (e.g., the first client has a higher service level agreement (SLA) with the hyperparameter tuning system 500), the dataset corresponding to the first client, i.e., domain 1 505A, may be assigned a higher weight than the weight of domain 2 505B. Moreover, it is appreciated that a system administrator of hyperparameter tuning system 500 determines policy 515 prior to training a machine learning model. Weighted domains 505 may be provided as a first input to hyperparameter tuner 550. Additionally, it is appreciated that a system administrator of hyperparameter tuning system 500 may define a default or general policy in which all domain weights are initialized to the same value or level, e.g., all domain weights=1.

[0109] The metric selection and weight assignment unit 520 selects one or more of metrics 540, e.g., Metric 1 540A, Metric 5 540B, and Metric M 540C, for evaluating the performance of the machine learning model with respect to one or more domains 505. Note that the metrics 540 correspond to metrics such as stability, accuracy, model size, and regression error, as shown in and described in connection with FIG. 4B . In one embodiment, the metric selection and weight assignment unit 520 selects one or more metrics, e.g., Metric 1 540A, Metric 5 540B, from the set of available metrics 540 based on certain criteria. For example, a first client of the hyperparameter tuning system 500 may desire to emphasize an accuracy parameter for the machine learning model, while another client of the hyperparameter tuning system 500 may desire to emphasize a different metric, e.g., a regression error metric, for the machine learning model. The requirements of different clients can be stored as one of the policies 515 based on which the metric selection and weight assignment unit 520 selects multiple metrics for evaluating the performance of the machine learning model.

[0110] The metric selection and weight assignment unit 520 is further configured to assign a weight to each of the selected metrics. The weight assigned to a particular metric indicates the importance of the metric to the performance of the machine learning model. In one embodiment, the metric selection and weight assignment unit 520 assigns weights to metrics based on the level of importance of the clients of the hyperparameter tuning system 500. For example, if a first client is more important than a second client (i.e., the first client has a higher SLA than the second client), the metric requested by the first client is assigned a higher weight than the metric requested by the second client. As another example, consider a machine learning model trained for domain detection. In such a case, there are two metrics used to evaluate the performance of the machine learning model: in-domain recall and out-of-domain recall. In such domain detection models, it is often desired that the model perform well, i.e., above a certain threshold level, for in-domain detection compared to out-of-domain detection. In this case, the weight assignment unit 520 assigns a higher weight to the in-domain recall metric compared to the out-of-domain recall metric. The weighted metrics are provided as a second input to the hyperparameter tuner 550 .

[0111] Each of the metrics 540 is associated with a corresponding specification set. A specification set includes multiple specification parameters that define or characterize the metric. As shown in FIG. 5, Metric 1 540A is associated with Specification Set 1 542A, Metric 2 540B is associated with Specification Set 5 542B, and Metric M 540C is associated with Specification Set M 542C. It will be appreciated that the specification set associated with a particular metric can be configured independently of the specification sets associated with other metrics. With reference to FIG. 4C , a specification set for a metric may include: (1) a training data set; (2) a validation data set; (3) a metric definition that defines a measure of how well the model meets a target objective for a data set, such as the training and validation data sets; (4) a target score for the metric (i.e., the score for the metric that the model is expected to meet); (5) a penalty factor for the metric; and (6) a bonus factor for the metric.

[0112] According to some embodiments, metrics may be allowed to share training and / or validation data sets. However, machine learning models are expected to produce results that are more robust when there is diversity in training and validation data sets among different metrics. The specification parameters of each of specification sets 542A, 542B, and 542C can be set to certain values ​​to achieve desired results. For example, for the regression error metric, the corresponding specification set can be configured as follows: 1. A training dataset is modeled based on a set of customers, e.g., important customers. A validation dataset contains examples that previous machine learning models have successfully classified and that are expected to be widely used by customers. 2. The metric definition of regression error is set to be accuracy. 3. The target accuracy score is set to be 95%. 4. The penalty factor is set to 130 and the bonus factor is set to 1, so that each percentage point below 95% is penalized 130 times more than each percentage point above 95%.

[0113] In one embodiment, for the stability metric, the corresponding specification set can be constructed as follows: 1. The training dataset is set to a small size, since a small dataset is expected to lead to higher instability. The validation dataset can be set to be reasonably larger than the training dataset. 2. The stability metric definition is set to be the standard deviation of the accuracy scores of the machine learning model on the validation dataset, for example, when the machine learning model is trained 13 times. 3. The target score is set to 8%, i.e., we want the accuracy score of the machine learning model to vary by a maximum of 8%. 4. The penalty factor is set to 13 and the bonus factor is set to 1. This means that each percentage point below the target score incurs a loss 13 times greater than a percentage point improvement above the target score.

[0114] In one embodiment, for the confidence score metric, the corresponding specification set can be constructed as follows: 1. The training dataset can range in size from small to large and can vary by domain. The validation dataset can contain examples within the domain that belong to those intended class labels. 2. The metric definition for the confidence score metric is set to be the percentage of sentences where the model's confidence threshold is greater than 55% (i.e., the prediction is correct and at least 55% more reliable than any other prediction). 3. The target score is set to be 90%, i.e., we want at least 90% of the examples to be reliably labeled. 4. The penalty factor is set to 13 and the bonus factor is set to 1.

[0115] It is appreciated that the configuration of the specification sets described above is intended to be exemplary and non-limiting. A system administrator may configure each of the specification sets in any other manner based on different requirements. Furthermore, utilizing different specification sets for validating machine learning models is described below with reference to hyperparameter tuner 550.

[0116] In some embodiments, hyperparameter tuning system 500 allows a user (e.g., a system administrator) to specify one or more constraints for hyperparameter tuning of a machine learning model. The one or more constraints are specified via constraint establishment unit 530. The one or more constraints are provided to hyperparameter tuner 550 as a third input. Each constraint is a requirement imposed on hyperparameter tuning of the machine learning model, i.e., each constraint is a requirement that should be satisfied by the trained machine learning model. Given the one or more constraints, hyperparameter tuner 550 is configured to identify a set of hyperparameters that affect each constraint, specify values ​​for the identified hyperparameters, and iteratively tune the hyperparameters until the trained machine learning model satisfies each of the one or more constraints, as described below.

[0117] In one embodiment, for each of the one or more constraints, hyperparameter tuner 550 identifies one or more hyperparameters from set of hyperparameters 555 that affect each constraint. Hyperparameter tuner 550 identifies the one or more hyperparameters that affect the constraint by varying the values ​​of the hyperparameters and determining whether the change in the value of the hyperparameter affects the value associated with the constraint. Furthermore, it will be appreciated that a first set of hyperparameters that affect a first constraint may differ from a second set of hyperparameters that affect a second constraint.

[0118] Once one or more hyperparameters that affect each constraint are identified, hyperparameter tuner 550 assigns values ​​to a set of hyperparameters 555 and iteratively tunes the hyperparameters until each of the constraints is satisfied. As an example, consider a set of hyperparameters 555 that includes five hyperparameters H=[h1, h2, h3, h4, h5]. Further, for illustrative purposes, consider that a user assigns two constraints C1 and C2, where hyperparameter tuner 550 identifies that hyperparameters h1 and h3 affect constraint C1, and that hyperparameters h2, h3, and h5 affect constraint C2.

[0119] The optimizer 551 (also referred to herein as a tuning unit) of the hyperparameter tuner 550 specifies values ​​for a set of hyperparameters H, i.e., V(H) = [v(h1), v(h2), v(h3), v(h4), v(h5)], referred to herein as a hyperparameter configuration. Note that the optimizer 551 randomly assigns the initial hyperparameter configuration. Furthermore, with respect to constraint C1, the optimizer iteratively changes the values ​​of hyperparameters h1 and / or h3 (while maintaining the values ​​of h2, h4, and h5) until constraint C1 is satisfied. Note that a constraint is satisfied when the values ​​of the hyperparameters that affect the constraint satisfy the requirements imposed by the constraint. With respect to constraint C2, the optimizer 551 iteratively changes / modifies the values ​​of hyperparameters h2, h4, and h5 (while maintaining the values ​​of h1 and h3) until constraint C2 is satisfied. Note that the optimizer performs the above-described iterations while training the machine learning model on one or more datasets. Specifically, as described below, the optimizer 551 determines an optimal configuration of hyperparameters that satisfies each of the constraints while optimizing an objective function (e.g., a cost or loss function) of the machine learning model with respect to multiple metrics. In one embodiment, examples of user-specified constraints include constraints such as:

[0120] The estimated latency for a given batch should be below a certain threshold, e.g., the latency should be below 80 ms for a batch size of 1.

[0121] - The maximum size of a trained model must be below some specified threshold (e.g., 13MB).

[0122] The training time of the model should be below some user-specified time threshold, for example, the training time should be 5 minutes or less.

[0123] Furthermore, in some embodiments, multiple user-specified constraints are prioritized. For example, each constraint is assigned a level of importance that reflects that constraint being satisfied. In some embodiments, constraints may be specified such that a trained machine learning model must satisfy some constraints, while other constraints are desirable but optional.

[0124] The optimizer 551 of the hyperparameter tuner 550 constructs / formulates an objective function to be optimized. The objective function may be a loss function or a cost function that serves as a performance measure for training / validating the machine learning model using one or more training / validation datasets. In one embodiment, the argument of the objective function is a set of hyperparameters associated with the machine learning model to be optimized by the optimizer 551. The value of the objective function is a weighted combination of the differences between the actual values ​​of each metric and the target values ​​configured for each metric. The weight of each metric in the weighted combination depends on whether the metric exceeds or fails to exceed the target value. In some cases, an asymmetric loss technique is utilized in which a higher weight associated with failure is assigned to a metric for achieving (as opposed to exceeding) the target value.

[0125] For example, according to one embodiment, the domain score can be formulated as follows: where v is a vector of hyperparameter values ​​and m i (v) is specified as the performance of the model for the i-th metric with respect to v, and t i is specified as the target performance of the ith metric, and p i and b i If L(v) is specified as the penalty coefficient and bonus coefficient for the i-th metric, respectively, then computing the score for a single domain (i.e., the domain score) (L(v)) can be formulated as follows:

[0126] L(v)=Σi b i max(m i (v)-t i ,0)-p i max(t i -m i (v),0)- Furthermore, the objective function or loss function (F_{tuning}) that calculates the scores of multiple domains (i.e., the objective score) can be formulated as follows:

[0127] F_{tuning}=w_0*f(D_0)+w_1*f(D_i)+...+w_N*f(D_N) where f(D_i) (i=1, 2, ..., N) is the domain score calculated for each domain using the objective function (L(v)), and w_0, w_1, ..., w_N are domain weights to be applied to each domain score. In this example, the goal of hyperparameter tuning is to maximize F_{tuning}. Other objective functions are contemplated, and for example, as described in further detail herein, the objective function can be modified to include scores including regression scores (e.g., regression score) or improvement scores (e.g., improvement score). The scores can be formulated to capture regression or improvement at the instance level. The instances can be inputs classified by a machine learning model, and scores formulated to capture instance-level regression or improvement can capture changes to the model's performance that are not reflected in the accuracy score.

[0128] The optimizer 551 tunes a set of hyperparameters 555 associated with the machine learning model to optimize an objective function (e.g., to obtain a minimum of a loss function) across one or more of the metrics. The optimizer may tune the set of hyperparameters using one or more hyperparameter tuning methods, such as a grid-based method (e.g., building a model for every possible combination of hyperparameter values ​​provided, evaluating each model, and selecting the model that produces the best results), a gradient search method (e.g., computing gradients with respect to the hyperparameters and then optimizing the hyperparameters using gradient descent), and a Bayesian method (defining a model to be built with a hyperparameter λ that is scored as v according to some evaluation metric after training, then calculating a posterior expectation of the hyperparameter space using previously evaluated hyperparameter values, selecting the optimal hyperparameter values ​​according to this posterior expectation as the next candidate model, and iteratively repeating this process until converging to an optimum).

[0129] More details regarding hyperparameter tuning are described herein with reference to FIGS. 6 and 7. In this manner, hyperparameter tuner 550 trains / validates a machine learning model by tuning a set of hyperparameters to achieve optimal performance for multiple different weighted metrics. In some instances, this process is performed while ensuring that each of one or more constraints is satisfied. In other words, hyperparameter tuning system 500 performs optimization of multiple weighted metrics by training a machine learning model for multiple different weighted domains, optionally supporting one or more user-specified constraints. Once the machine learning model is optimized for multiple metrics, hyperparameter tuning system 500 outputs a trained / validated ML model along with a configuration of hyperparameters that maximizes the objective function and, in some cases, satisfies one or more constraints. In the above-described embodiment, optimizer 551 constructs and optimizes the objective function. Note that the configuration of hyperparameter tuner 550 as described above is not intended to limit the scope of the present disclosure. For example, hyperparameter tuner 550 may include an objective function formulation unit (not shown) that formulates the objective function to be optimized by optimizer 551.

[0130] Hyperparameter tuning techniques FIG. 6 illustrates a simplified flow diagram 600 showing a training process performed by a hyperparameter tuning system (e.g., the hyperparameter tuning system 500 described in connection with FIG. 5 ) according to an embodiment. The process illustrated in FIG. 6 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 6 and described below is intended to be exemplary and non-limiting. While FIG. 6 illustrates various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, the steps may be performed in some different order, or some steps may also be performed in parallel.

[0131] At block 610, domains for training the machine learning model are obtained. For example, a user or a subsystem may provide domains for training the machine learning model. Each domain may be associated with one or more sets of training data. At block 620, the hyperparameter tuning system assigns weights to each of the obtained domains according to a policy. Note that assigning different weights to different domains allows the domains to have appropriate influence (i.e., according to their respective weights) on the training of the machine learning model. In some cases, the policy dictates that all domain weights be initialized to the same value, e.g., domain weight for each domain = X.

[0132] In block 630, one or more metrics are selected to evaluate the performance of the machine learning model for the obtained domain. For example, one or more metrics as shown in FIG. 4B are selected by a user (e.g., a system administrator) to evaluate the performance of the machine learning model. For each selected metric, a weight may be assigned to the metric according to another policy to indicate the importance of the metric to the performance of the machine learning model (block 640). In optional block 650, the user establishes one or more constraints. Each constraint is a quality or characteristic that the user desires to achieve in the trained machine learning model. In other words, each constraint is a requirement imposed on the machine learning model.

[0133] At block 660, the process formulates / constructs a function (i.e., an objective function) based on the input weighted metrics and the set of hyperparameters. In one embodiment, the objective function is a loss or cost function that serves as a performance measure for training machine learning models in one or more domains.

[0134] At block 670, the process iteratively tunes a set of hyperparameters associated with the machine learning model to optimize the machine learning model with respect to one or more of the metrics (e.g., to obtain an optimal value of the objective function. For example, in training the machine learning model with respect to a weighted domain, one or more hyperparameters that affect one or more constraints and / or functions are identified by varying the value of the hyperparameter and determining whether the change in the value of the hyperparameter affects the value associated with the function and / or constraint.

[0135] In some embodiments, in the process of tuning hyperparameters, the process evaluates the current configuration of the hyperparameters (i.e., the values ​​of the hyperparameters), the value of the function, and determines whether the model has converged and / or whether the current configuration satisfies each of one or more constraints. Convergence is the point in model training after which the change in learning rate becomes lower and the error generated by the model in training reaches a minimum (e.g., the model achieves a state during training where the loss settles within an error range around the final value). When a model converges, further training no longer significantly reduces the model error. Convergence can be of two types: global or local. Thus, convergence can be the point in model training after which the error or performance is within an error range around a local / global minimum. Mathematically, convergence can be thought of as the study of series and sequences. A model can be considered to be convergent when the series is a convergent series. If at least one of the constraints is violated and / or the value of the function is not optimal, the tuning process modifies the values ​​of one or more hyperparameters to obtain a new configuration of the hyperparameters and continues training the machine learning model based on the new configuration. Note that determining whether the model has converged and / or whether the current configuration violates a particular constraint may be performed by determining whether the values ​​of one or more hyperparameters affecting the model and the constraint satisfy the requirements imposed by a regression analysis (e.g., no regression for all domains) and / or the constraint. Furthermore, determining whether the value of the function (which indicates the performance of the machine learning model with respect to the current configuration) is optimal is performed by comparing the value of the function with new values ​​of the function obtained through different configurations of the hyperparameters.

[0136] In this manner, the tuning process iterates through the space of hyperparameter values ​​until a configuration that results in an optimal value of the function (and, optionally, does not violate any constraints) is achieved. It is further appreciated that the process of tuning hyperparameters can begin with an initial configuration of the hyperparameters that are randomly assigned. Furthermore, the tuning process can perform one of a random search, a Bayesian search, a branch-and-bound method, a grid search, a genetic algorithm, or the like, in searching the space of hyperparameter values ​​to obtain new hyperparameter configurations. Once the machine learning model is optimized, it is output to the user as a trained machine learning model (along with the values ​​of the hyperparameters that achieve the optimized machine learning model).

[0137] FIG. 7 illustrates a simplified flow diagram 700 depicting a validation process performed by a hyperparameter tuning system (e.g., the hyperparameter tuning system 500 described in connection with FIG. 5 ) according to an embodiment. The process illustrated in FIG. 7 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 7 and described below is intended to be exemplary and non-limiting. While FIG. 7 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, the steps may be performed in some different order, or some steps may also be performed in parallel.

[0138] In block 710, one or more metrics for evaluating the performance of the machine learning model are selected, with respect to which the hyperparameters of the machine learning model will be tuned. In block 720, a specification set associated with each selected metric is configured according to a criterion. Specifically, values ​​are assigned to the specification parameters included in the specification set based on the criterion. For example, if it is desired to reduce the regression error, the specification set associated with the regression error metric is configured as follows: a low value is set for the target score, and a high value is set for the penalty coefficient corresponding to the regression error metric. As an example, the target accuracy improvement may be set to 120%, i.e., at least 120% of the previously correctly labeled training examples are expected to be correctly labeled by the current version of the machine learning model. Furthermore, setting a high penalty coefficient, such as a penalty coefficient of 130, suggests that each percentage point below 120% (i.e., the performance of the machine learning model) is penalized 130 times more than a percentage point above 120%. In constructing such a regression error metric specification set, the regression error dominates the objective function (i.e., loss function), and the hyperparameter tuner of FIG. 5 performs tuning of hyperparameters to minimize the regression error.

[0139] At block 730, a metric score is calculated for each metric. Specifically, one or more selected metrics are evaluated against the validation data set, thereby generating a metric score for each metric. At block 740, the calculated metric score for each metric is compared to the metric's corresponding target score. In one embodiment, the difference between the metric score and the metric's target score is calculated. If the metric score is higher than the target score, the difference is multiplied by a bonus coefficient associated with the metric. On the other hand, if the metric score is lower than the target score, the difference is multiplied by a penalty coefficient. At block 750, an objective function, such as a loss function, is formulated based on the processing performed at block 740. Specifically, the loss function is determined to be the sum of the difference (between the metric score and the target score) multiplied by either the corresponding bonus coefficient or penalty coefficient. The process then moves to block 760, where the formulated objective function is optimized.

[0140] At block 760, a hyperparameter tuner iteratively tunes hyperparameters associated with the machine learning model to optimize the formulated objective function (e.g., to obtain a minimum value of the loss function) formulated at block 750. In one embodiment, in tuning the hyperparameters, the hyperparameter tuner evaluates the result of the loss function (e.g., a first objective score) for the current configuration (i.e., the value of the hyperparameters). The hyperparameter tuner further determines whether the result of the loss function (e.g., the first objective score) is optimal by comparing the result of the loss function (e.g., the first objective score) with the result of the loss function (e.g., a second objective score) obtained via a different configuration of the hyperparameters.

[0141] In this manner, the tuning process iterates through the space of hyperparameter values ​​until a configuration that results in an optimal value of the formulated objective function is achieved. The process of tuning hyperparameters can begin with an initial configuration of the hyperparameters that are randomly assigned. Furthermore, the tuning process can execute a search algorithm to search the hyperparameter space to obtain new hyperparameter configurations. The hyperparameter tuner can utilize one of a random search method, a Bayesian search method, a branch-and-bound method, etc. in searching the hyperparameter space. Once the objective function is optimized (i.e., a minimum value of the loss function is achieved), the hyperparameter tuner provides the validated machine learning model to the user as output, along with the values ​​of the hyperparameters that achieve the optimized objective function.

[0142] Objective function optimization in target-based hyperparameter tuning Disclosed herein are techniques for optimizing an objective function in target-based hyperparameter tuning, including modifying the objective function to minimize instance-level regression, to address data points with unstable predictions, to incorporate one or more domain regression tolerance levels, or any combination thereof.

[0143] Objective function optimization Attempting to find a uniform configuration of hyperparameters for thousands of different domains or experiments is a challenging hyperparameter tuning task. To accomplish this task, it is important to consider which domains are more and which domains are less important for achieving a specified goal, such as model accuracy performance. Domain weights, which indicate the importance of a domain, can be defined as part of the tuning objective function, with domains with higher weights receiving more attention during hyperparameter tuning than domains with lower weights. As discussed herein, for hyperparameter tuning optimized over N domains, e.g., D={D_0, D_1, ..., D_N}, the objective function can be defined as follows:

[0144] F_{tuning}=w_0*f(D_0)+w_1*f(D_1)+...+w_N*f(D_N) where f(D_i) (i=0,1,...,N) is the domain score calculated for each domain, and w_0, w_1,...,w_N are the domain weights that will be applied to each domain score.

[0145] A domain score is a measure of how well a model is performing, calculated according to the improvement and regression compared to the baseline score of the domain in trial t using model M_t, e.g., f(D_i,t).

[0146] improvement_weight*[improvement_score]=(score_{trial_t}(M_t,D_i)-score_{baseline}(M_t,D_i)) ---------if score_{trial_t}(M_t,D_i)>=score_{baseline}(M_t,D_i) - (minus) regression_weight*[regression_score]=(score_{trial_t}(M_t,D_i)-score_{baseline}(M_t,D_i))---------if score_{trial_t}(M_t,D_i) <score_{baseline}(M_t,D_i) As discussed herein, score_{trial_t}(M_t,D_i) for each domain may be calculated as various evaluation metrics, such as accuracy, F1, precision, recall, etc. Hyperparameter tuning strongly relies on the domain scores calculated for each domain to optimize hyperparameters. For example, the ratio between regression_weight and improvement_weight indicates whether hyperparameter tuning focuses on tuning hyperparameters to improve performance (while allowing some regression) or whether hyperparameter tuning focuses on tuning hyperparameters to minimize regression (while allowing less overall improvement).

[0147] Nevertheless, target-based hyperparameter tuning has several challenges because tuning relies heavily on domain scores calculated for each domain to optimize hyperparameters. First, it has been found that dataset-level domain evaluation scores do not reflect true improvement and regression in some cases. More specifically, domain scores indicate improvement and regression at the dataset level, but not necessarily at the instance level. For example, the overall accuracy of a model in a current trial may be the same as the baseline accuracy, but there are 1,000 more correctly predicted instances and 1,000 more incorrectly predicted instances. Because the instance-level improvement and instance-level regression cancel each other out, the overall accuracy remains the same. However, the 1,000 more incorrectly predicted instances may cause problems in model deployment and application. To overcome this challenge, the objective function can be modified to include the number of correctly and incorrectly predicted instances in the calculation of the domain score. This allows hyperparameter tuning to be performed to take into account not only dataset-level improvements and regressions, but also instance-level improvements and regressions. The advantage of this tuning technique is that instance-level regressions can be minimized during target-based hyperparameter tuning, improving the overall performance of the model and improving the performance of the computing system running the model (e.g., increasing the speed and efficiency of the underlying computing device and / or reducing the processing requirements or memory usage of the underlying computing device).

[0148] To modify the objective function to include the number of correctly and incorrectly predicted instances in the domain score calculation, the objective function is formulated to normalize the instance-level improvement and instance-level regression over the total number of instances to obtain the improvement_score and regression_score as follows:

[0149] · f(D_i,t)=improvement_weight*improvement_score-regression_weight*regression_score improvement_score = count(correct {trial_t}(M_t,D_j) ∩ incorrect {baseline}(M_b,D_j)) / count(instances) ○ regression_score=count(incorrect{trial_t}(M_t,D_i)∩correct{baseline}(M_b,D_i)) / count(instances) where correct{trial_t}(M_t,D_j) is the set of instances correctly predicted by the model in trial t for domain D_j, incorrect{baseline}(M_b,D_j) is the set of instances incorrectly predicted by the baseline model for domain D_j, incorrect{trial_t}(M_t,D_i) is the instance incorrectly predicted by the model in trial t for domain D_j, correct{baseline}(M_b,D_i) is the set of instances correctly predicted by the baseline model for domain D_j, and count() calculates the total number of instances in the instance set. Correct and incorrect can be determined by comparing predictions from the model against ground truth established for each example or instance.

[0150] Non-limiting examples include: 1. Given a test dataset of size 5000 utterances (i.e., instances) for domain A. 2. The baseline model accuracy performance could be 60% with 3000 correct predictions and 2000 incorrect predictions. 3. For the model in the current trial T, 2000 of the 3000 correctly predicted utterances based on the baseline model can be correctly predicted, and 1000 of the 2000 incorrectly predicted utterances based on the baseline model can also be correctly predicted. → Therefore, 3000 can be correctly predicted and 2000 can be incorrectly predicted, which equates to an accuracy performance of 60%. 4. However, 1000 of the 3000 correctly predicted utterances based on the baseline model are incorrectly predicted here, which can be considered "instance-level regressions." 1000 instance-level regressions may cause problems in model deployment and application. However, according to the "dataset-level" accuracy, 0% regressions are observed (60% → 60%). 5. Accordingly, 1000 of the 2000 incorrectly predicted utterances based on the baseline model can now be correctly predicted and considered "instance-level improvement." 6. The "instance-level improvement" (e.g., 1000) and "instance-level regression" (e.g., -1000) may be normalized over the total number of instances (e.g., 5000) to obtain the improvement_score and regression_score.

[0151] a. For example, improvement_score=1000 / 5000=0.2, regression_score=-1000 / 5000=-0.2.

[0152] b. If regression is assumed to be 100 times more important than improvement, e.g., improvement_weight:regression_weight=100:1, then f(D_i,t)=100.0*(-0.2)+1.0*(0.2)=-19.8.

[0153] Second, due to the statistical nature of machine learning or deep learning models, the domain score is calculated, in part, from data points that affect unstable prediction results. When tuning is performed based on these data points, the tuner is forced to chase random noise during the latter stages of tuning, which is not ideal. For example, a 0% regression tolerance rate may be configured for all customer datasets. However, during tuning, it may be observed that some customer datasets may not be perfect (i.e., outlier detection tools and manual analysis indicate that predictions for N test cases are unstable). Therefore, these N test cases should be excluded from hypertuning to ensure that the hypertuner is not chasing test cases containing random noise or errors. To overcome this challenge, the objective function can be modified to exclude unstable instances from the domain score calculation. This allows hyperparameter tuning to be performed to take the unstable instances into account. The advantage of this tuning technique is that target-based hyperparameter tuning avoids chasing randomly noisy or erroneous test cases while optimizing the hyperparameters, improving the overall performance of the model and improving the performance of the computing system running the model (e.g., increasing the speed and efficiency of the underlying computing device and / or reducing the processing requirements or memory usage of the underlying computing device).

[0154] To modify the objective function to exclude unstable instances, the objective function is formulated to subtract the number of unstable instances and normalize the instance-level improvement and instance-level regression over the total number of instances to obtain the improvement_score and regression_score, as follows:

[0155] · f(D_i,t)=improvement_weight*improvement_score-regression_weight*regression_score improvement_score = (count(correct{trial_t}(M_t,D_j)∩incorrect{baseline}(M_b,D_j))-count(unstable_instances)) / count(instances) ○ regression_score=(count(incorrect{trial_t}(M_t,D_i)∩correct{basedline}(M_b,D_i))-count(unstable_instances)) / count(instances) Unstable instances can be determined and counted by running an instability experiment on the model. For example, the model may be run a predetermined number of times (e.g., 10 times) on the same example or instance, and instances with unstable predictions over the predetermined number of runs are determined and counted as unstable. Unstable predictions are different predictions for the same instance run on the same model (e.g., if 9 out of 10 runs are predicted as "red" and 1 as "blue," this would indicate instability). In some cases, the instability determination is thresholded to ensure that there are a predetermined number of different predictions before a determination is made that the instance is unstable (e.g., if 6 out of 10 runs are predicted as "red" and 4 as "blue," and the threshold is set to 3 or greater, the experiment would indicate instability).

[0156] Non-limiting examples include: 1. Continuing with the example above, given 1000 "instance-level regressions" and 1000 "instance-level improvements" for domain A. 2. We consider that 200 of the regressions are determined from instability experiments to be potentially due to the quality of the test examples, e.g., utterances very close to the decision boundary. 3. In this example, the 200 unstable "instance-level regressions" can be excluded from the objective function calculation (e.g., -1000-(-200)=-800) so that hypertuning will no longer chase this random noise. 4. Additionally, the "instance-level improvement" (e.g., 1000) and "instance-level regression" (e.g., -800) may be normalized over the total number of instances (e.g., 5000) to obtain the improvement_score and regression_score.

[0157] a. For example, improvement_score=1000 / 5000=0.2, regression_score=-800 / 5000=-0.16.

[0158] b. If regression is assumed to be 100 times more important than improvement, e.g., improvement_weight:regression_weight=100:1, then f(D_i,t)=100.0*(-0.16)+1.0*(0.2)=-15.8.

[0159] Finally, it has been found that an acceptable level of regression (e.g., regression ratio m) can be defined for different domains according to various user goals (e.g., business requirements). For example, 0% regression can be configured or tolerated for domain A, while 3% regression can be configured or tolerated for domain B. To enable this augmentation, the objective function can be modified to include parameters for the acceptable level of regression in one or more of the domains. For example, a 2% acceptable regression rate level can be configured or tolerated for the training / test split dataset, while a 0% acceptable regression rate level can be configured or tolerated for the customer dataset due to its greater impact on a "custom set" of users. This allows hyperparameter tuning to be performed to take into account the acceptable regression levels of one or more domains. An advantage of this tuning technique is that target-based hyperparameter tuning allows for more granular configuration of domain importance assignment during hyperparameter tuning; for example, some users may be more interested in regression, while some may be more interested in improvement. This incorporates the regression tolerance level of the domain while optimizing the hyperparameters, improving the overall performance of the model and improving the performance of the computing system running the model (e.g., increasing the speed and efficiency of the underlying computing device and / or reducing the processing requirements or memory usage of the underlying computing device).

[0160] To modify the objective function to incorporate the regression tolerance levels of the domains, the objective function is formulated to include parameters for the acceptable regression ratios m of one or more of the multiple domains as follows:

[0161] · f(D_i,t)=improvement_weight*improvement_score-regression_weight*(0 if -regression_score <m else regression_score) ○ improvement_score=count(correct{trial_t}(M_t,D_j)∩incorrect{baseline}(M_b,D_j)) / count(instances) ○ regression_score=count(incorrect{trial_t}(M_t,D_i)∩correct{basedline}(M_b,D_i)) / count(instances) It should be noted that while the objective functions described above are modified based on each other to incorporate the number of correctly and incorrectly predicted instances in the calculation of the domain score, to filter out unstable instances, and / or to incorporate the regression tolerance level of the domain, it should be understood that each modification of the objective function described herein can be incorporated into the objective function individually or in any combination.

[0162] Non-limiting examples include: 1. Continuing with the example above, given 1000 (20%) "instance-level regressions" and 1000 (20%) "instance-level improvements" out of 5000 test cases for domain A. 2. Suppose business requirements require 0% regression tolerance for domain A.

[0163] a Since 20%>0%, regression_score=-1000 / 5000=-0.2. 3. Additionally, domain B may have 30 (1.5%) "instance-level regressions" and 50 (2.5%) "instance-level improvements" out of 2000 test cases. 4. Consider that a regression tolerance of 3% is required for Domain B as per business requirements.

[0164] a. Since 1.5%<3%, regression_score=0. 5. On the other hand, if the regression acceptability of domain B is 1%, then the regression_score of domain B will be regression_score=-30 / 2000=-0.015.

[0165] Experimental example The systems and methods implemented in various embodiments can be better understood with reference to the following experimental examples: Consider three test types for which a model developer would like to ensure the absence of regressions as a highest priority: naive, user regression, and QA user regression. Simple test types include simple test cases for which it is desired to ensure a high accuracy score such that no failures occur for these simple test cases during model generation / deployment. User (e.g., customer) regressions and QA user regressions include test cases shared by users, which are desired to ensure there are no regressions in the model generation / deployment. For each failure, it is desired to provide a proper justification.

[0166] It can be observed through experiments with the objective function modifications described herein applied to target-based hyperparameter tuning that it is possible to minimize the overall number of stable regressions for each test type while maintaining or improving accuracy.

[0167] [Table 1]

[0168] As shown in Figure 1, the number of stable regressions for the simple test type drops to 4, the number of stable regressions for the customer regression test type drops to 35, and the number of stable regressions for the QA customer regression test type drops to 27.

[0169] Objective Function Optimization and Tuning Workflow FIG. 8 is a simplified diagram of a tuning workflow 800 according to various embodiments. An objective function optimization workflow can be used to evaluate proposed sets of hyperparameter values ​​by calculating an objective score for each proposed set. The objective score can be calculated using an objective function that includes a domain score and a weight domain for each domain in the hyperparameter search space, as described in detail with respect to FIGS. 4A-7. As described in more detail above, the domain score is calculated according to the improvement and regression, e.g., f(D_i,t), compared to the baseline score for the domain in trial t using model M_t.

[0170] Referring more specifically to the objective function optimization workflow 800, at block 805, a domain for training the machine learning model is obtained. For example, a user or a subsystem may provide the domain for training the machine learning model. Each domain may be associated with one or more sets of training data. At block 810, one or more metrics are selected to evaluate the performance of the machine learning model for the obtained domain. For example, one or more metrics such as those shown in FIG. 4B are selected by a user (e.g., a system administrator) to evaluate the performance of the machine learning model. For each selected metric, a weight may be assigned according to a policy for the metric to indicate the importance of the metric to the performance of the machine learning model. At optional block 820, the user establishes one or more constraints. Each constraint is a quality or characteristic that the user desires to achieve in the trained machine learning model. In other words, each constraint is a requirement imposed on the machine learning model. At block 825, the user establishes modifications or augmentations to the objective function (e.g., minimizing instance-level regression, incorporating a regression tolerance level for one or more domains, addressing data points with unstable predictions, or any combination thereof). Each modification or augmentation is a quality or characteristic that a user desires to achieve in target-based hyperparameter tuning -- in other words, each modification or augmentation is a requirement imposed on the tuning process via the objective function.

[0171] At block 830, the process formulates / constructs an objective function based on the domains, metrics, optional constraints, and modifications or augmentations. As described herein, the objective function is an expression including domain weights and domain scores for each domain, where the domains include the training dataset and the evaluation dataset. In one embodiment, the objective function is a loss or cost function that serves as a performance measure for training the machine learning model within one or more domains. The arguments of the objective function are a set of hyperparameters associated with the machine learning model that are optimized by an optimizer (e.g., optimizer 551 described in connection with FIG. 5). The value of the objective function is a weighted combination of the differences between the actual values ​​of each metric and the target values ​​configured for each metric. The weight of each metric in the weighted combination depends on whether the metric exceeds or fails to exceed the target value. Specifically, an asymmetric loss technique is utilized in which a higher weight (than exceeding the target value) is assigned to a metric associated with failing to achieve the target value.

[0172] For example, an objective function (e.g., a weighted objective function) can be formulated as follows: where v is a vector of hyperparameter values ​​and m i (v) is the value of the i-th metric for v, and t i is specified as the target value of the ith metric, and p i and b i If L(v) is specified as the regression coefficient and improvement coefficient (e.g., weight) of the i-th metric, respectively, the objective function or loss function (L(v)) can be formulated as Equation (1):

[0173] L(v)= Σ i p i max(t i -m i (v),0)-b i max(m i (v)-t i ,0) (1) Other objective functions are contemplated; for example, the objective function can be modified to include a score including a regression score (e.g., regression score) or an improvement score (e.g., improvement score). The score can be formulated to capture regression or improvement at the instance level. A domain score formulated to capture instances that are example inputs to the machine learning model and instance-level regression or improvement can capture changes to the model's performance that are not reflected in the accuracy score. In some circumstances, the domain score can include a normalized change in correctly predicted instances by dividing the change in correctly predicted instances by the total number of instances.

[0174] Unstable instances, or instances within a threshold distance of the decision boundary, can cause the model to chase noise, and the domain score can be formulated to exclude unstable instances. Unstable instances can include unstable accurate instances, correctly classified instances but too close to the decision boundary, or unstable inaccurate instances, including incorrectly classified instances that are too close to the decision boundary. A normalized difference in the number of correctly or incorrectly classified instances can be generated by subtracting the number of unstable instances from the number of correctly or incorrectly classified instances and dividing this amount by the total number of instances.

[0175] In some situations, different domains of the set of hyperparameters may not be equally important, and the objective function can be formulated to allow different amounts of regression in different domains. A regression tolerance threshold (e.g., percentage regression) can be established for one or more domains of the set of hyperparameters. The regression tolerance threshold can be based on various goals, such as business requirements.

[0176] For example, an objective function (e.g., a weighted objective function) can be formulated as follows: where v is a vector of hyperparameter values ​​and p_score ij (v) is specified as the regression score of the ith metric in the jth domain, and b_score ij (v) is the improvement score of the ith metric in the jth domain, and p ij and b ij If L(v) is specified as the regression coefficient and improvement coefficient (e.g., weight) of the i-th metric in the j-th domain, respectively, the objective function or loss function (L(v)) can be formulated as Equation (2):

[0177] L(v)=Σ ij p ij *p_score ij (v)-b ij *b_score ij (v) (2) In another case, the objective function can be formulated as follows: where v is a vector of hyperparameter values ​​and p_score ij (v) is specified as the regression score of the ith metric in the jth domain, and b_score ij (v) is the improvement score of the ith metric in the jth domain, and p ij and b ij are specified as the regression coefficient and improvement coefficient of the ith metric in the jth domain, respectively, and r ij If ν is the regression tolerance threshold for the i-th metric in the j-th domain, the objective function or loss function (L(v)) can be formulated as Equation (3):

[0178]

number

[0179] In some cases, the improvement score can be formulated to capture the instance-level regression or improvement as follows: where v is a vector of hyperparameter values ​​and b_score ij (v) is the improvement score for the jth domain and the ith metric for v, and correct ij Let (v) be the number of correctly classified instances in the jth domain using the ith metric for v, and let incorrect_t ij is the target (baseline) number of inaccurate instances for the jth domain and the ith metric, and instances j If can be the number of instances in the,j,th domain, the improvement score can be formulated as,Equation (4).

[0180] b_score ij (v)=[correct ij (v)∩incorrect_t ij ] / instances j (4) In some cases, the regression score can be formulated to capture instance-level regression or improvement as follows: where v is a vector of hyperparameter values ​​and p_score ij (v) is the regression score for the jth domain and the ith metric for v, and ij Let (v) be the number of incorrectly classified instances in the jth domain using the ith metric for v, and correct_t ij is the target (baseline) number of qualifying instances for the jth domain and the ith metric, and instances j If can be the number of instances in the,j,th domain, the improvement score can be formulated as,Equation (5).

[0181] p_score ij (v)=[incorrect ij(v)∩correct_t ij ] / instances j (5) In some cases, the improvement score can be formulated to be tolerant to the influence of unstable instances as follows: where v is a vector of hyperparameter values ​​and b_score ij (v) is the improvement score for the jth domain and the ith metric for v, and correct ij Let (v) be the number of correctly classified instances in the jth domain using the ith metric for v, and let incorrect_t ij is the target number of inexact instances for the jth domain and the ith metric, and instances j be the number of instances in the jth domain, and unstable ij If can be the number of unstable instances, the improvement score can be formulated as,Equation (6).

[0182] b_score ij (v)=[(correct ij (v)∩incorrect_t ij )-unstable ij ] / instances j (6) In some cases, the regression score can be formulated to be more tolerant to unstable instances as follows: where v is a vector of hyperparameter values ​​and p_score ij (v) is the regression score for the jth domain and the ith metric for v, and ij Let (v) be the number of incorrectly classified instances in the jth domain using the ith metric for v, and correct_t ij is the target number of qualifying instances for the jth domain and the ith metric, and instances jbe the number of instances in the jth domain, and unstable ij If can be the number of unstable instances, the improvement score can be formulated as,Equation (7).

[0183] p_score ij (v)=[(incorrect ij (v)∩correct_t ij )-unstable ij ] / instances j (7) In block 835, values ​​of domain weights to be used in the objective function formulated in block 830 are initialized. The domain weight values ​​are assigned to the objective function during initialization or as part of periodic domain weight updates, and domain scores are calculated by performing the evaluation process described in block 845 using the hyperparameter values ​​proposed in block 840. In some embodiments, all domain weights may be initialized as a uniform distribution (e.g., all domain weights may be set equal to 1.0 according to a general policy), and the uniform distribution may be stored in a database (as shown in FIG. 8). The domain weights may be initialized for the first attempt in a continuous tuning technique, and after initialization, the domain weights in the database may be updated every K attempts. For example, the update may be performed when an attempt counter reaches a threshold (e.g., determining whether n attempts == 0). In other cases, the domain weights in the database may be updated after a given evaluation time period (e.g., a time period threshold such as 6 hours).

[0184] At block 840, hyperparameter values ​​for trial i are searched for and proposed for the model. As described herein, hyperparameters are parameters used to control the architecture and learning process of a model. The search and selection of hyperparameters can be performed using a new hyperparameter tuning algorithm (e.g., random search, Bayesian search, branch and bound, grid search, genetic algorithm, etc.). The hyperparameter tuning algorithm determines a new set of hyperparameter values ​​for trial i using objective scores calculated from previous trials and corresponding trial hyperparameter values ​​(e.g., based on a matrix of prior objective scores and corresponding values ​​for the set of hyperparameters), as described in detail with respect to FIGS. 4A-7.

[0185] At block 845, trial i is evaluated. During trial evaluation, a model is trained on one or more training data sets associated with the domain using the proposed hyperparameter values ​​for trial i, the trained model is evaluated on one or more evaluation data sets associated with the domain using the proposed hyperparameter values ​​for trial i, and a domain score f(D_i) is calculated for the domain based on the training and / or evaluation. This process is repeated using a model for each domain. The domain score f(D_i) is calculated using the objective function formulated / constructed at block 830.

[0186] In block 850, an objective score is calculated for trial i. The objective score is calculated using the domain scores from block 845 and the domain weights stored for each domain in the database. For example, the objective score may be calculated by fetching the domain weights from the database and substituting the domain weights and the domain scores of all domains from block 845 into an integrated objective function, such as a linear or weighted linear combination of each domain score, to calculate the objective score for trial i.

[0187] At decision block 855, the attempt counter or timer is checked to determine if a threshold has been reached. If the attempt counter or timer has not reached the threshold, the tuning technique proceeds to the next attempt i+1 by proposing a new set of hyperparameter values ​​at block 850. If the attempt counter or timer has reached the threshold, the tuning technique proceeds to block 860, where the domain weights are updated and the objective score for the previous attempt is recalculated based on the updated domain weights.

[0188] In block 860, the domain weights are updated, and the objective score of the previous trial is recalculated based on the updated domain weights. The domain weight of each domain may be updated if the domain score meets certain requirements. For example, in threshold-based domain weight adjustment, the domain weight of a given domain may be adjusted if the domain score is below a domain score threshold. In other embodiments, the domain weights may be adjusted for some domains with the M lowest domain scores (e.g., the domain weights may be adjusted for the M domains with the poorest performance in the trial).

[0189] After adjusting the domain weights, the updated domain weights are saved to a database, the objective scores from the previous trial are recalculated based on the updated domain weights, and the recalculated objective scores from the previous trial are saved to a database. The objective scores from the previous trial are updated using the updated domain weights, because the previous objective scores and the current objective scores (along with all correlated hyperparameter values) are used by the hyperparameter tuning algorithm to search for and select new hyperparameters for each trial evaluation. By using the updated domain weights to update the previous objective scores and the current objective scores, all correlations between the previous objective scores and the evaluated hyperparameter values ​​are effectively updated, so that the hyperparameter tuning algorithm can better search for and select new hyperparameters for each trial evaluation. In some cases, the objective scores of all previous trials are recalculated based on the updated domain weights. In other cases, the objective score of at least one previous trial is recalculated based on the updated domain weights.

[0190] After the domain weights are updated and the objective scores for the previous trial are recalculated, the tuning technique can proceed to the next trial i+1 and propose a new set of hyperparameter values ​​in block 840. For the next trial i+1, one or more hyperparameter tuning processes use the recalculated objective scores from the previous trial and the corresponding trial hyperparameter values ​​(e.g., based on a matrix of the recalculated previous objective scores and corresponding values ​​of the set of hyperparameters) to determine a new set of hyperparameter values ​​for the next trial i+1. This process continues iteratively until the values ​​of the hyperparameters converge to optimal values ​​or a stopping condition is met (e.g., a threshold number of trials have been performed).

[0191] Objective function optimization and tuning techniques FIG. 9 illustrates a simplified flow diagram 900 depicting an objective function optimization and tuning technique performed by a hyperparameter tuning system (e.g., the hyperparameter tuning system 500 described in connection with FIG. 5 ) according to various embodiments. The process illustrated in FIG. 9 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The technique presented in FIG. 9 and described below is intended to be illustrative and non-limiting. While FIG. 9 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, the steps may be performed in some different order, or some steps may also be performed in parallel.

[0192] The machine learning algorithm is initialized with a set of hyperparameter values ​​at block 905. Initialization includes configuring the machine learning algorithm and a training protocol for the machine learning algorithm using the set of hyperparameter values.

[0193] At block 910, a hyperparameter objective function is accessed that is defined at least in part for multiple domains of a search space associated with the machine learning algorithm. The search space includes a training data set and an evaluation data set, and each domain includes a subdivision of the search space having at least one training data set and at least one evaluation data set. The hyperparameter objective function includes a domain score for each domain that is calculated based on the number of instances in the at least one evaluation data set that are predicted correctly or incorrectly by the machine learning algorithm during a given trial.

[0194] In some cases, the hyperparameter objective function is formulated to normalize the instance-level improvement and instance-level regression over the total number of instances to obtain an improvement score and a regression score. A domain score for each domain may be calculated based on the improvement score and the regression score. In some situations, the improvement score is calculated based on a total count of (i) the number of instances in at least one evaluation dataset that are correctly predicted by the machine learning model during a given trial, (ii) the number of instances in at least one evaluation dataset that are incorrectly predicted by the baseline machine learning model, and (iii) the number of instances in the at least one evaluation dataset. In some situations, the regression score is calculated based on (i) the number of instances in at least one evaluation dataset that are incorrectly predicted by the machine learning model during a given trial, (ii) the number of instances in at least one evaluation dataset that are correctly predicted by the baseline machine learning model, and (iii) the number of instances in the at least one evaluation dataset.

[0195] In some cases, the hyperparameter objective function is formulated to exclude unstable instances, which are instances in at least one evaluation dataset that have been determined to yield different predicted results from each other using the same machine learning model. In some cases, eliminating unstable instances includes (i) subtracting a count of the excluded unstable instances from the number of instances in the at least one evaluation dataset that are correctly predicted by the machine learning model during a given trial, and (ii) subtracting a count of the excluded unstable instances from the number of instances in the at least one evaluation dataset that are incorrectly predicted by the machine learning model during a given trial.

[0196] In some cases, the hyperparameter objective function is formulated to include a parameter for an acceptable regression ratio m for one or more of the multiple domains. In some cases, the parameter is defined based on a regression score, and if the regression score is less than the acceptable regression ratio m, the regression score is set to zero; otherwise, the regression score is calculated based on a total count of (i) the number of instances in at least one evaluation dataset that are incorrectly predicted by the machine learning model during a given trial, (ii) the number of instances in at least one evaluation dataset that are correctly predicted by the baseline machine learning model, and (iii) the number of instances in the at least one evaluation dataset.

[0197] In block 915, blocks 920-950 and 960 are executed iteratively for each trial of the hyperparameter tuning process.

[0198] At block 920, a machine learning algorithm for each domain is trained using at least one training dataset and a set of hyperparameter values ​​associated with each domain. The training outputs multiple machine learning models, including a machine learning model for each domain.

[0199] At block 925, the machine learning model for each domain is evaluated using at least one evaluation dataset and a set of hyperparameter values ​​associated with each domain, including generating a domain score for each domain.

[0200] At block 930, an objective score for the current trial is calculated based on the domain score for each domain and the domain weight associated with each domain using a hyperparameter objective function.

[0201] The objective score for the current attempt is stored in a database at block 935. The database includes the objective scores for the attempts from the current attempt and previous attempts of the hyperparameter tuning process, the domain weights associated with each domain, the domain scores for each domain from the current attempt and previous attempts of the hyperparameter tuning process, and the set of hyperparameter values ​​and all other sets of hyperparameter values ​​from previous attempts of the hyperparameter tuning process.

[0202] At block 940, a determination is made as to whether the machine learning model has reached convergence based on the objective score of the current trial.

[0203] In response to determining that the machine learning model has not reached convergence based on the objective score of the current trial, a new set of hyperparameters is determined for use in a subsequent trial of the hyperparameter tuning process at block 945. The new set of hyperparameters is determined based on the objective score of the current trial, the trial objective score from a previous trial of the hyperparameter tuning process, the set of hyperparameter values, and all other sets of hyperparameter values ​​from the previous trials of the hyperparameter tuning process.

[0204] At block 950, in response to determining that the machine learning model has reached convergence, at least one of the plurality of machine learning models is provided. In some cases, providing includes deploying at least one of the plurality of machine learning models in a digital assistant or chatbot system such as those described in connection with FIGS. 1-3. In some cases, providing includes displaying and / or communicating at least one of the plurality of machine learning models to a user and / or other systems (e.g., external systems). In some cases, providing includes deploying at least one of the plurality of machine learning models in an inference phase to automatically analyze text and categorize the text into intent. In some cases, providing includes storing at least one of the plurality of machine learning models in a system (e.g., an external system or a storage device). In some cases, providing includes deploying at least one of the plurality of machine learning models in an inference phase to automatically analyze text and create a dialog with a user and / or respond to a query posed by a user.

[0205] Exemplary System 10 shows a simplified diagram of a distributed system 1000. In the illustrated example, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 can be configured to run one or more applications.

[0206] In various examples, server 1012 can be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 1012 can also provide other services or software applications, which may include non-virtual and virtual environments. In some examples, these services can be provided to users of client computing devices 1002, 1004, 1006, and / or 1008 as web-based or cloud services, such as based on a software-as-a-service (SaaS) model. Users operating client computing devices 1002, 1004, 1006, and / or 1008 can then utilize one or more client applications to interact with server 1012 and utilize the services provided by these components.

[0207] 10 , server 1012 may include one or more components 1018, 1020, and 1022 that perform functions performed by server 1012. These components may include software components that may be executed by one or more processors, hardware components, or combinations thereof. It should be appreciated that a variety of different system configurations are possible that may differ from distributed system 1000. Thus, the example shown in FIG. 10 is one example of a distributed system for implementing an exemplary system and is not intended to be limiting.

[0208] A user may use client computing devices 1002, 1004, 1006, and / or 1008 to execute one or more applications, models, or chatbots that may generate one or more events or models that may then be executed or serviced in accordance with the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user via this interface. Although FIG. 10 shows only four client computing devices, any number of client computing devices may be supported.

[0209] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems such as Google Chrome™ OS), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®). Portable handheld devices may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays, and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.), etc. The client devices may be capable of running a variety of different applications, such as various Internet-related apps, communication applications (e.g., email applications, short message service (SMS) applications), etc., and may use a variety of communication protocols.

[0210] Network 1010 may be any type of network familiar to those skilled in the art capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, network 610 may be a local area network (LAN), an Ethernet-based network, a Token-Ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating according to any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0211] The servers 1012 may be comprised of one or more general-purpose computers, specialized server computers (including, by way of example, PC (personal computer) servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable configuration and / or combination. The servers 1012 may also include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as an elastic pool of one or more logical storage devices that can be virtualized to maintain virtual storage devices for the servers. In various examples, the servers 1012 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0212] The computing systems within server 1012 may run one or more operating systems, including any of those described above, as well as any commercially available server operating system. Server 1012 may also run any of a variety of additional server and / or middle-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA® server, a database server, etc. Exemplary database servers include, without limitation, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc.

[0213] In some embodiments, server 1012 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. By way of example, the data feeds and / or event updates may include real-time updates received from one or more third-party sources and continuous data streams that may include, without limitation, Twitter® feeds, Facebook® updates, or real-time events related to sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 1012 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0214] The distributed system 1000 may also include one or more data repositories 1014, 1016. These data repositories, in one example, may be used to store data and other information. For example, one or more of the data repositories 1014, 1016 may be used to store information related to chatbot performance or information such as generated models for use by a chatbot used by the server 1012 in performing various functions according to various embodiments. The data repositories 1014, 1016 may reside in various locations. For example, the data repository used by the server 1012 may be local to the server 1012 or may be remote from the server 1012 and communicate with the server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 may be of different types. In one example, the data repository used by the server 1012 may be a database, such as a relational database, such as databases provided by Oracle Corporation® and other suppliers. One or more of these databases may be adapted to allow the storage, updating, and retrieval of data to and from the database in response to SQL-formatted commands.

[0215] In some examples, one or more of the data repositories 1014, 1016 may also be used by an application to store application data. The data repositories used by the application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general storage repository backed by a file system.

[0216] In some examples, the functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 11 is a simplified block diagram of a cloud-based system environment in which various services can be provided as cloud services, according to some examples. In the example shown in FIG. 11, a cloud infrastructure system 1102 can provide one or more cloud services that can be requested by users using one or more client computing devices 1104, 1106, and 1108. The cloud infrastructure system 1102 can include one or more computers and / or servers, which may include those described above for server 1012. The computers in the cloud infrastructure system 1102 may be organized as general-purpose computers, specialized server computers, server farms, server clusters, or any other suitable configuration and / or combination.

[0217] The network 1110 can facilitate communication and the exchange of data between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network 1110 can include one or more networks. The networks can be of the same or different types. The network 1110 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0218] The example shown in Figure 11 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in some other examples, cloud infrastructure system 1102 may have more or fewer components than those shown in Figure 11, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 11 shows three client computing devices, in alternative examples, any number of client computing devices may be supported.

[0219] The term cloud service is generally used to refer to services made available to users on demand and over a communications network, such as the Internet, by a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Thus, customers can use cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system may host applications, and users can order and use the applications as needed over the Internet, without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, Java cloud services, etc., are offered by Oracle Corporation®, located in Redwood Shores, California.

[0220] In one example, cloud infrastructure system 1102 can provide one or more cloud services using various models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including hybrid service models. Cloud infrastructure system 1102 may include a set of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0221] The SaaS model allows applications or software to be delivered to customers as a service over a communications network, such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 1102. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, etc.

[0222] The IaaS model is generally used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) as cloud services to customers to provide elastic computing and storage capabilities. Various IaaS services are offered by Oracle Corporation.

[0223] The PaaS model is generally used to provide platform and environment resources as a service, allowing customers to develop, run, and manage applications and services without the customer having to acquire, build, or maintain such resources. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Services (JCS), Oracle Database Cloud Services (DBCS), data management cloud services, various application development solution services, etc.

[0224] Cloud services are generally provided in an on-demand, self-service manner, on a subscription basis, elastically scalable, and highly reliable, available, and secure. For example, a customer may order one or more services provided by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent), as described above, and / or provide the services of a chatbot system as described herein. Cloud infrastructure system 1102 may be configured to provide one or even more cloud services.

[0225] Cloud infrastructure system 1102 can provide cloud services through several different deployment models. In a public cloud model, cloud infrastructure system 1102 may be owned by a third-party cloud service provider, and cloud services are offered to any public customer, where the customer may be an individual or a business. In another example, under a private cloud model, cloud infrastructure system 1102 may be operated within an organization (e.g., within a corporate organization) and services offered to customers within the organization. For example, customers may be various departments of a company, such as a human resources department, a payroll department, etc., or even individuals within the company. In another example, under a community cloud model, cloud infrastructure system 1102 and the services offered may be shared by several organizations within an associated community. Various other models, such as hybrids of the above models, may also be used.

[0226] Client computing devices 1104, 1106, and 1108 may be of different types (e.g., client computing devices 1002, 1004, 1006, and 1008 shown in FIG. 10 ) and may be capable of running one or more client applications. Users may use the client devices to interact with cloud infrastructure system 1102, such as to request services provided by cloud infrastructure system 1102. For example, users may use the client devices to request information or actions from a chatbot, as described in this disclosure.

[0227] In some examples, the processing performed by cloud infrastructure system 1102 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing the data in parallel, performing simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 1102 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0228] 11 , cloud infrastructure system 1102 may include infrastructure resources 1130 utilized to facilitate the provision of various cloud services provided by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc. In one example, a storage virtual machine available to service storage requested by an application may be part of cloud infrastructure system 1102. In other examples, the storage virtual machine may be part of a different system.

[0229] In one example, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 1102 to different customers, resources can be bundled into sets of resources or resource modules (also referred to as “pods”). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In one example, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provided for database services, a second set of pods may be provided for Java services, and so on, which may include a different combination of resources than the pods in the first set of pods. For some services, resources allocated to provide the service may be shared between services.

[0230] Cloud infrastructure system 1102 may itself use services 1132 internally that are shared by different components of cloud infrastructure system 1102 and that facilitate the provision of services by cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and restore services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0231] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 11 , the subsystems may include a user interface subsystem 1112 that allows users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 through which cloud services offered by cloud infrastructure system 1102 are advertised and available for purchase by consumers, and other interfaces 1118. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 1102 (service request 1134) using one or more of interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 1102, and place a subscription order for one or more services offered by cloud infrastructure system 1102 to which the customer wishes to subscribe. The service request may include information identifying a customer and one or more services to which the customer wishes to subscribe. For example, a customer may place a subscription order for services provided by cloud infrastructure system 1102. As part of the order, the customer may provide information identifying a chatbot system to which the service will be provided, and optionally one or more authentication credentials for that chatbot system.

[0232] 11 , cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this processing, OMS 1120 may be configured to create an account for the customer if not already done so, receive billing and / or accounting information from the customer that will be used to bill the customer for providing the customer with the requested services, verify the customer information, and upon verification, register the customer's order and coordinate various workflows to prepare the order for delivery.

[0233] Upon proper validation, the OMS 1120 can then invoke an order fulfillment subsystem (OPS) 1124, which is configured to provide resources for the order, including processing, memory, and networking resources. The fulfillment may include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provided for the order and the type of resources provided may depend on the type of cloud service ordered by the customer. For example, according to one workflow, the OPS 1124 can be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for that specific cloud service. The number of pods allocated to the order may depend on the size / amount / level / scope of the service being requested. For example, the number of pods allocated may be determined based on the number of users to be supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized to provide the requested service for the specific requesting customer.

[0234] In one example, the setup phase process described above may be performed as part of the provisioning process by cloud infrastructure system 1102. Cloud infrastructure system 1102 may generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by cloud infrastructure system 1102 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 1102.

[0235] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service is ready for use at that time. In some cases, information (e.g., a link) may be sent to the customer to enable the customer to begin using and taking advantage of the benefits of the requested service. In one example, for the customer requesting the service, the response may include a chatbot system ID generated by cloud infrastructure system 1102 and information identifying the chatbot system selected by cloud infrastructure system 1102 for the chatbot system corresponding to the chatbot system ID.

[0236] Cloud infrastructure system 1102 may provide services to multiple customers. For each customer, cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1102 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0237] Cloud infrastructure system 1102 can provide services to multiple customers in parallel. Cloud infrastructure system 1102 can store information for these customers, possibly including sensitive information. In one example, cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage customer information and provide separation of managed information so that information associated with one customer is not accessible by another customer. IMS 1128 can be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0238] 12 illustrates an example of a computer system 1200. In some examples, the computer system 1200 can be used to run a digital assistant or chatbot system in a distributed environment, as well as any of the various servers and computer systems described above. As shown in FIG. 12, the computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 may include a non-transitory computer-readable storage medium, including a storage medium 1222, and a system memory 1210.

[0239] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. While bus subsystem 1202 is shown schematically as a single bus, alternative examples of a bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a Mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0240] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may be a single-core or multi-core processor. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1204 may include one or more special-purpose coprocessors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 1204 may be implemented using custom circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0241] In some examples, processing units within processing subsystem 1204 can execute instructions stored in system memory 1210 or on computer-readable storage medium 1222. In various examples, the processing units can execute various program or code instructions and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code being executed can reside in system memory 1210 and / or on computer-readable storage medium 1222, which in some cases includes one or more storage devices. Through appropriate programming, processing subsystem 1204 can provide the various functions described above. In instances where computer system 1200 is running one or more virtual machines, one or more processing units can be allocated to each virtual machine.

[0242] In one example, a processing acceleration unit 1206 may optionally be provided to perform customized processing or to offload portions of the processing performed by the processing subsystem 1204 so as to accelerate the overall processing performed by the computer system 1200.

[0243] I / O subsystem 1208 may include devices and mechanisms for inputting information into computer system 1200 and / or outputting information from or through computer system 1200. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, acoustic input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion-sensing and / or gesture-recognition devices such as a Microsoft Kinect® motion sensor, a Microsoft Xbox® 360 game controller, or devices that provide an interface for receiving input using gesture and voice commands, allowing a user to control and interact with the input device. The user interface input device may also include an eye gesture recognition device, such as a Google Glass® blink detector, that detects eye activity from the user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gesture as input to the input device (e.g., Google Glass®). Additionally, the user interface input device may include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., Siri® Navigator) through voice commands.

[0244] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, and eye-tracking devices. Additionally, user interface input devices may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0245] In general, use of the term output device(s) is intended to include all possible types of devices and mechanisms for outputting information from computer system 1200 to a user or to another computer. User interface output devices may also include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may be flat-panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touch screens, etc. For example, user interface output devices may include various display devices that visually convey textual, graphical, and audio / video information, such as, but not limited to, monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0246] The storage subsystem 1218 provides a repository or data store for storing information and data used by the computer system 1200. The storage subsystem 1218 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data structures that provide some example functionality. The storage subsystem 1218 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above. The software may be executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 can also provide authentication according to the teachings of the present disclosure.

[0247] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 12, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include several memories, including volatile main random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory, in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 1200, such as during start-up, may typically be stored in ROM. The RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 1204. In some embodiments, the system memory 1210 may include several different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0248] 12, system memory 1210 may load executing application programs 1212, program data 1214, and operating system 1216, which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc. By way of example, operating system 1216 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0249] The computer-readable storage medium 1222 can store programming and data structures that provide some example functionality. The computer-readable medium 1222 can provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1200. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above may be stored in the storage subsystem 1218. By way of example, the computer-readable storage medium 1222 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, and an optical disk drive, such as a Blu-ray® disk or other optical media. The computer-readable storage medium 1222 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash device, a Secure Digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, etc., DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0250] In some examples, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may be further connected to a computer-readable storage medium 1222. Reader 1220 may receive data from a memory device such as a disk, flash drive, or the like and may be configured to read the data.

[0251] In some examples, computer system 1200 may support virtualization technology, including, but not limited to, virtualization of processing and memory resources. For example, computer system 1200 may provide support for running one or more virtual machines. In some examples, computer system 1200 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may have allocated memory resources, computational resources (e.g., processors, cores), I / O resources, and networking resources. Each virtual machine generally operates independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines executed by computer system 1200. Thus, multiple operating systems may potentially be run simultaneously by computer system 1200.

[0252] The communications subsystem 1224 provides an interface to other computer systems and networks. The communications subsystem 1224 serves as an interface for receiving data from other systems and transmitting data from the computer system 1200 to other systems. For example, the communications subsystem 1224 can enable the computer system 1200 to establish a communications channel to one or more client devices over the Internet to receive and transmit information from the client devices. For example, when the computer system 1200 is used to execute the bot system 120 shown in FIG. 1, the communications subsystem can be used to communicate with a chatbot system selected for the application.

[0253] The communications subsystem 1224 can support both wired and / or wireless communications protocols. In certain examples, the communications subsystem 1224 can include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some examples, the communications subsystem 1224 can provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0254] The communications subsystem 1224 can receive and transmit data in various formats. In some examples, in addition to other formats, the communications subsystem 1224 can receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. For example, the communications subsystem 1224 can be configured to receive (or transmit) data feeds 1226 in real time from users of social media networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0255] In one example, the communications subsystem 1224 can be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230, which may be continuous or unlimited in nature with no apparent end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0256] The communications subsystem 1224 may also be configured to communicate data from the computer system 1200 to other computer systems or networks. The data may be communicated in a variety of different forms, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1200.

[0257] Computer system 1200 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, the description of computer device 1200 shown in FIG. 12 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 12. It will be appreciated that there are other manners and / or ways to implement the various examples based on the present disclosure and the teachings provided herein.

[0258] While specific examples have been described, various modifications, variations, alternative configurations, and equivalents are possible. The examples are not limited to operation in one particular data processing environment, but can freely operate in multiple data processing environments. Additionally, while certain examples are described using a particular sequence of transactions and steps, those skilled in the art will appreciate that this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may have additional steps not included in the figures. Various features and aspects of the examples described above may be used individually or jointly.

[0259] Additionally, while certain examples are described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. An example may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or on multiple different processors in any combination.

[0260] Where a device, system, component, or module is described as being configured to perform a certain operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, such as by executing computer instructions or code, or by a processor or core that is programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0261] Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the examples. This description provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the preceding description of the examples provides those skilled in the art with an empowering description for practicing various examples. Various changes can be made in the function and arrangement of elements.

[0262] Accordingly, the specification and drawings should be regarded in an illustrative, and not a restrictive, sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the appended claims. Accordingly, while particular examples have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0263] In the foregoing specification, aspects of the present disclosure have been described with reference to particular examples thereof, but those skilled in the art will recognize that the present disclosure is not limited thereto. Various features and aspects of the above-described disclosure may be used individually or jointly. Moreover, the examples may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the specification and drawings should be considered illustrative and not restrictive.

[0264] In the above description, for purposes of illustration, the methods are described in a particular order. It should be appreciated that in alternative examples, the methods may be performed in an order different from that described. It should also be appreciated that the methods described above may be performed by hardware components or may be embodied in a series of machine-executable instructions that can be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuitry that has been programmed with the instructions, to perform the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, or any other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0265] Where an element is described as being configured to perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0266] Although illustrative examples of the present application have been described in detail herein, it is to be understood that the inventive concepts may be variously embodied and utilized in other manners, and that the appended claims are intended to be construed to include such modifications except insofar as limited by the prior art.

Claims

1. 1. A computer-implemented method comprising: initializing a machine learning algorithm with a set of hyperparameter values; accessing a hyperparameter objective function that is at least partially defined for a plurality of domains of a search space associated with the machine learning algorithm; Including, the search space includes a training dataset and an evaluation dataset, each domain includes a subdivision of the search space having at least one training dataset and at least one evaluation dataset, and the hyperparameter objective function includes a domain score for each domain calculated based on the number of instances in the at least one evaluation dataset that are predicted correctly or incorrectly by the machine learning algorithm during a given trial; The computer-implemented method comprises: For each trial in the hyperparameter tuning process, training the machine learning algorithm for each domain using the at least one training dataset associated with each domain and the set of hyperparameter values; the training outputs a plurality of machine learning models, including a machine learning model for each domain; The computer-implemented method comprises: For each trial in the hyperparameter tuning process, evaluating the machine learning model for each domain using the at least one evaluation dataset associated with each domain and the set of hyperparameter values; the evaluating includes generating a domain score for each domain; The computer-implemented method comprises: For each trial in the hyperparameter tuning process, calculating an objective score for the current trial based on the domain scores for each domain and a domain weight associated with each domain using the hyperparameter objective function; determining whether the machine learning model has reached convergence based on the objective score of the current trial; providing at least one of the plurality of machine learning models in response to determining that the machine learning model has reached convergence; A computer-implemented method, including:

2. 2. The computer-implemented method of claim 1, wherein the hyperparameter objective function is formulated to normalize instance-level improvements and instance-level regressions over a total number of instances to obtain improvement scores and regression scores, and the domain score for each domain is calculated based on the improvement scores and the regression scores.

3. 3. The computer-implemented method of claim 2, wherein the improvement score is calculated based on a total count of (i) the number of instances in the at least one evaluation dataset that are correctly predicted by the machine learning model during the given trial, (ii) the number of instances in the at least one evaluation dataset that are incorrectly predicted by a baseline machine learning model, and (iii) the number of the instances in the at least one evaluation dataset; and the regression score is calculated based on (i) the number of instances in the at least one evaluation dataset that are incorrectly predicted by the machine learning model during the given trial, (ii) the number of instances in the at least one evaluation dataset that are correctly predicted by a baseline machine learning model, and (iii) the total count of the number of the instances in the at least one evaluation dataset.

4. 2. The computer-implemented method of claim 1, wherein the hyperparameter objective function is formulated to exclude unstable instances, which are instances in the at least one evaluation dataset that have been determined to produce different predicted outcomes from one another using the same machine learning model.

5. 5. The computer-implemented method of claim 4, wherein filtering out unstable instances comprises: (i) subtracting a count of the unstable instances to be filtered out from a number of the instances in the at least one evaluation data set that are correctly predicted by the machine learning model during the given trial; and (ii) subtracting the count of the unstable instances to be filtered out from a number of the instances in the at least one evaluation data set that are incorrectly predicted by the machine learning model during the given trial.

6. The computer-implemented method of claim 1 , wherein the hyperparameter objective function is formulated to include parameters for an acceptable regression ratio m for one or more of the multiple domains.

7. 7. The computer-implemented method of claim 6, wherein the parameter is defined based on a regression score, and if the regression score is smaller than the acceptable regression ratio m, the regression score is set to zero; otherwise, the regression score is calculated based on the total count of (i) the number of the instances in the at least one evaluation dataset that are incorrectly predicted by the machine learning model during the given trial, (ii) the number of the instances in the at least one evaluation dataset that are correctly predicted by a baseline machine learning model, and (iii) the number of the instances in the at least one evaluation dataset.

8. 1. A system comprising: one or more processors; one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations; Equipped with The operation is initializing a machine learning algorithm with a set of hyperparameter values; accessing a hyperparameter objective function that is at least partially defined for a plurality of domains of a search space associated with the machine learning algorithm; Including, the search space includes a training dataset and an evaluation dataset, each domain includes a subdivision of the search space having at least one training dataset and at least one evaluation dataset, and the hyperparameter objective function includes a domain score for each domain calculated based on the number of instances in the at least one evaluation dataset that are predicted correctly or incorrectly by the machine learning algorithm during a given trial; The operation is For each trial in the hyperparameter tuning process, training the machine learning algorithm for each domain using the at least one training dataset associated with each domain and the set of hyperparameter values; the training outputs a plurality of machine learning models, including a machine learning model for each domain; The operation is For each trial in the hyperparameter tuning process, evaluating the machine learning model for each domain using the at least one evaluation dataset associated with each domain and the set of hyperparameter values; the evaluating includes generating a domain score for each domain; The operation is For each trial in the hyperparameter tuning process, The domain score for each domain using the hyperparameter objective function, and calculating an objective score for the current trial based on a domain weight associated with each domain; determining whether the machine learning model has reached convergence based on the objective score of the current trial; providing at least one of the plurality of machine learning models in response to determining that the machine learning model has reached convergence; Including, the system.

9. 9. The system of claim 8, wherein the hyperparameter objective function is formulated to normalize instance-level improvements and instance-level regressions over a total number of instances to obtain improvement scores and regression scores, and the domain score for each domain is calculated based on the improvement scores and the regression scores.

10. 10. The system of claim 9, wherein the improvement score is calculated based on a total count of (i) the number of instances in the at least one evaluation dataset that are correctly predicted by the machine learning model during the given trial, (ii) the number of instances in the at least one evaluation dataset that are incorrectly predicted by a baseline machine learning model, and (iii) the number of the instances in the at least one evaluation dataset, and the regression score is calculated based on (i) the number of instances in the at least one evaluation dataset that are incorrectly predicted by the machine learning model during the given trial, (ii) the number of instances in the at least one evaluation dataset that are correctly predicted by a baseline machine learning model, and (iii) the total count of the number of the instances in the at least one evaluation dataset.

11. 10. The system of claim 8, wherein the hyperparameter objective function is formulated to exclude unstable instances, which are instances in the at least one evaluation dataset that have been determined to produce different predicted outcomes from one another using the same machine learning model.

12. 12. The system of claim 11, wherein filtering out unstable instances comprises: (i) subtracting a count of the unstable instances to be filtered out from a number of the instances in the at least one evaluation data set that are correctly predicted by the machine learning model during the given trial; and (ii) subtracting the count of the unstable instances to be filtered out from a number of the instances in the at least one evaluation data set that are incorrectly predicted by the machine learning model during the given trial.

13. The system of any one of claims 8 to 12, wherein the hyperparameter objective function is formulated to include a parameter for an acceptable regression ratio m of one or more of the plurality of domains.

14. 14. The system of claim 13, wherein the parameter is defined based on a regression score, and if the regression score is smaller than the acceptable regression ratio m, the regression score is set to zero; otherwise, the regression score is calculated based on the total count of (i) the number of the instances in the at least one evaluation dataset that are incorrectly predicted by the machine learning model during the given trial, (ii) the number of the instances in the at least one evaluation dataset that are correctly predicted by a baseline machine learning model, and (iii) the number of the instances in the at least one evaluation dataset.

15. A program for causing a computer to execute the method described in any one of claims 1 to 7.

Images