Calibrating confidence scores for machine learning models trained as natural language interfaces

Abstract

A technique for calibrating a confidence score of a machine learning model trained to convert natural language into semantic language is disclosed herein. The technique includes obtaining one or more raw beam scores generated from one or more beam levels of a decoder of the machine learning model trained to convert natural language into logical forms, each of the one or more raw beam scores being a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more beam levels, and further including classifying, by a calibration model, the logical form output by the machine learning model as accurate or incorrect based on the one or more raw beam scores, and providing the logical form with a confidence score determined based on the classification of the logical form.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Nonprovisional Patent Application No. 18 / 107,624, filed February 9, 2023, which claims the benefit of and priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 400,037, filed August 22, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] Field The present disclosure relates generally to converting natural language into semantic representation languages, and more particularly to machine learning techniques for calibrating confidence scores of machine learning models that have been trained to convert natural language into semantic representation languages ​​such as Oracle Semantic Representation Language (OMRL). [Background technology]

[0003] background Many users around the world rely on instant messaging or chat platforms for instant responses. Organizations often use these instant messaging or chat platforms to engage with customers (or end users) in live conversations. However, it can be very costly for organizations to hire service personnel to participate in live communications with customers or end users. Chatbots or bots are beginning to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with bots through messaging apps that the end users already have installed and use. Intelligent bots, generally equipped with artificial intelligence (AI), can communicate more intelligently and contextually in live conversations, thus enabling more natural conversations between bots and end users for an improved conversational experience. Instead of end users learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots may be able to understand the end user's intent based on user utterances in natural language and respond accordingly.

[0004] Chatbots may have both analog (human) and digital (machine) interfaces for interacting with humans and connecting to back-end systems. When humans speak using natural language, it is advantageous to be able to extract and parse the meaning of the utterance (e.g., a request) regardless of how the back-end system addresses the utterance. As an example, the request may be for data that needs to be retrieved from a relational database, or the requested data may need to be extracted from a knowledge graph. A semantic representation language (MRL) is a generic representation of natural language utterances that a chatbot can translate into any number of target machine-oriented languages. Thus, MRL can be utilized by chatbots to communicate interchangeably with both humans and a variety of back-end systems, including systems that communicate using Structured Query Language (SQL), Application Programming Interfaces (APIs), Representational State Transfer (REST), Graph Query Language (GraphQL), Property Graph Query Language (PGQL), etc.

[0005] For example, SQL is a standard database management language for interacting with relational databases. SQL can be used to store, manipulate, retrieve, and / or otherwise manage data held in a relational database management system (RDBMS) and / or for stream processing in a relational data stream management system (RDSMS). SQL includes statements or commands used to interact with relational databases. SQL statements or commands are categorized into, among others, Data Query Language (DQL) statements, Data Definition Language (DDL) statements, Data Control Language (DCL) statements, and Data Manipulation Language (DML) statements. To interact with a relational database using SQL, a user must know how the database is structured (e.g., knowledge of the tables and the rows and columns within each table), SQL syntax, and how the syntax relates to the database structure. Without this knowledge, it is often difficult for a user to interact with these relational databases using SQL.

[0006] Natural language interfaces (e.g., chatbots) to database systems (NLIDBs) such as RDBMSs provide users with a means to intuitively interact with these relational databases without requiring knowledge of database management languages. For example, using natural language statements and queries (i.e., natural language queries), users can interact with these relational databases through NLIDBs using easy-to-understand language. Recently, text-to-SQL systems have become popular, and deep learning techniques for converting natural language queries into SQL queries have proven promising. Semantic parsing can be used to convert natural language statements, requests, and questions (i.e., sentences) into machine-oriented language that can be executed by applications (e.g., chatbots, models, programs, machines, etc.). For example, semantic parsing can convert natural language sentences into general-purpose programming languages ​​such as Python, Java, and SQL. Processes for converting natural language sentences into SQL queries typically involve rule-based, statistical-based, and / or deep learning-based systems. Rule-based systems typically use a set of fixed rules to convert natural language sentences into SQL queries. These rule-based systems are generally domain-specific and therefore considered inelastic, and do not generalize well to novel use cases (e.g., across different domains). Statistical-based systems, such as slot-filling, label tokens (i.e., words or phrases) in an input natural language sentence according to their semantic role within the sentence and use the labels to fill slots in an SQL query. These statistical-based systems are generally limited in the types of sentences they can parse (e.g., the sentence must be capable of being represented as a parse tree). Deep learning-based systems, such as sequence-to-sequence models, involve training deep learning models that directly convert natural language sentences into machine-oriented language and have been shown to generalize across tasks, domains, and datasets.However, such deep learning systems require a large amount of training data for supervised learning, and it is difficult to obtain labeled data (e.g., natural language query-SQL statement pairs). Therefore, deep learning-based conversion of natural language sentences into machine-oriented language may inevitably require a large amount of labeled training data. Summary of the Invention

[0007] Quick Overview Machine learning techniques are provided (e.g., methods, systems, non-transitory computer-readable media storing code or instructions executable by one or more processors) for techniques for calibrating confidence scores of machine learning models that have been trained to convert natural language into a semantic representation language, such as Oracle Semantic Representation Language (OMRL).

[0008] In various embodiments, a computer-implemented method (e.g., for processing natural language) includes obtaining one or more raw beam scores generated from one or more beam levels of a decoder of a machine learning model trained to convert natural language into a logical form, each of the one or more raw beam scores being a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more beam levels, and the computer-implemented method further includes classifying the logical form output by the machine learning model as accurate or incorrect based on the one or more raw beam scores, wherein the classifying includes inputting the one or more raw beam scores into a calibration model and Thus, a computer-implemented method is provided that includes predicting a correct or incorrect class of the logical form based on one or more raw beam scores, where the correct or incorrect class represents a determination by the calibration model as to whether the subtree with the highest raw beam score at the last beam level of the decoder represents a correct or incorrect transformation of the natural language utterance, the subtree with the highest raw beam score representing the logical form, and the classifying further includes outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest raw beam score is correct or incorrect, and the computer-implemented method further includes providing the logical form with a confidence score determined based on the conditional probability score.

[0009] In some embodiments, the computer-implemented method further includes converting the natural language utterance into a logical form by a machine learning model, the conversion comprising a bottom-up generation process performed by a heuristic search algorithm, the bottom-up generation process including: for a first beam level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form and selecting the top K subtrees to retain using the raw beam scores associated with each subtree; for each beam level after the first beam level, generating F possible subtrees of the natural language utterance based on the vocabulary or grammar associated with the logical form and the top K subtrees retained from the previous beam level and selecting another top K subtrees to retain using the raw beam scores associated with each subtree; and for the last beam level, determining the subtree with the highest raw beam score as the final output of the decoder.

[0010] In some embodiments, the one or more beam levels include a first beam level, a last beam level, one or more beam levels between the first beam level and the last beam level, or any combination thereof.

[0011] In some embodiments, the one or more raw beam scores are obtained from the first beam level, the last beam level, one or more beam levels between the first beam level and the last beam level, or any combination thereof.

[0012] In some embodiments, the one or more raw beam scores are beam scores of the top K subtrees retained from the first beam level, the last beam level, and one or more beam levels between the first beam level and the last beam level.

[0013] In some embodiments, the conditional probability scores are modeled using a sigmoid function, and the conditional probability scores are used as confidence scores for the logical form.

[0014] In some embodiments, the computer-implemented method further includes determining that the confidence score exceeds a predetermined threshold, and, in response to determining that the confidence score exceeds the predetermined threshold, executing a query against a database based on the logical form.

[0015] In various embodiments, a system is provided that includes one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform some or all of one or more methods disclosed herein.

[0016] In various embodiments, a computer program product tangibly embodied in one or more non-transitory machine-readable media that includes instructions executed by one or more processors to perform some or all of one or more of the methods disclosed herein.

[0017] In various embodiments, an apparatus is provided that includes means for performing some or all of the operations and / or methods disclosed herein.

[0018] In various embodiments, a computer program product is provided that includes computer instructions that, when executed by one or more processors, perform some or all of the operations and / or methods disclosed herein.

[0019] The techniques described above and below can be implemented in multiple ways and in multiple contexts. Some exemplary implementations and contexts are provided with reference to the accompanying drawings, as described in more detail below. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a simplified block diagram of a distributed environment incorporating an illustrative embodiment. [Figure 2] FIG. 1 is a simplified block diagram of a computing system implementing a masterbot according to one embodiment. [Figure 3] FIG. 1 is a simplified block diagram of a computing system implementing a skillbot according to one embodiment. [Figure 4A] FIG. 1 illustrates an exemplary confidence diagram for calibration of a conversation-to-OMRL (C2OMRL) model in accordance with various embodiments. [Figure 4B] FIG. 1 illustrates an exemplary confidence diagram for calibration of a conversation-to-OMRL (C2OMRL) model in accordance with various embodiments. [Figure 5] FIG. 1 is a simplified block diagram of a C2OMRL architecture in accordance with various embodiments. [Figure 6] FIG. 1 illustrates a beam search process for a decoder according to various embodiments. [Figure 7] FIG. 1 illustrates a confidence score calibration system for confidence scores according to various embodiments. [Figure 8] FIG. 1 illustrates a model system for training and deploying models according to various embodiments. [Figure 9] FIG. 1 illustrates a process flow for calibrating confidence scores of machine learning models trained to convert natural language into semantic language, according to various embodiments. [Figure 10] FIG. 1 is a simplified diagram of a distributed system for implementing various embodiments. [Figure 11]FIG. 1 is a simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment system can be provided as cloud services, according to various embodiments. [Figure 12] FIG. 1 illustrates an exemplary computer system that can be used to implement various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0021] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of certain embodiments. It will be understood, however, that various embodiments may be practiced without these specific details. The drawings and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0022] introduction In recent years, the amount of data feeding into various industries and their systems has grown exponentially. Most business information is stored in the form of relational databases, which store, process, and retrieve data. Databases power information systems across multiple industries, such as consumer technology (e.g., orders, cancellations, refunds), supply chains (e.g., raw materials, inventory, suppliers), healthcare (e.g., medical records), finance (e.g., financial performance indicators), customer support, and search engines. For modern data-driven enterprises, tracking the real-time state of their business is essential to quickly understand and diagnose any emerging issues, trends, or anomalies in the data and take immediate corrective action. This task is typically performed manually by analysts, who write complex queries in query languages ​​such as SQL, PGQL, logical database queries (e.g., database query languages ​​such as declarative query languages), and API query languages ​​such as GraphQL and REST. Writing such queries can be used to derive insightful information from data stored in multiple tables. These results are typically processed in the form of charts or graphs to allow users to quickly visualize the results and facilitate data-driven decision making.

[0023] While common database queries (e.g., SQL queries) are often pre-defined and built into products, any new or later-created queries still need to be manually coded by analysts. Such static interaction between database queries and the consumption of corresponding results requires time-consuming manual intervention, resulting in a slow feedback cycle. It is far more efficient to allow non-technical users (e.g., business leaders, physicians, or other users of data) to interact directly with analytical tables through natural language (NL) queries that abstract the underlying query language (e.g., SQL) code. Defining database queries requires a deep understanding of database schemas and query language syntax, which can quickly become overwhelming for novices and non-technical stakeholders. Efforts to bridge this communication gap have resulted in the development of a new type of processing called NLIDB. This natural search functionality has become more popular in recent years as companies develop deep learning methods for natural language to logical form (NL2LF) translation, such as natural language to SQL (NL2SQL). Logical forms may refer to semantic and / or machine-oriented languages. NL2SQL seeks to convert natural language questions into SQL, allowing individuals to run unstructured queries against databases. Converted SQL may also enable digital assistants, such as chatbots, to improve their responses when answers can be found in multiple different databases or tables.

[0024] However, in addition to logical forms such as SQL or Oracle's proprietary SQL-like semantic representation language, i.e., Oracle Semantic Representation Language (OMRL), it is important to have a confidence score associated with the logical form output produced by an NL2LF model (e.g., an NL2SQL model or a Conversational-OMRL (C2OMRL) model). A confidence score is a metric, such as a number between 0 and 1, that represents the likelihood that the machine learning model's output is accurate and will satisfy the user's request. Confidence scores are important because output from NL2LF models can sometimes be difficult to interpret, and it is difficult to understand how confident the model was about the predicted logical form (e.g., OMRL) just by looking at the raw output. Confidence scores enable better interpretation of responses, analysis of model shortcomings, and presentation of uncertainty to end users. In particular, when a C2OMRL model is used as a skill in a digital assistant (DA), a low confidence score can be used to trigger a disambiguation prompt that is exposed to the user. Confidence scores can also be leveraged for internal evaluation studies to better understand model vulnerabilities for certain categories of example inputs. However, for a confidence score to be useful, it is essential that it be calibrated, i.e., that it can be directly interpreted as a probability of model accuracy.

[0025] Therefore, an approach to address these and other challenges is needed. The approach described herein involves training and deploying a confidence score calibration model that utilizes intermediate “raw” scores produced in one or more steps of the decoding process of an NL2LF model to determine a calibrated confidence score for the NL2LF model. More specifically, the decoder of the NL2LF model is based on a bottom-up generation process, and the final generated output is a logical form tree (i.e., a tree-based structure representing the entire logical form). The generation process operates in steps also referred to as “beam levels.” In each step (e.g., “step i”), the process generates all possible subtrees that can be obtained by expanding the current subtree (from step “i−1”), and then retains only the top K subtrees using the raw scores associated with each subtree. These scores are referred to herein as “raw beam scores,” and thus the top K intermediate results (considered in the next generation step) are the K intermediate results with the highest raw beam scores. A confidence score calibration model (also referred to herein as a calibration model) is a binary classifier that takes as input a vector having one or more raw beam scores for one or more beam levels and uses the one or more raw beam scores to predict whether the final output of the NL2LF model (e.g., the final ORML tree) will be accurate or inaccurate. The probability score from the calibration model (e.g., the output of a sigmoid function in logistic regression) becomes the calibrated NL2LF model confidence score. Advantageously, the raw beam scores of candidate subtrees can reveal information about the NL2LF model uncertainty in the bottom-up generation process (e.g., caused by model vulnerability to some types of examples or by ambiguous inputs), and therefore they are a good source of information for the calibrated confidence score.

[0026] In various embodiments, a computer-implemented method is provided that includes obtaining one or more raw beam scores generated from one or more beam levels of a decoder of a machine learning model trained to convert natural language into a logical form. Each of the one or more raw beam scores is a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more beam levels. The computer-implemented method further includes classifying the logical form output by the machine learning model as correct or incorrect based on the one or more raw beam scores. The classification includes inputting the one or more raw beam scores into a calibration model and predicting, by the calibration model, a correct or incorrect class of the logical form based on the one or more raw beam scores, where the correct or incorrect class represents a determination by the calibration model as to whether a subtree with the highest raw beam score at the last beam level of the decoder represents an accurate or incorrect conversion of a natural language utterance, the subtree with the highest raw beam score representing the logical form, and outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest raw beam score is accurate or incorrect. The computer-implemented method further includes providing a confidence score, determined based on the conditional probability score, to the logical form.

[0027] As used herein, when an action is "based on" something, this means that the action is at least partially based on at least a portion of that something. As used herein, the terms "similarly," "substantially," "approximately," and "about" are defined as being loosely specified (and being fully specified), as understood by those of ordinary skill in the art. In any disclosed embodiment, the terms "similarly," "substantially," "approximately," or "about" may be substituted for "within [a percentage] of" what is specified, where percentage includes 0.1, 1, 5, and 10 percent.

[0028] Bots and Analytics Systems A bot (also referred to as a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. A bot can generally respond to natural language messages (e.g., questions or comments) through a messaging application using natural language messages. A business can use one or more bot systems to communicate with end users through messaging applications. The messaging application, sometimes referred to as a channel, may be an end user's preferred messaging application that the end user already has installed and is familiar with. Thus, an end user does not need to download and install a new application to chat with a bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat functionality or voice-based input (such as devices or apps with interfaces that use Siri, Cortana, Google® Voice, or other voice input for interaction).

[0029] In some examples, a bot system may be associated with a Uniform Resource Identifier (URI). The URI may identify the bot system using a string of characters. The URI may be used as a webhook for one or more messaging application systems. The URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The bot system may be designed to receive a message (e.g., a Hypertext Transfer Protocol (HTTP) POST call message) from the messaging application system. The HTTP POST call message may be directed from the messaging application system to the URI. In some embodiments, the message may be different from an HTTP POST call message. For example, the bot system may receive a message via Short Message Service (SMS). While discussions herein may refer to communications received by the bot system as a message, it should be understood that the message may be an HTTP POST call message, an SMS message, or any other type of communication between two systems.

[0030] End users can interact with bot systems through conversational interactions (sometimes referred to as a conversational user interface (UI)), just like interactions between humans. In some cases, an interaction may involve the end user saying "Hello" to the bot, and the bot responding "Hi" and asking the end user how it can help. In some cases, the interaction may also be a transactional interaction, for example with a banking bot, such as transferring money from one account to another, an informational interaction, for example with an HR bot, such as checking remaining vacation days, or an interaction, for example with a retail bot, such as discussing a purchase return or requesting technical support.

[0031] In some embodiments, the bot system can intelligently handle end-user interactions without interacting with an administrator or developer of the bot system. For example, an end user may send one or more messages to the bot system to achieve a desired goal. The messages may include content, such as text, emojis, audio, images, videos, or other methods of conveying a message. In some embodiments, the bot system can convert the content into a standardized format (e.g., a Representational State Transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system can also initiate communication with the end user rather than passively responding to end-user utterances. Various techniques for identifying explicit invocations of a bot system and determining inputs to the bot system being invoked are described herein. In one embodiment, explicit invocation analysis is performed by a master bot based on detecting an invocation name in the utterance. In response to detecting an invocation name, the utterance can be refined for inputs to a skill bot associated with the invocation name.

[0032] A conversation with a bot may follow a specific conversational flow that includes multiple states. The flow can define what happens next based on input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intent) and actions to take in the states or from state to state. The conversation can take different paths based on end-user input, which can influence the decisions the bot makes for the flow. For example, in each state, based on the end-user input or utterance, the bot can determine the end-user's intent to determine the appropriate action to take next. As used herein and in the context of utterances, the term "intent" refers to the intent of the user who provided the utterance. For example, a user may intend to engage a bot in a conversation to order a pizza, and thus the user's intent may be expressed through the utterance "Order pizza." The user's intent may be directed toward a specific task the user wants the chatbot to perform on their behalf. Thus, an utterance may be phrased as a question, command, request, etc. that reflects the user's intent. Intent may include a goal that the end user wishes to achieve.

[0033] In the context of configuring a chatbot, the term “intent” is used herein to refer to configuration information for mapping a user's utterance to a specific task / action or category of task / action that the chatbot can perform. To distinguish between the intent of an utterance (e.g., a user's intent) and the intent of a chatbot, the latter may be referred to herein as a “bot intent.” A bot intent may include a set of one or more utterances associated with the intent. For example, the intent to order a pizza may be communicated by various permutations of an utterance expressing a desire to place a pizza order. These associated utterances can be used to train the chatbot's intent classifier, enabling the intent classifier to subsequently determine whether an input utterance from a user matches the intent to order a pizza. A bot intent may be associated with one or more dialog flows to initiate a conversation with a user in a state. For example, the first message of an intent to order a pizza may be the question, “What kind of pizza would you like?” In addition to the associated utterance, a bot intent may further include a named entity associated with the intent. For example, an intent to order a pizza may include variables or parameters used to perform the task of ordering a pizza, such as, for example, topping 1, topping 2, pizza type, pizza size, quantity of pizza, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0034] FIG. 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to one embodiment. The environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as shown in FIG. 1, a user 104 representing a particular business can use the DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, the DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform can be used by multiple businesses to create digital assistants. As another example, a restaurant (e.g., a pizza shop) owner may use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0035] For purposes of this disclosure, a "digital assistant" is an entity that helps a user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity that runs using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or executed in various physical systems or devices, such as a computer, a mobile phone, a watch, an appliance, a vehicle, etc. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0036] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and get responses 112 back from the digital assistant 106. A conversation can include one or more of the inputs 110 and responses 112. Through these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested task and respond to the user with an appropriate response.

[0037] User input 110 is generally in the form of natural language and is referred to as speech. User utterance 110 can be in text form, such as when a user types in a sentence, a question, a text fragment, or even a single word and provides it as input to digital assistant 106. In some embodiments, user utterance 110 can be in the form of acoustic input or utterance, such as when a user says or speaks something that is provided to digital assistant 106 as input. The utterance is typically in the language that user 108 speaks. For example, the utterance may be in English or some other language. When the utterance is in the form of an utterance, the utterance input is converted into text form speech in that particular language, and the text speech is then processed by digital assistant 106. Various utterance-to-text processing techniques may be used to convert the utterance or acoustic input into text speech that is then processed by digital assistant 106. In some embodiments, the utterance-to-text conversion may be performed by digital assistant 106 itself.

[0038] The utterance, which may be a text utterance or a spoken utterance, can be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing of the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, including identifying one or more intents and one or more entities that correspond to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 can perform one or more actions or behaviors in response to the understood meaning or intent. For purposes of this disclosure, it is assumed that the utterance is either provided directly by a user 108 of the digital assistant 106 or is a text utterance that is the result of converting an input spoken utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0039] For example, user 108 input can request that a pizza be ordered by providing an utterance such as, "I want to order a pizza." Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action may include, for example, responding to the user with a question requesting user input regarding the type of pizza the user desires to order, the size of the pizza, any toppings on the pizza, etc. The responses provided by digital assistant 106 may also be in natural language format and typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 may perform natural language generation (NLG). For a user ordering a pizza, through a conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the information required for the pizza order so that the pizza is ordered at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0040] At a conceptual level, digital assistant 106 performs various processing in response to utterances received from a user. In some embodiments, this processing involves a sequence or pipeline of processing steps including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to be performed in response to the utterance, causing the action to be performed, if appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing can include analyzing the received input utterance to understand the structure and meaning of the utterance, refining and reshaping the utterance to develop a better understandable form (e.g., logical form) or structure of the utterance, etc. Generating a response may include using NLG techniques.

[0041] NLU processing performed by a digital assistant such as digital assistant 106 can include various NLP-related processes such as sentence analysis (e.g., tokenization, lemmatization, identifying utterance part tags for sentences, identifying named entities within sentences, generating dependency trees to represent sentence structure, dividing sentences into clauses, analyzing individual clauses, resolving anaphora, performing chunking, etc.). In one embodiment, the NLU processing, or portions thereof, is performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of input spoken sentences can be identified by processing the sentences using a parser, utterance part tagger, and / or named entity recognizer. In one embodiment, for the English language, parsers, utterance part taggers, and named entity recognizers, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0042] Although the various examples provided in this disclosure show speech in the English language, this is intended as an example only. In some embodiments, the digital assistant 106 is also capable of handling speech in languages ​​other than English. The digital assistant 106 can provide subsystems (e.g., components that perform NLU functions) configured to perform processing for multiple different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes NLU processing flexible and extensible for each language, including allowing for different orders of processing. Language packs can be provided for individual languages, and the language packs can register a list of subsystems that can be serviced by the NLU core server.

[0043] A digital assistant, such as the digital assistant 106 shown in FIG. 1, can be available or accessible to its user 108 through a variety of different channels, including, but not limited to, through an application, through a social media platform, through various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for it, such that the digital assistant can be powered by and accessed on multiple different services simultaneously.

[0044] A digital assistant or chatbot system generally encompasses or is associated with one or more skills. In one embodiment, these skills are individual chatbots (referred to as skillbots) that are configured to interact with a user and perform specific types of tasks, such as tracking inventory, submitting a timecard, creating an expense report, ordering food, checking a bank account, making a reservation, purchasing a widget, etc. For example, in the embodiment shown in FIG. 1 , the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skillbot” and “skillbots,” respectively.

[0045] Each skill associated with a digital assistant helps a user of the digital assistant complete a task through a conversation with the user, where the conversation may include a combination of text or audio input provided by the user and responses provided by a skill bot. These responses may be in the form of text or audio messages to the user and / or may use simple user interface elements (e.g., selection lists) presented to the user for the user to make a selection.

[0046] There are various ways in which skills or skill bots can be associated with or added to a digital assistant. In some cases, skill bots can be developed by a company and then added to a digital assistant using DABP 102. In some cases, skill bots can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In still other cases, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills targeted at a wide range of tasks. Skills offered through the skill store may also expose various cloud services. To add a skill to a digital assistant being created using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and indicate that the selected skill should be added to the digital assistant being created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in a modified form (e.g., a user of DABP102 may select and clone a particular skillbot provided by the skill store, make customizations or modifications to the selected skillbot, and then add the modified skillbot to a digital assistant created using DABP102).

[0047] A variety of different architectures may be used to implement a digital assistant or chatbot system. For example, in one embodiment, a digital assistant created and deployed using DABP 102 may be implemented using a masterbot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a masterbot that interacts with one or more child bots, which are skillbots. For example, in the embodiment shown in FIG. 1, the digital assistant 106 includes a masterbot 114 and skillbots 116-1, 116-2, e.g., that are child bots of the masterbot 114. In one embodiment, the digital assistant 106 itself may act as the masterbot.

[0048] A digital assistant implemented according to a master-child bot architecture allows a user of the digital assistant to interact with multiple skills through a unified user interface, i.e., via a master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself. If not, the master bot selects a suitable skill bot to handle the user request and routes the conversation to the selected skill bot. This allows a user to converse with the digital assistant through a common, single interface while still providing the ability to use several skill bots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the digital assistant's master bot may interface with skill bots having specific functions, such as a CRM bot for performing functions related to customer relationship management (CRM), an ERP bot for performing functions related to enterprise resource planning (ERP), an HCM bot for performing functions related to human capital management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master bot interface, and behind the scenes, multiple skill bots are provided to handle user requests.

[0049] In one embodiment, in a masterbot / childbot infrastructure, the masterbot is configured to recognize an available list of skillbots. The masterbot has access to various available skillbots and, for each skillbot, metadata identifying the skillbot's capabilities, including tasks that can be performed by the skillbot. Upon receiving a user request in the form of an utterance, the masterbot is configured to identify or predict a particular skillbot from multiple available skillbots that can best service or address the user request. The masterbot then routes the utterance (or a portion of the utterance) to that particular skillbot for further handling. Thus, control flows from the masterbot to the skillbot. The masterbot can support multiple input and output channels. In one embodiment, routing may be performed with the assistance of processing performed by one or more available skillbots. For example, as described below, a skillbot can be trained to infer the intent of an utterance and determine whether the inferred intent matches an intent for which the skillbot is configured. Thus, routing performed by the masterbot may involve the skillbot communicating to the masterbot an indication of whether the skillbot is configured with an intent suitable for handling the utterance.

[0050] 1 illustrates a digital assistant 106 with a masterbot 114 and skillbots 116-1, 116-2, and 116-3, but this is not intended to be limiting. A digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an embodiment using a combination of software and hardware.

[0051] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants, including one or more skillbots associated with the digital assistant. In some cases, a skillbot can be created by cloning an existing skillbot, for example, by cloning a skillbot provided by a skill store. As indicated above, DABP 102 provides a skill store or skill catalog that provides multiple skillbots for performing various tasks. A user of DABP 102 can clone a skillbot from the skill store. Modifications or customizations can be made to the cloned skillbot as needed. In some other cases, a user of DABP 102 created a skillbot from scratch using tools and services provided by DABP 102. As indicated above, a skill store or skill catalog provided by DABP 102 can provide multiple skillbots for performing various tasks.

[0052] In one embodiment, at a high level, creating or customizing a skillbot involves the following steps: (1) Configure the settings for the new skill bot. (2) Configure one or more intents for the skillbot. (3) Constitute one or more entities of one or more intentions. (4) Train the skill bot. (5) Create a dialog flow for the skill bot. (6) Add custom components to your skill bot as needed. (7) Test and deploy the skill bot.

[0053] Each of the above steps is briefly described below. (1) Configure Settings for a New Skillbot—Various settings can be configured for a skillbot. For example, a skillbot designer can specify one or more invocation names for the skillbot being created. These invocation names can then be used by users of the digital assistant to explicitly invoke the skillbot. For example, a user can enter the invocation name in the user's utterance to explicitly invoke the corresponding skillbot.

[0054] (2) Configure one or more intents and associated example utterances for the skillbot—A skillbot designer specifies one or more intents (also referred to as bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes that the skillbot is trained to infer for input utterances. Upon receiving an utterance, the trained skillbot infers the intent of the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes an appropriate action to respond to the utterance based on the inferred intent for that utterance. In some cases, the intents of the skillbot represent tasks that the skillbot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot being trained for banking, the intents specified for the skillbot may include “CheckBalance,” “TransferMoney,” “DepositCheck,” etc.

[0055] For each intent defined for a skillbot, the skillbot designer may also provide one or more example utterances that express and illustrate the intent. These example utterances are intended to represent utterances a user might input into the skillbot for that intent. For example, for a CheckBalance intent, example utterances may include, "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account," etc. Thus, various permutations of typical user utterances may be specified as example utterances for the intent.

[0056] The intents and their associated example utterances are used as training data to train the skill bot. A variety of different training techniques may be used. This training results in a predictive model being generated that is configured to receive an utterance as input and output an intent inferred by the predictive model for the utterance. In some cases, the input utterance is provided to an intent analysis engine that is configured to predict or infer the intent of the input utterance using the trained model. The skill bot can then take one or more actions based on the inferred intent.

[0057] (3) Configuring One or More Intent Entities for a Skill Bot—In some cases, additional context may be required for a skill bot to respond appropriately to a user utterance. For example, there may be situations in which a user input utterance resolves to the same intent within a skill bot. For example, in the above example, "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests asking about different things. To disambiguate such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "savings" can enable the skill bot to parse the user request and respond appropriately. In the above example, the utterances resolve to the same intent, but the values ​​associated with the AccountType entity are different for the two utterances. This allows the skill bot to perform potentially different actions for the two utterances, even though they resolve to the same intent. One or more entities may be specified for an intent configured for a skill bot. Thus, entities are used to add context to the intent itself: they help to more fully describe the intent and enable the skill bot to complete the user request.

[0058] In one embodiment, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by a skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, repeating time period, currency, phone number, URL, etc. Custom entities are used for more customized purposes. For example, for a banking skill, an AccountType entity can be defined by the skill bot designer that enables various banking transactions by checking user input for keywords such as checking, savings, and credit cards.

[0059] (4) Train the Skillbot—The skillbot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select an intent associated with the received user input. As indicated above, the skillbot must be trained for this. In one embodiment, the skillbot is trained based on the intents configured for the skillbot and example utterances associated with the intents (collectively, training data) so that the skillbot can resolve user input utterances into one of its configured intents. In one embodiment, the skillbot is trained using the training data and uses a predictive model that enables the skillbot to determine what the user says (or in some cases, what they intend to say). DABP 102 provides a variety of different training techniques that can be used by the skillbot designer to train the skillbot, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In one embodiment, a portion (e.g., 80%) of the training data is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as a trained skill bot) can then be used to address and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for a skill bot. This enables the skill bot to output a response to a user request without having to update the dialog definition. A Q&A intent is created similarly to a regular intent. The dialog flow of a Q&A intent may differ from that of a regular intent.

[0060] (5) Create a Dialog Flow for the Skill Bot—The dialog flow specified for a skill bot describes how the skill bot reacts as its different intents are resolved in response to incoming user input. Dialog flow defines the actions or behaviors the skill bot performs, such as how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. Dialog flow is like a flowchart that the skill bot follows. Skill bot designers specify the dialog flow using a language such as Markdown. In one embodiment, a version of YAML called OBotML may be used to specify the dialog flow for a skill bot. The dialog flow definition for a skill bot acts as a model of the conversation itself, i.e., allowing the skill bot designer to orchestrate the interaction between the skill bot and the user it serves.

[0061] In one embodiment, a skill bot's dialog flow definition contains the following three sections: (a) Context Section (b) Default transition section (c) Status section Context Section - The context section allows the skill bot designer to define variables that will be used within the conversation flow. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to recognize and assert user preferences, etc.

[0062] Default Transition Section - Transitions for a skill bot can be defined in the dialog flow state section or the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there are no applicable transitions defined in a state or when the conditions required to trigger a state transition cannot be met. The default transition section can be used to define routing that allows the skill bot to gracefully handle unexpected user behavior.

[0063] State Section - The dialog flow and its associated behavior are defined as a series of transition states that govern the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality needed at that point in the dialog. Thus, states are built around components. States encompass component-specific characteristics and define transitions to other states that are triggered after the component is executed.

[0064] The state section can be used to address special case scenarios. For example, there may be times within a digital assistant when you want to temporarily exit a first skill that the user is engaged in and provide the user with the option to do something in a second skill. For example, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), the user may want to jump to a banking skill (e.g., the user may want to ensure they have enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, you can configure an action within a first skill to initiate an interaction with a second, different skill within the same digital assistant and then return to the original flow.

[0065] (6) Adding Custom Components to a Skillbot—As described above, states specified in a skillbot's dialog flow nominate components that provide the required functionality for that state. Components enable a skillbot to perform its functions. In one embodiment, DABP 102 provides a set of pre-configured components for performing a wide range of functions. A skillbot designer can select one or more of these pre-configured components and associate them with states in the skillbot's dialog flow. A skillbot designer can also create custom or new components using tools provided by DABP 102 and associate the custom components with one or more states in the skillbot's dialog flow.

[0066] (7) Testing and Deploying Skillbots - DABP102 provides several features that allow skillbot designers to test the skillbots they are developing, which can then be deployed and included within a digital assistant.

[0067] While the above description describes how to create a skillbot, similar techniques may also be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without invoking a skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit, which applies when a user signals a desire to exit the current conversation or context within the digital assistant; (2) Help, which applies when a user seeks help or orientation; and (3) UnresolvedIntent, which applies to user input that does not match well with the exit and help intents. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information allows the masterbot to select a specific skillbot to handle an utterance.

[0068] At the MasterBot or Digital Assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and associated conversation. The digital assistant makes this determination using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance should be routed to a specific skill for action, should be actioned by the digital assistant or MasterBot itself via a built-in system intent, or should be actioned as a different state within the current conversation flow.

[0069] In one embodiment, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skill bot corresponding to the invocation name. In such a scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further handling. In the absence of a specific or explicit invocation, in one embodiment, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and the skill bot associated with the digital assistant. The calculated score for the skill bot or system intent represents how likely the user input represents the task the skill bot is configured to perform or represents the system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further handling of the user input utterance. In one embodiment, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model of each skill), and a confidence score is determined for each intent. Generally, any intent with a confidence score above a threshold (e.g., 70%) is treated as a candidate intent. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0070] FIG. 2 is a simplified block diagram of a MasterBot (MB) system 200 according to one embodiment. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skillbot invoker 240, and a data store 250. The MB system 200 shown in FIG. 2 is only one example of an arrangement of components within a MasterBot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some other embodiments, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0071] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in a variety of modalities, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the audio to text using an utterance-to-text converter (not shown) that inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text.

[0072] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. The manner in which the utterance 202 is addressed depends on the language, as each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of the utterance.

[0073] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (e.g., VB), etc. The language parser 214 can also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and lemmatize the words. A lemma is the primary form of a set of words as represented in a dictionary (e.g., "run" is a lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking of compound expressions, such as combining "credit" and "card" into a single expression, "credit_card." The language parser 214 can also identify relationships between words within the utterance 202. For example, in some embodiments, language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., particular nouns) are direct objects, which parts of the utterance are prepositions, etc. The results of the processing performed by language parser 214 form extracted information 205, which, along with utterance 202 itself, is provided as input to MIS 220.

[0074] As indicated above, utterance 202 can include more than one sentence. For purposes of multiple-intent detection and explicit invocation, utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in some embodiments, preprocessing, such as by preprocessing subsystem 210, can be performed to identify single sentences among multiple sentences for multiple-intent analysis and explicit invocation analysis. In general, the results produced by MIS 220 and EIS 230 are substantially the same whether utterance 202 is processed at the individual sentence level or as a single unit containing multiple sentences.

[0075] The MIS 220 determines whether the utterance 202 expresses multiple intents. While the MIS 220 can detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not involve determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the processing to determine whether the intent of the utterance 202 matches a bot intent can be performed by the intent classifier 242 of the MB system 200 or by an intent classifier of a skill bot (e.g., as shown in the embodiment of FIG. 3). The processing performed by the MIS 220 assumes that there is a bot (e.g., a particular skill bot or the master bot itself) that can address the utterance 202. Thus, the processing performed by the MIS 220 does not require knowledge of which bots are in the chatbot system (e.g., the identities of the skill bots registered with the master bot) or which intents are configured for a particular bot.

[0076] To determine that utterance 202 contains multiple intents, MIS 220 applies one or more rules from a set of rules 252 in data store 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include a sentence pattern that indicates the presence of multiple intents. For example, the sentence pattern may include a coordinating conjunction joining two parts of a sentence (e.g., a conjunction), where both parts correspond to separate intents. If utterance 202 matches the sentence pattern, it can be inferred that utterance 202 expresses multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents targeted at different bots or different intents within the same bot). Instead, the utterance may have separate instances of the same intent, such as, for example, "Place a pizza order using payment account X, then place a pizza order using payment account Y."

[0077] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines which portions of utterance 202 are associated with each intent. For each intent representing an utterance that encompasses multiple intents, MIS 220 constructs a new utterance in place of the original utterance for separate processing, such as, for example, utterance “B” 206 and utterance “C” 208 as shown in FIG. 2 . Thus, original utterance 202 can be split into two or more separate utterances that are addressed at one time. MIS 220 determines which of the two or more utterances should be addressed first using extracted information 205 and / or from an analysis of utterance 202 itself. For example, MIS 220 may determine that utterance 202 includes a marker word indicating that a particular intent should be addressed first. The newly formed utterance corresponding to this particular intent (e.g., one of utterance 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (e.g., or has been temporarily put on hold), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) can then be sent to EIS 230 for processing.

[0078] The EIS 230 determines whether an utterance it receives (e.g., utterance 206 or utterance 208) includes a skillbot's invocation name. In one embodiment, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes the skillbot from other skillbots in the chatbot system. A list of invocation names may be maintained in the data store 250 as part of the skillbot information 254. When an utterance includes words that match the invocation name, the utterance is considered to be an explicit invocation. If the bot is not explicitly invoked, the utterance received by the EIS 230 is considered an implicit invocation utterance 234 and is input to the masterbot's intent classifier (e.g., intent classifier 242) to determine which bot to use to address the utterance. In some cases, the intent classifier 242 determines that the masterbot should address the implicit invocation utterance. In other cases, the intent classifier 242 determines which skillbot to route the utterance to for handling.

[0079] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that a masterbot needs to perform. For example, when there is an explicit call, the masterbot may not need to perform any intent classification analysis (e.g., using intent classifier 242) or a reduced intent classification analysis to select a skillbot. Thus, the explicit call analysis can enable the selection of a particular skillbot without using intent classification analysis.

[0080] There may also be situations where there is overlap in functionality among multiple skill bots. This can occur, for example, when the intents addressed by two skill bots overlap or are very close to each other. In such situations, it may be difficult for the master bot to identify which of the multiple skill bots to select based solely on intent classification analysis. In such scenarios, explicit invocation eliminates ambiguity about the specific skill bot to be used.

[0081] In addition to determining that an utterance is an explicit invocation, EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the skill bot being explicitly invoked. In particular, EIS 230 can determine that a portion of the utterance is not associated with an invocation. EIS 230 can make this determination through analysis of the utterance and / or analysis of extracted information 205. EIS 230 can send the portion of the utterance that is not associated with an invocation to the skill bot being invoked instead of sending the entire sentence received by EIS 230. In some cases, the input to the skill bot being invoked is formed simply by removing any portion of the utterance that is associated with an invocation. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using Pizza Bot" is related to the invocation of Pizza Bot but not to any processing that will be performed by Pizza Bot. In some cases, the EIS 230 can reformat the portion to be sent to the bot being invoked, for example, to form a complete sentence. Thus, the EIS 230 determines not only that there is an explicit invocation, but also what to send to the skill bot when there is an explicit invocation. In some cases, there may not be any text to be entered into the bot being invoked. For example, if the utterance was "Pizza Bot," the EIS 230 may determine that the Pizza Bot is being invoked, but there is no text to be processed by the Pizza Bot. In such a scenario, the EIS 230 can instruct the skill bot invoker 240 that there is nothing to send.

[0082] The skillbot invoker 240 invokes a skillbot in various manners. For example, the skillbot invoker 240 can invoke a bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invocation. The indication 235 can be sent by the EIS 230 along with input for the skillbot being explicitly invoked. In this scenario, the skillbot invoker 240 hands over control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response can be to perform a particular action or to start a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0083] Another manner in which the skillbot invoker 240 can invoke a skillbot is through implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 is trained for multiple different classes, one class for each skillbot. For example, whenever a new skillbot is registered with the masterbot, a list of example utterances associated with the new skillbot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. Parameters (e.g., a set of values ​​for the parameters of a machine learning model) produced as a result of this training can be stored as part of the skillbot information 254.

[0084] In one embodiment, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training the machine learning model may involve inputting at least a subset of utterances from example utterances associated with various skill bots and generating, as the output of the machine learning model, a guess as to which bot is the correct bot to address any particular training utterance. For each training utterance, an indication of the correct bot to use for the training utterance may be provided as ground truth information. The behavior of the machine learning model may then be adapted (e.g., through backpropagation) to minimize the difference between the guesses generated and the ground truth information.

[0085] In one embodiment, the intent classifier 242 determines a confidence score for each skill bot registered with the master bot, indicating the skill bot's likelihood of handling the utterance (e.g., an implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skill bot invoker 240 invokes the bot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either the identity of the system intent or the identity of a particular skill bot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain win margin. Imposing such a condition enables routing to a particular skill bot when the confidence scores of multiple skill bots each exceed the threshold confidence score value.

[0086] After a bot is identified based on the evaluation of the confidence score, the skillbot invoker 240 hands over processing to the identified bot. In the case of system intent, the identified bot is a masterbot. Otherwise, the identified bot is a skillbot. Furthermore, the skillbot invoker 240 determines what to provide as input 247 to the identified bot. As indicated above, in the case of explicit invocation, the input 247 can be based on a portion of the utterance that is not associated with the invocation, or the input 247 can be none (e.g., an empty string). In the case of implicit invocation, the input 247 can be the entire utterance.

[0087] The data store 250 includes one or more computing devices that store data used by various subsystems of the masterbot system 200. As described above, the data store 250 includes rules 252 and skillbot information 254. The rules 252 include, for example, rules for determining, by the MIS 220, when an utterance expresses multiple intents and how to split an utterance expressing multiple intents. The rules 252 further include rules for determining, by the EIS 230, which portions of an utterance that explicitly invokes a skillbot to send to the skillbot. The skillbot information 254 includes, for example, invocation names of skillbots in the chatbot system, such as a list of invocation names of all skillbots registered with a particular masterbot. The skillbot information 254 can also include information used by the intent classifier 242 to determine a confidence score for each skillbot in the chatbot system, such as, for example, parameters of a machine learning model.

[0088] 3 is a simplified block diagram of a Skillbot system 300 according to one embodiment. Skillbot system 300 is a computing system that can run solely in software, solely in hardware, or a combination of hardware and software. In one embodiment, such as the embodiment shown in FIG. 1, Skillbot system 300 can be used to run one or more Skillbots within a digital assistant.

[0089] Skillbot system 300 includes MIS 310, intent classifier 320, and conversation manager 330. MIS 310 is similar to MIS 220 of FIG. 2 and provides similar functionality, including being operable to use rules 352 in data store 350 to (1) determine whether an utterance expresses multiple intents, and if so, (2) determine how to split the utterance into separate utterances for each of the multiple intents. In one embodiment, the rules applied by MIS 310 to detect multiple intents and split the utterance are the same as those applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. Extracted information 304 is similar to extracted information 205 of FIG. 1 and can be generated using language parser 214 or a language parser local to Skillbot system 300.

[0090] The intent classifier 320 can be trained similarly to the intent classifier 242 described above in connection with the embodiment of FIG. 2 and as described in more detail herein. For example, in one embodiment, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of example utterances associated with that particular skill bot as training utterances. The ground truth for each training utterance is the particular bot intent associated with the training utterance.

[0091] The utterance 302 can be received directly from a user or can be provided through a masterbot. For example, if the utterance 302 is provided through a masterbot as a result of processing through the MIS 220 and the EIS 230 in the embodiment shown in FIG. 2 , the MIS 310 can be bypassed to avoid repeating processing already performed by the MIS 220. On the other hand, if the utterance 302 is received directly from a user, for example, during a conversation that occurs after routing to a skillbot, the MIS 310 can process the utterance 302 to determine whether the utterance 302 represents multiple intents. If so, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, such as, for example, utterance “D” 306 and utterance “E” 308. If the utterance 302 does not represent multiple intents, the MIS 310 forwards the utterance 302 to the intent classifier 320 for intent classification without splitting the utterance 302.

[0092] The intent classifier 320 is configured to match an incoming utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As explained above, a skillbot can be configured with one or more intents, each including at least one example utterance associated with the intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the masterbot system 200 is trained to determine a confidence score for each individual skillbot and a confidence score for the system intent. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skillbot system 300. While the classification performed by the intent classifier 242 is at the bot level, the classification performed by the intent classifier 320 is at the intent level and is therefore more granular. The intent classifier 320 can have access to intent information 354. For each intent associated with the skillbot system 300, the intent information 354 includes a list of utterances that represent the intent, indicate its meaning, and are typically associated with tasks that can be performed by the intent. The intent information 354 can further include parameters that result from training on this list of utterances.

[0093] The conversation manager 330 receives as an output of the intent classifier 320 an indication 322 of the particular intent identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some cases, the intent classifier 320 is unable to determine any match. For example, if the utterance is targeted to the system intent or the intent of a different skill bot, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value. When this occurs, the skill bot system 300 may refer the utterance to the master bot for action, for example, to route it to a different skill bot. However, if the intent classifier 320 is successful in identifying the intent within the skill bot, the conversation manager 330 begins a conversation with the user.

[0094] The conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute the dialog flow of the identified intent. The state machine may include a default starting state (e.g., the intent is invoked without any additional input) and one or more additional states, each associated with an action to be performed by the skill bot (e.g., completing a purchase transaction) and / or a dialog to be presented to the user (e.g., question, response). Thus, the conversation manager 330 can determine an action / dialog 335 upon receiving an instruction 322 identifying the intent, and can determine the additional action or dialog in response to subsequent utterances received during the conversation.

[0095] Data store 350 includes one or more computing devices that store data used by various subsystems of skillbot system 300. As shown in Figure 3, data store 350 includes rules 352 and intent information 354. In an embodiment, data store 350 can be integrated with a masterbot or digital assistant data store, such as, for example, data store 250 of Figure 2.

[0096] Trust Score Calibration Confidence calibration is defined as the ability of a model to provide an accurate probability of accuracy for any of its predictions. For example,

[0097]

number

[0098] is the model prediction output (in OMRL logical form in the case of C2OMRL),

[0099]

number

[0100] Assume that ,is the associated confidence score.

[0101]

number

[0102] We expect that σ represents the true probability of the model's accuracy. In other words, if a model makes 100 predictions with the same confidence level of 0.9, we would expect 90 predictions to be accurate (out of 100). A poorly calibrated model is "under-confident" if its accuracy exceeds its confidence level, or "over-confident" if its accuracy is less than its confidence level. In this case, the model's confidence score is well-calibrated. While such calibrated confidence scores have traditionally been important in various "high-stakes" applications (e.g., self-driving cars, medical diagnostics, etc.) where inaccurate predictions are highly problematic, calibrated probability scores associated with each prediction are also becoming important for other applications, such as data analysis and customer satisfaction, because they allow low-quality predictions to be identified and discarded. Thus, even if machine learning model outputs cannot yet be fully explained, confidence calibration provides a practical means to avoid errors in practice by associating each prediction with an accurate uncertainty / confidence score.

[0103] Confidence calibration is a useful property, but it is not easily measured relative to specific performance metrics such as accuracy or loss. Therefore, various metrics for confidence calibration have been proposed over time, each with its own advantages and disadvantages. For example, the Brier score is a suitable scoring rule that measures the squared error between the predicted probability vector and the one-hot encoded true label. Lower scores correspond to more accurate calibration. The Brier score is a good metric for measuring the calibration of a network's predictions, but is less sensitive to probabilities associated with rare events. Therefore, measuring calibration using multiple metrics in addition to the Brier score often provides useful insights. Negative log-likelihood is also a suitable scoring rule that can be used to assess model uncertainty on held-out data, with lower scores corresponding to better calibration. Typically used as an objective function for training, the negative log-likelihood characterizes the difference between predicted and actual confidence in the true label, reaching a perfect score of zero when all data is accurately predicted with 100% confidence. However, negative log-likelihoods overemphasize tail probabilities, i.e., even small deviations from accurate and perfect confidence predictions can result in large increases in the negative log-likelihoods.

[0104] The expected calibration error (ECE) calculates the expected difference between confidence and accuracy. Such expected difference can be calculated as shown below:

[0105]

number

[0106] The above equation cannot be calculated in closed form, but can be approximated by dividing the model predictions into distinct bins based on their associated confidence scores. The difference between the average confidence and accuracy can then be calculated within each bin. The ECE then calculates the difference between the average confidence and accuracy within each bin and takes a weighted average of these values ​​based on the relative size of each bin.

[0107] The maximum calibration error (MCE) follows this same process, but is equal to the maximum difference between the average confidence and accuracy across the bins. For both ECE and MCE, lower scores correspond to better-calibrated predictions. Neither ECE nor MCE is a suitable scoring rule; there is a trivial solution (e.g., uniform random probability prediction) with optimal calibration error. Additionally, due to the binning procedure, neither of these metrics monotonically decreases as prediction improves. Nevertheless, these metrics may be used due to their ability to provide simple and interpretable estimates of model calibration. MCE is commonly used in applications where a reliable confidence measure is absolutely necessary (i.e., large errors in calibration are harmful), while ECE provides a more holistic averaged metric across bins.

[0108] If the user is interested in obtaining a plot that characterizes the calibration error instead of a scalar metric like ECE or MCE, the accuracy and confidence measures per bin can be converted into a confidence diagram. A reality diagram depicts accuracy on the y-axis and average confidence on the x-axis. The average confidence and accuracy measures within each bin are plotted on the diagram to form a line plot. A perfect calibration would result in a diagonal line on the confidence diagram, where confidence equals accuracy within each individual bin. For example, perfect calibration would mean:

[0109]

number

[0110] The main idea of ​​the confidence diagram is to compare the observed model accuracy, i.e.

[0111]

number

[0112] The goal is to plot the predictions against the expected confidence P. For a finite sample, this is achieved by grouping the predictions into M bins of equal size 1 / M.

[0113]

number

[0114] Bin B of predictions with confidence falling into M Assuming that, the observation accuracy is calculated as follows:

[0115]

number

[0116] On the other hand, the expected reliability is calculated as follows:

[0117]

number

[0118] If the model is well calibrated, its observed accuracy is equal to its expected confidence, i.e., for all bins B M Regarding, it becomes as follows:

[0119] acc(B M )=conf(B M ) 4A and 4B show exemplary confidence diagrams for the calibration of a C2OMRL model according to the calibration techniques disclosed herein. The calibration results were measured in terms of ECE (average of the gap bars in the reality diagram). The desired calibration goal was an ECE<0.1. In FIG. 4A, ECE=0.29, and therefore the C2OMRL model did not meet the calibration goal. In FIG. 4B, ECE=0.05, and therefore the C2OMRL model met the calibration goal.

[0120] x-axis is bin location B M Represents. Black bar: Observation accuracy acc(B M ) are plotted at the bin locations.

[0121] Gray bar: Expected reliability conf(B M ) are plotted at the bin locations.

[0122] The top of the grey bar follows the identity function (dashed diagonal line). Therefore, the gap bar (i.e., the revealed grey box) is the difference between the expected confidence and the observed accuracy.

[0123] The confidence diagram in Figure 4A shows a poorly calibrated model, with gap bars both small and large across the confidence values. This model is an overestimation.

[0124] The confidence diagram in Figure 4B shows that the gap bar is relatively small, i.e., acc(B M )≒conf(B M ), which indicates an overall well-calibrated model.

[0125] In both Figures 4A and 4B, the black bars obscure the gray bars, i.e., the observation accuracy acc(B M )>Expected reliability conf(B M ) There are cases where this model is an underestimation.

[0126] While confidence diagrams are useful for visualizing how well a model's confidence scores are calibrated, scalar metrics such as ECE or MCE are also important for quantitatively comparing different models and calibration methods. Calibration scalar metrics allow confidence diagrams to be collapsed into a single number for comparison. ECE can be defined as:

[0127]

number

[0128] In the formula, B M is the bin location, and acc(B M ) is the observation accuracy, and conf(B M ) is the expected confidence and n is the total number of predictions. Therefore, the ECE can be interpreted as the average of the gap bars in the confidence diagram.

[0129] Several conventional methodologies have been proposed to improve the confidence calibration of deep learning models (e.g., neural networks) (e.g., temperature scaling, Pratt scaling, etc.). While some of these methodologies are simple processes that can be added to the training process, others require a post-processing step that utilizes a holdout validation set or even significant modifications to the network architecture. However, existing calibration techniques are only suitable for classification models and therefore cannot be applied to NL2LF. For example, Pratt scaling is the preferred calibration method for binary classifiers, while temperature scaling is the standard technique for calibrating multiclass classifiers. This is because these methodologies derive confidence scores directly from the final "activation layer" of the classification model, such as "Softmax" for multiclass classification and "Sigmoid" for binary classification.

[0130] Calibration of deep learning-based semantic parsers (NL2LF models fall into this category) is largely unexplored. The only relevant work on deep learning-based semantic parsers is described in Dong, Li et al., "Confidence Modeling for Neural Semantic Parsing," Annual Meeting of the Association for Computational Linguistics (2018). This paper explores confidence by identifying the most "uncertain" parts of the model-generated output and then "backtracking" them to specific parts of the model input. However, the described approach is specific to a particular encoder-decoder architecture (LSTM, where the encoder and decoder are two different recurrent neural networks with long- and short-term memory units) and therefore cannot be generally applied to most NL2LF models, such as the C2OMRL model described herein. To address these and other deficiencies, this paper describes a calibration technique that leverages intermediate "raw" scores produced at each step of the NL2LF model's decoding process and combines them to obtain a calibrated confidence score.

[0131] An overview of systems for converting natural language into semantic expression language. FIG. 5 illustrates a C2OMRL system 500 equipped with a machine learning model to enable the conversion of natural language (NL) utterances (e.g., utterances within a digital assistant platform such as those described in connection with FIGS. 1-3) into logical form (LF) descriptions, such as OMRL queries or commands, that can then be executed to query existing systems, such as relational databases. This machine learning model (referred to herein as a “C2OMRL semantic parser” or “C2OMRL model”) is trained with hundreds to thousands of annotated example pairs (natural language-logical form pairs) to convert natural language utterances into logical form descriptions. As shown, an example 505 (a concatenation of a natural language utterance with a database schema, e.g., a set of table and column names) is input into the C2OMRL model 510. The example 505 is first processed by an encoder 515, which contextually captures a representation of the natural language utterance and the database schema. A decoder 520 then receives the encoded input and predicts a logical form 525 (eg, OMRL, which is an SQL-like query) based on the captured representation of the natural language utterance and the database schema.

[0132] In the C2OMRL model 510, the encoder 515 includes two main components: (1) a pre-trained language model (PLM) 530 and (2) a relation-aware transformer (RAT) 535. The PLM 530 is used to embed natural language utterances and database schemas while contextually capturing their representations. In one example, a transformer-based PLM called Decoding Enhanced BERT with Separated Attention (DeBERTa) is used as the PLM 530. Transformer-based PLMs use self-supervised learning to learn universal language representations from large amounts of text data and transfer this knowledge to downstream tasks. These models provide downstream tasks with good background knowledge, avoiding the need to train downstream models from scratch. The RAT 535 encodes associations between entities in the database schema and words in the natural language utterance (these associations are called "schema-linked" associations).

[0133] The decoder 520 is based on a bottom-up generation process (i.e., the bottom-up generation process generates a tree from left to right), and the final generation output is an OMRL tree (i.e., a tree-based structure representing the entire OMRL logical form) that can be mapped to the final OMRL logical form 525. The bottom-up generation process is performed using beam search, a heuristic search algorithm that explores a graph by expanding the most promising nodes in a restricted set. As shown in FIG. 6, the beam search 600 operates in steps (e.g., approximately 10 steps), also referred to as "beam levels." At each step (e.g., "step i"), the beam search algorithm generates F possible subtrees for the input sequence that can be obtained by expanding the current subtree (from step "i-1"), and then selects the top K subtrees to retain (known as the beamwidth) using the conditional probability associated with each subtree. The conditional probabilities are referred to herein as "raw beam scores," and thus the top K intermediate results (considered in the next generation step) are the K intermediate results with the highest raw beam scores. Additional information for the bottom-up generation process can be found in "Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressive Bottom-up Semantic Parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311-324, Online. Association for Computational Linguistics." The entire contents of which are incorporated herein by reference for all purposes. The final decoder 520 output is the subtree with the highest raw beam score in the last step N.

[0134] For example, in the first step (beam level 1), the encoded input utterance and database schema are input to the decoder 520, which applies a softmax function to all tokens in the vocabulary or grammar to find the best choice for the first subtree (e.g., the first token or node in the tree). To generate F possible subtrees (known as the frontier), the decoder 520 makes predictions that represent the conditional probability of each token in the vocabulary or grammar that comes next in the sequence (finding the most likely value of yi+1 given the previous tokens y1, ..., yi and the context variable c created by the encoder to represent the input sequence). The vocabulary or grammar is obtained from a corpus containing words or terms in a target logical form (e.g., OMRL). In some cases, the corpus further includes rules for words or terms in the target logical form. The rules define how words or terms are used to create appropriate phrases or actions in the target logical form (e.g., combinations of terms that work together for appropriate OMRL queries). The beam search algorithm then selects the top K subtrees with the highest conditional probabilities or raw beam scores as the most likely choices for that time step. In this example, the top K subtrees or beam widths are 2, and the subtrees with the highest conditional probabilities P(y1|c) in the first step are subtree_1 and subtree_12. The top K results may be selectable and / or optimizable hyperparameters. Subtree_1 and subtree_12 and the corresponding conditional probabilities or raw beam scores are saved in memory.

[0135] In the second step (beam level 2), the two selected trees (sub-tree_1 and sub-tree_12) from the first step are input to decoder 520, which applies a softmax function to all tokens in the vocabulary or grammar to find the two best choices for the second subtree (e.g., the first and second tokens or nodes of the tree). In doing so, the beam search algorithm uses conditional probabilities to determine the most likely combination of first and second tokens or nodes to form a pair or second subtrees. In other words, for every y2∈Y, the beam search algorithm calculates P(sub-tree_1,y2|c)=P(sub-tree_1|c)P(y2|sub-tree_1,c), P(sub-tree_12,y2|c)=P(sub-tree_12|c)P(y2|sub-tree_12,c), and selects the two largest among these values, e.g., P(sub-tree_22|c) and P(sub-tree_37|c). Sub-tree_22 and sub-tree_37 and the corresponding conditional probabilities or raw beam scores are saved in memory.

[0136] In the third step (beam level 3), the two selected trees (sub-tree_22 and sub-tree_37) from the second step are input to decoder 520, which applies a softmax function to all tokens in the vocabulary or grammar to find the two best choices for the third subtree (e.g., the first, second, and third tokens or nodes of the tree). While doing so, the beam search algorithm uses conditional probabilities to determine the combination of the first, second, and third tokens or nodes that are most likely to form a string or third subtree. In other words, for all y3∈Y, the beam search algorithm calculates P(sub-tree_22,y3|c)=P(sub-tree_22|c)P(y3|sub-tree_22,c), P(sub-tree_37,y3|c)=P(sub-tree_37|c)P(y3|sub-tree_37,c) and selects the top K subtrees. The top K subtrees and corresponding conditional probabilities or raw beam scores are saved in memory. This process continues until N beam levels (which may be optimized or selected hyperparameters) have been completed. The final model output is the subtree with the highest conditional probability or raw beam score at the last step N (beam level N). The tokens or nodes of this final subtree can then be mapped to a final logical form, such as an OMRL logical form description 525.

[0137] The predicted OMRL logical form description 525 (i.e., the OMRL tree with the highest raw beam score in the final step N) can then be input to a language converter 540, such as (OMRL2SQL), to convert the semantic representation language into a system language query or command, such as SQL, API, REST, GraphQL, PGQL, etc. The system language query or command can then be used to query or perform operations on a system 545 (e.g., a relational database) and obtain an output 550 as a result of the query or command. While the system and model for natural language-semantic representation language are described herein with respect to C2OMRL, it should be understood that the techniques described herein are applicable to any system and / or model for converting natural language into a semantic representation language or logical form that utilizes a bottom-up generative process for a decoder (i.e., any NL2LF model, such as NL2SQL).

[0138] Trust score calibration system and technique The confidence score calibration system and technique of this disclosure utilizes a calibration model, i.e., an auxiliary model to that of the C2OMRL model, that takes as input raw beam scores from one or more beam levels of the decoder of the C2OMRL model and outputs confidence scores associated with the C2OMRL output. In various embodiments, the calibration model is a binary classifier, such as a logistic regression binary classifier, that takes as input a vector having one or more raw beam scores (e.g., the top K raw beam scores) from one or more beam levels, and the calibration model uses the information in the vector to predict whether the final result (i.e., the OMRL tree with the highest raw beam score in the final step N) will be accurate or inaccurate. The probability score from the calibration model (e.g., the output of a sigmoid function in logistic regression) becomes the calibrated C2OMRL confidence score. The theory behind this approach is that the raw beam scores of subtree candidates can reveal information about model uncertainty in the generation process (e.g., due to model vulnerability to some types of examples or caused by ambiguous inputs), and therefore these raw scores are a good source of information for the calibrated confidence score.

[0139] FIG. 7 illustrates a confidence score calibration system 700 that utilizes a calibration model 705. Inputs 710 to the calibration model 705 are raw beam scores for one or more beam levels obtained from a decoder 715 of an NL2LF machine learning model (e.g., the C2OMRL model 510 described in connection with FIGS. 5 and 6). The raw beam scores for one or more beam levels are preprocessed to generate a vector of raw beam scores for one or more beam levels. The vector comprises a stack or sequential organization of raw beam scores. In some cases, the order of the raw beam scores from the beam levels is maintained in the vector (i.e., raw beam scores from the first beam level come first in the vector, followed by raw beam scores from the second beam level, followed by raw beam scores from the third beam level, and so on). In some cases, raw beam scores or a subset of raw beam scores from all beam levels are vectorized. In some cases, the top K raw beam scores from all beam levels are vectorized. In other cases, a subset of the top K raw beam scores from all beam levels is vectorized. In some cases, raw beam scores or a subset of raw beam scores from a subset of beam levels are vectorized (e.g., the last N beam levels or the last three beam levels). In some cases, the top K raw beam scores from a subset of beam levels are vectorized. In other cases, a subset of the top K raw beam scores from a subset of beam levels is vectorized. For example, using only the top K raw beam scores from a subset of beam levels can improve training efficiency and processing time (fewer features are available for classification), while using more raw beam scores from more beam levels, such as all beam levels, can improve the accuracy of the calibration model (more features are available for classification).

[0140] The calibration model 705 is trained as a binary classifier that takes in the input 710 and uses the information in the vector to predict whether the final result (i.e., the OMRL tree with the highest raw beam score in the last step N) will be accurate or incorrect. The class with an accurate state can be assigned class label 0, and the class with an incorrect state can be assigned class label 1. The algorithm trained for the calibration model 705 can be an algorithm typically used for binary classification problems, such as logistic regression, K-nearest neighbors, decision tree, support vector machine, or naive Bayes. In one example, the algorithm trained for the calibration model 705 is logistic regression. Logistic regression is characterized by a logistic function to model the conditional probability of a variable X given a label Y. In the calibration model 705, Y captures the accurate or incorrect state, and X is a feature related to the model uncertainty in the generation process (i.e., the raw beam score). Then, the conditional probability that the output of a machine learning model trained to convert natural language into logical form is accurate or inaccurate given an observation X can be modeled as a sigmoid function. The sigmoid function maps a continuous variable to a closed set [0,1], which can then be interpreted as a probability. All data points to the right of the cutoff are interpreted as y=1, and all data points to the left of the cutoff are interpreted as y=0.

[0141] Training the calibration model 705 may be performed before, after, or simultaneously with training the NL2LF machine learning model to convert natural language into a logical form. In any case, training the calibration model 705 does not change any of the weights / parameters of the NL2LF machine learning model. In other words, the calibration process does not affect the predicted LF (e.g., OMRL) output of the NL2LF machine learning model. To avoid overfitting the calibration model 705, the calibration model 705 may be trained using a set of training examples that is different from the set of training examples used to train the NL2LF machine learning model. The set of training examples used to train the calibration model 705 is annotated with ground truth labels regarding whether the NL2LF machine learning model accurately or inaccurately predicted the LF given a NL utterance. The labels may be generated by comparing the LF output from the NL2LF machine learning model to the LF ground truth established for the utterance and determining whether the LF output is the same or substantially the same as the LF ground truth established for the utterance.

[0142] The calibration model 705 produces two outputs 720: (i) a binary classification representing whether the subtree with the highest conditional probability or raw beam score in the last step N is correct or incorrect, and (ii) a conditional probability score regarding whether the subtree with the highest conditional probability or raw beam score in the last step N is correct or incorrect. In one example, the conditional probability score is the output of a sigmoid function. The conditional probability score is used as a calibrated confidence score for the NL2LF machine learning model.

[0143] FIG. 8 shows a block diagram illustrating an embodiment of a model system 800 configured to train and deploy machine learning models (e.g., the NL2LF machine learning models and calibration models described in connection with FIGS. 5-7 ). The model system 800 in this example includes various stages: a training stage 805 for training machine learning models; an NL2LF conversion stage 810 for converting natural language text (e.g., an utterance) into a logical form such as OMRL; a confidence score calibration stage 815 for obtaining confidence scores associated with the logical forms output from the NL2LF conversion stage 810; and a query execution stage 820 for executing the logical forms on a system such as a database to obtain results (e.g., answers to queries in the utterance). The training stage 805 builds and trains one or more machine learning models 425 a- 425 n (where “n” represents any natural number) (sometimes referred to herein individually as models 825 or collectively as models 825) that are used by the other stages. For example, the models 825 may include a first model for converting a natural language utterance into a logical form and a second model for predicting whether the logical form output from the first model is accurate or inaccurate. Still other types of predictive models (e.g., intent classifiers) may be implemented in other examples in accordance with this disclosure.

[0144] Model 825 may be a deep learning model, such as a convolutional neural network ("CNN"), e.g., an Inception Neural Network, a Residual Neural Network ("Resnet"), or a recurrent neural network, e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or other variants of a deep neural network ("DNN") (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier). Model 825 may also be a logistic regression classifier, a naive Bayes classifier, a linear classifier, a support vector machine, a random forest model, etc. The model system 800 may utilize the same or different types of models for various tasks, such as converting natural language into a logical form and / or binary classification.

[0145] In some embodiments, model 825 is configured as a semantic parser or NL2LF model, such as RAT-SQL, DuoRAT, Grammar-Augmented Pre-Training Model for Table Semantic Parsing (GRAPPA), RAT-SQL+GRAPPA, RoBERTa, Semi-Autoregressive Bottom-Up Semantic Parsing Model (SMBOP), etc. Additional information on the RAT-SQL model can be found in "RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers," Wang et al., published in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, the contents of which are incorporated herein by reference in their entirety for all purposes. Additional information for the DuoRAT model can be found in "DuoRAT: Towards Simpler Text-to-SQL Models," Scholak et al., Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, the entire contents of which are incorporated herein by reference for all purposes. Additional information for the SMBOP model can be found in Ohad Rubin and Jonathan Berant. 2021. "SmBoP: Semi-autoregressive Bottom-up Semantic Parsing." In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311-324, Online. Association for Computational Linguistics, the entire contents of which are incorporated herein by reference for all purposes.

[0146] To train various models 825, the training stage 805 consists of two main subsystems or services: a dataset preparer 830 and a model trainer 840. The dataset preparer 830 facilitates the process of loading data assets 845, splitting the data assets 845 into training and validation sets (845-a-n) so that the system can train and test the models 825, and performing basic natural language preprocessing (e.g., standardization, normalization, data tokenization, annotation, augmentation, materialization, etc.). The data assets 845 include natural language utterances (e.g., natural language questions / requests) obtained from one or more sources, such as human annotators (e.g., Spider, SparC, and / or CoSQL datasets), databases (not shown), computing systems (e.g., data preprocessing subsystems), etc. In some cases, the utterances are provided by a client or customer. In other cases, the utterances are generated automatically and / or retrieved from a library of utterances (e.g., identifying utterances from the library that are specific to the task for which the model is to learn). Data assets 845 can include input text or acoustics (or input features of text or acoustic frames) and labels 850 corresponding to the input text or acoustics (or input features) as a matrix or table of values. For example, for each utterance, the corresponding label may include an indication of the corresponding logical form (e.g., a description / query, such as an OMRL query), which serves as ground truth information for the utterance, and / or an indication of whether the logical form is accurate or inaccurate for the utterance. The model's behavior can then be adapted (e.g., through backpropagation) to minimize the difference between the generated guesses and the ground truth information.For example, data asset 845 may include the natural language utterance "What is the average life expectancy in the United States of America?" and a corresponding logical form "SELECT AVG(life_expectancy) FROM country = United States of America" ​​(and possibly a label containing whether the predicted logical form was "accurate" or "inaccurate").

[0147] In some cases, data assets 845 also include database schema information. A database schema defines how data is organized in a database, such as a relational database, and includes logical constraints such as table names, fields, data types, and relationships between these entities. A relational database can be formed from one or more tables, each of which includes one or more columns, and each of which includes one or more values. Each table and column in a relational database can be named by a unique identifier, each of which can include one or more words. In some cases, one or more columns in a relational database can serve as a primary key, and each of the values ​​in the one or more columns that serve as a primary key are unique to one another. In some cases, one or more columns in a relational database can serve as a foreign key, which serves to link a table containing one or more columns with another table in the relational database. In some cases, the database schema information includes a unique identifier for one or more tables, a unique identifier for one or more columns, and one or more data structures for storing each relational database value. The unique identifiers and values ​​may be stored in one or more vectors and / or matrices by the dataset preparer 830. In some embodiments, the data structure that stores schema information for the relational database may store a directed graph that represents the unique identifiers and values.

[0148] In some cases, the data assets 845 also include beam score information. The beam score information for one or more beam levels can be obtained from a decoder of a trained NL2LF machine learning model (e.g., the C2OMRL model 510 described in connection with FIGS. 5 and 6). The raw beam scores for one or more beam levels are preprocessed by the dataset preparer 830 to generate a vector of raw beam scores for one or more beam levels. The vector includes a stack or sequence of raw beam scores. The beam score information can be annotated with a ground truth label regarding whether the trained NL2LF machine learning model correctly or incorrectly predicted the LF given a NL utterance. The label can be generated by comparing the LF output from the trained NL2LF machine learning model to the LF ground truth established for the utterance and determining whether the LF output is the same or substantially the same as the LF ground truth established for the utterance.

[0149] Once the data asset 845 is obtained, the dataset can be split into a training dataset and a validation dataset. The splitting may be performed randomly (e.g., 90 / 10% or 70 / 30%), or the splitting may be performed according to more complex validation techniques, such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out group cross-validation, nested cross-validation, etc., to minimize sampling bias and overfitting. Before or after the splitting, basic natural language preprocessing may be performed on the data asset 845. In some cases, the preprocessing includes tokenizing the utterances of the data asset 845. Tokenization is the division of phrases, sentences, paragraphs, or entire passages into smaller units, such as individual words or terms. Each of these smaller units is called a token. The smaller units are created by locating boundaries, such as word boundaries, where one word ends and the next word begins. For example, the text "How many employees work for company X" can be tokenized into the words "How," "many," "employees," "work," "for," "company," and "X." These tokens help the model understand the context and develop a model for a given task. There are various tokenization techniques that can be used to perform tokenization based on the language and modeling task. For example, tokenization may be performed using a natural language toolkit, whitespace tokenization, dictionary-based tokenization, rule-based tokenization, Keras tokenization, Penn Tree-based tokenization, spaCy tokenization, Moses tokenization, subword tokenization, etc.

[0150] In some cases, tokens of data assets 845 may then be embedded into word embeddings (e.g., contextualized word embeddings). Word embeddings are learned representations of text in which words with the same meaning have similar representations. Word embeddings are generated by embedding techniques in which individual words in a predefined vector space are represented as real-valued vectors so that they can be understood by deep learning algorithms. The embedding technique can be a combined or separate embedding technique, such as including an embedding layer within a deep learning algorithm or using a separate model such as a BERT-based pre-trained language model (e.g., BERT, RoBERTa, DeDERTa). A BERT-based model is a pre-trained language model that learns the deep meaning of words and context using self-supervised learning. The embedding layer is a word embedding that is trained together with a neural network model for a specific natural language processing task, such as natural language-logical form transfer (e.g., natural language-logical form (NL-LF) algorithm). In some cases, other embedding techniques such as Word2Vec or GloVe can be used. Word2Vec is a statistical technique that uses models such as continuous bag-of-words or continuous skip-gram models to learn word embeddings independent of a text corpus. GloVe, short for global vector, is a model for creating word embeddings based on global corpus statistics. It is trained on the non-zero entities of a global word co-occurrence matrix, which tallies how frequently words co-occur with each other in a given corpus.

[0151] Model training generally involves selecting hyperparameters for the model 825 and using an optimization algorithm (e.g., a stochastic gradient descent algorithm or its variants, such as batch gradient descent or mini-batch gradient descent) to find the model parameters that correspond to the best fit between predicted and actual outputs. Hyperparameters are settings that can be tuned or optimized to control the behavior of the model 825. Most models explicitly define hyperparameters that control various aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined and optimized to adapt the model to specific scenarios. For example, hyperparameters may include the number of hidden units of the model, the learning rate of the model, the convolution kernel width, the number of kernels in the model, the top-K results, the number of beam levels N, etc.

[0152] During training by the model trainer 840, an error is calculated as the difference between the actual output (e.g., as dictated by the labels 850) and the predicted output (e.g., as determined by the model 825). The function used to calculate this error is known as the objective function (e.g., a loss function or cost function). The error is a function of the model's internal parameters, such as weights and biases. For accurate predictions, the error must be minimized. To minimize the error, the model parameters are incrementally updated by minimizing the objective function over training examples from the preprocessed data assets 845. The objective function can be constructed to measure the difference between the output predicted using the model and the ground truth, which is annotated to the samples using labels. For example, for a supervised learning-based model, the training objective is to learn a function “h( )” (sometimes referred to as an assumption function) that maps the training input space X to the target value space Y such that h(x) can successfully predict the corresponding value of y (h:X → Y). A variety of different techniques may be used to learn this assumption function. In some machine learning algorithms, such as neural networks, this is done using backpropagation. The current error is typically propagated backward to the previous layer, where it is used to modify the weights and bias values ​​so that the error is minimized. The weights are modified using an optimization function. The optimization function usually calculates the error gradient, i.e., the partial derivative of the objective function with respect to the weights, and the weights are modified in the opposite direction of the calculated error gradient. For example, techniques such as backpropagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), Hebbian learning, etc. are used to update the model parameters to minimize or maximize this objective function. This cycle is repeated until a minimum of the objective function is reached.

[0153] In some cases, the natural language-to-logical form (NL2LF) algorithm is trained by a model trainer 840 using preprocessed data assets 845 (e.g., tokenized data assets). In some cases, the NL2LF algorithm includes an encoder-decoder neural network. The encoder consists of an input layer and one or more encoding layers. The one or more encoding layers may include multiple recurrent units, such as long-short-term memories (LSTMs), each of which receives input in the form of a single element of an input sequence, collects data for that particular element, and generates and transmits it. The encoder converts the associated text (and optionally a database schema) into a number / vector representation according to an embedding procedure, preserving terms and connections between words and sentences, so that a machine can grasp patterns associated with any text, decipher the context of sentences, and, optionally, learn the relationship between words and a given database schema. The result of the encoder is a state vector or context vector. This state vector becomes the input of the decoder. The decoder consists of an input layer, one or more decoding layers, a dense layer, and an output layer (e.g., a layer with a softmax function). The one or more decoding layers may include multiple recurrent units, such as LSTMs, where the output of every time step is predicted, as described in connection with Figures 5 and 6. The current recurrent unit receives a hidden state from the previous recurrent unit. The decoder result is a logical form, such as an OMRL query transformed from the utterance in the preprocessed data asset 845.

[0154] In some cases, a binary classification algorithm is trained by a model trainer 840 using preprocessed data assets 845 (e.g., vectorized data assets). In some cases, the binary classification algorithm includes logistic regression. The logic behind a logistic regression algorithm is that it finds relationships and probabilities of certain outcomes. In some cases, a binary classifier is trained to predict whether a logical form output by an NL2LF model is correct or incorrect. The data assets 845 include features on the model uncertainty (i.e., raw beam scores) in the generation process of the NL2LF model, and the target / response variable has two values: correct or incorrect / 1 or 0. This type of task is called binary logistic regression because the response / target variable holds only two values: 0 and 1 or true / false. Inputs are fed to the binary classification algorithm, which returns either 1 or 0. As discussed with respect to Figure 7, the results of the logistic regression produce two outputs: (i) a binary classification (i.e., the target / response variable) representing whether the subtree with the highest conditional probability or raw beam score in the final step N is accurate or incorrect, and (ii) a conditional probability score regarding whether the subtree with the highest conditional probability or raw beam score in the final step N is accurate or incorrect.

[0155] Once the set of model parameters has been identified by the model trainer 840, the model 825 is trained and a verifier is configured to validate the model 825 using a validation dataset. The validation process performed by the verifier involves iterative operations of inputting the validation dataset into the trained model 825 using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, group leave-one-out cross-validation, nested cross-validation, etc. to tune the model parameters and ultimately find an optimal set of model parameters. Once the optimal set of model parameters is obtained, a reserved test set of data from the validation dataset is input into the trained model 825 to obtain outputs that are evaluated against ground truth values ​​using correlation techniques such as Bland-Altman and Spearman rank correlation coefficients to calculate performance metrics such as error, accuracy, precision, recall, and receiver operating characteristic curves (ROC). In some cases, obtaining, training, and validating data processes within model system 800 can be performed iteratively (tuned) by model trainer 840 until predetermined conditions are met, and a final set of model parameters can be provided by model trainer 840.

[0156] As will be appreciated, other training / validation mechanisms are contemplated and may be implemented within model system 800. For example, model 825 may be trained and model parameters may be tuned on a dataset from a subset of the obtained or filtered dataset, and the dataset from the subset of the obtained or filtered dataset may be used solely to test and evaluate the performance of model 825. Moreover, while the training mechanisms described herein focus on training new model 825, these training mechanisms may also be utilized to fine-tune existing models that have been trained from other datasets. For example, in some cases, model 825 may have been pre-trained using datasets from one or more different modalities or tasks. In those cases, model 825 may be used for transfer learning and retrained / validated using training and validation data as described above.

[0157] The training stage 805 outputs a trained model 825 having an optimized set of model parameters for use in the NL2LF conversion stage 810 or the confidence score calibration stage 815. The NL2LF conversion stage 810 includes a predictor for converting input data 855 (e.g., a natural language utterance) into a logical form 860 (e.g., an OMRL query). For example, the predictor inputs the input data 855 to a trained NL2LF model 865 and uses the trained NL2LF model 865 to perform a process for generating the logical form 860 based on features in the input data 855. In some cases, the trained NL2LF model 865 performs one or more semantic parsing tasks to generate predictions based on features extracted from the input data 855. The NL2LF conversion stage 810 outputs the logical form 860, which can be used in the confidence score calibration stage 815 and the query execution stage 820. The NL2LF conversion stage 810 also outputs one or more raw beam scores for one or more beam levels of the trained NL2LF model 865, as described in detail herein with respect to Figures 6 and 7, which can be used in the confidence score calibration stage 815.

[0158] The confidence score calibration stage 815 comprises a predictor for classifying the logical form 860 (e.g., an OMRL query) as accurate or incorrect. For example, the predictor inputs one or more raw beam scores of one or more beam levels of the trained NL2LF model 865 into a trained calibration model 870 and uses the trained calibration model 870 to perform a process for generating a binary classification based on features in the one or more raw beam scores. In some cases, the trained calibration model 870 performs a logistic regression task to generate a binary classification based on features extracted from the calibration model 870. The confidence score calibration stage 815 outputs a binary classification and a conditional probability score 875, which can be used as a confidence score for the logical form 860 in the query execution stage 820.

[0159] The query execution stage 820 includes one or more executors configured to execute the logical form 860 on a system, such as a database 880, to obtain results 885 (e.g., answers to queries in natural language utterances). For example, the one or more executors may be configured to convert or translate the semantic expression language of the logical form 860 into a system language query or command, such as SQL, API, REST, GraphQL, PGQL, etc., and execute the system language query or command against a relational database to obtain an answer to the query posed in the natural language utterance. The confidence score and conditional probability score 875 obtained from the binary classification enable better interpretation of the logical form 860 and results 885, analysis of shortcomings in the trained NL2LF model 865, and presentation of uncertainty to the end user. In particular, when C2OMRL is used as a skill in a digital assistant (DA), a low confidence score can be used to trigger a disambiguation prompt that is exposed to the user. The confidence score can also be utilized for internal evaluation studies to better understand the vulnerabilities of the trained NL2LF model 865 against certain categories of example inputs.

[0160] Although not explicitly shown, it will be appreciated that model system 800 can further include a developer device associated with a developer. Communications from the developer device to components of model system 800 can dictate the type of input data, utterances, and / or database schema to be used for the models, the number and type of models to be used, the hyperparameters of each model, e.g., learning rate and number of hidden layers, how data requests are formatted, what training data should be used (e.g., and how to obtain access to the training data), and what validation techniques should be used, and / or how the controller process should be configured.

[0161] FIG. 9 is a flowchart illustrating a process 900 for calibrating confidence scores of machine learning models trained to convert natural language into semantic language, according to various embodiments. The process illustrated in FIG. 9 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 9 and described below is intended to be exemplary and non-limiting. While FIG. 9 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, steps may be performed in some different order, or some steps may also be performed in parallel. In an embodiment, such as the embodiment illustrated in FIGS. 1-8, the process illustrated in FIG. 9 can be performed by a model system (e.g., model system 800) to create, train, update, and deploy one or more machine learning models configured to convert natural language into a logical form and use raw beam scores to determine calibrated confidence scores for the logical form.

[0162] In step 905, one or more raw beam scores are obtained. The one or more raw beam scores are generated from one or more beam levels of a decoder of a machine learning model that is trained to convert natural language into a logical form. Each of the one or more raw beam scores is a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more beam levels. The natural language may be expressed as a natural language utterance (e.g., a natural language query), and the logical form may be expressed as a logical form description, such as a query or action in a semantic representation language (e.g., OMRL).

[0163] In some cases, the machine learning model converts a natural language utterance into a logical form using a bottom-up generation process implemented by a heuristic search algorithm. The bottom-up generation process can include: (i) for a first beam level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form and selecting another K top subtrees to retain using a raw beam score associated with each subtree; (ii) for each beam level after the first beam level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form and the top K subtrees retained from the previous beam level and selecting another K top subtrees to retain using a raw beam score associated with each subtree; and (iii) for a final beam level, determining the subtree with the highest raw beam score as the final output of the decoder. The subtree with the highest raw beam score is converted into the logical form.

[0164] In some cases, the one or more beam levels include the first beam level, the last beam level, one or more beam levels between the first beam level and the last beam level, or any combination thereof. In some cases, the one or more raw beam scores are obtained from the first beam level, the last beam level, one or more beam levels between the first beam level and the last beam level, or any combination thereof. In some embodiments, the one or more raw beam scores are beam scores of the top K subtrees retained from the first beam level, the last beam level, and one or more beam levels between the first beam level and the last beam level.

[0165] In some cases, one or more raw beam scores are preprocessed to generate a vector. The vector includes a stack or sequential arrangement of one or more raw beam scores. In some cases, the order of one or more raw beam scores from a beam level is maintained in the vector (i.e., raw beam scores from the first beam level come first in the vector, followed by raw beam scores from the second beam level, followed by raw beam scores from the third beam level, and so on). In some cases, one or more raw beam scores or a subset of one or more raw beam scores from all beam levels are vectorized. In some cases, the top K raw beam scores from all beam levels are vectorized. In other cases, a subset of the top K raw beam scores from all beam levels is vectorized. In some cases, a subset of raw beam scores or raw beam scores from a subset of beam levels is vectorized (e.g., the last beam level or the last three beam levels). In some cases, the top K raw beam scores from a subset of beam levels are vectorized. In other cases, a subset of the top K raw beam scores from some beam level is vectorized.

[0166] In step 910, the logical form output by the machine learning model is classified as correct or incorrect based on one or more raw beam scores. The classification includes inputting the one or more raw beam scores into a calibration model and predicting, by the calibration model, a correct or incorrect class of the logical form based on the one or more raw beam scores. The correct or incorrect class represents a determination by the calibration model as to whether the subtree with the highest raw beam score at the last beam level of the decoder represents a correct or incorrect transformation of the natural language utterance, and the subtree with the highest raw beam score represents the logical form. In some cases, the conditional probability score is modeled using a sigmoid function. The classifying further includes outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest raw beam score is correct or incorrect. In some cases, the calibration model also outputs a correct or incorrect class of the logical form.

[0167] In step 915, a confidence score determined based on the conditional probability score is provided for the logical form. In some cases, the conditional probability score is used as the confidence score for the logical form. In some cases, a correct or incorrect class of the logical form is also provided.

[0168] In optional step 920, the process may be controlled based on the confidence score of the logical form. In some cases, training or retraining of a machine learning model may be controlled or optimized based on the confidence score of the logical form. In some cases, a determination may be made as to whether the confidence score exceeds a predetermined threshold, and a query may be performed against a database based on the logical form in response to a determination that the confidence score exceeds the predetermined threshold. In some cases, the confidence score of the logical form is reported to an end user.

[0169] Exemplary System 10 shows a simplified diagram of a distributed system 1000. In the illustrated example, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 can be configured to run one or more applications.

[0170] In various examples, server 1012 can be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 1012 can also provide other services or software applications, which may include non-virtual and virtual environments. In some examples, these services can be provided to users of client computing devices 1002, 1004, 1006, and / or 1008 as web-based or cloud services, such as based on a software-as-a-service (SaaS) model. Users operating client computing devices 1002, 1004, 1006, and / or 1008 can then utilize one or more client applications to interact with server 1012 and utilize the services provided by these components.

[0171] 10 , server 1012 may include one or more components 1018, 1020, and 1022 that perform functions performed by server 1012. These components may include software components that may be executed by one or more processors, hardware components, or combinations thereof. It should be appreciated that a variety of different system configurations are possible that may differ from distributed system 1000. Thus, the example shown in FIG. 10 is one example of a distributed system for implementing an exemplary system and is not intended to be limiting.

[0172] A user may use client computing devices 1002, 1004, 1006, and / or 1008 to execute one or more applications, models, or chatbots that may generate one or more events or models that may then be executed or serviced in accordance with the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user via this interface. Although FIG. 10 shows only four client computing devices, any number of client computing devices may be supported.

[0173] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux®-like operating systems such as Google Chrome™ OS), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®). Portable handheld devices may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays, and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.), etc. The client devices may be capable of running a variety of different applications, such as various Internet-related apps, communication applications (e.g., email applications, short message service (SMS) applications), etc., and may use a variety of communication protocols.

[0174] Network 1010 may be any type of network familiar to those skilled in the art capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, network 1010 may be a local area network (LAN), an Ethernet-based network, a Token-Ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating according to any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0175] The servers 1012 may be comprised of one or more general-purpose computers, specialized server computers (including, by way of example, PC (personal computer) servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable configuration and / or combination. The servers 1012 may also include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as an elastic pool of one or more logical storage devices that can be virtualized to maintain virtual storage devices for the servers. In various examples, the servers 1012 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0176] The computing systems within server 1012 may run one or more operating systems, including any of those described above, as well as any commercially available server operating system. Server 1012 may also run any of a variety of additional server and / or middle-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA® server, a database server, etc. Exemplary database servers include, without limitation, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc.

[0177] In some embodiments, server 1012 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. By way of example, the data feeds and / or event updates may include real-time updates received from one or more third-party sources and continuous data streams that may include, without limitation, Twitter® feeds, Facebook® updates, or real-time events related to sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 1012 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0178] The distributed system 1000 may also include one or more data repositories 1014, 1016. These data repositories, in one example, may be used to store data and other information. For example, one or more of the data repositories 1014, 1016 may be used to store information related to chatbot performance or information such as generated models for use by a chatbot used by the server 1012 in performing various functions according to various embodiments. The data repositories 1014, 1016 may reside in various locations. For example, the data repository used by the server 1012 may be local to the server 1012 or may be remote from the server 1012 and communicate with the server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 may be of different types. In one example, the data repository used by the server 1012 may be a database, such as a relational database, such as databases provided by Oracle Corporation® and other suppliers. One or more of these databases may be adapted to allow the storage, updating, and retrieval of data to and from the database in response to SQL-formatted commands.

[0179] In some examples, one or more of the data repositories 1014, 1016 may also be used by an application to store application data. The data repositories used by the application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general storage repository backed by a file system.

[0180] In some examples, the functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 11 is a simplified block diagram of a cloud-based system environment in which various services can be provided as cloud services, according to some examples. In the example shown in FIG. 11, a cloud infrastructure system 1102 can provide one or more cloud services that can be requested by users using one or more client computing devices 1104, 1106, and 1108. The cloud infrastructure system 1102 can include one or more computers and / or servers, which may include those described above for server 1012. The computers in the cloud infrastructure system 1102 may be organized as general-purpose computers, specialized server computers, server farms, server clusters, or any other suitable configuration and / or combination.

[0181] The network 1110 can facilitate communication and the exchange of data between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network 1110 can include one or more networks. The networks can be of the same or different types. The network 1110 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0182] The example shown in Figure 11 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in some other examples, cloud infrastructure system 1102 may have more or fewer components than those shown in Figure 11, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 11 shows three client computing devices, in alternative examples, any number of client computing devices may be supported.

[0183] The term cloud service is generally used to refer to services made available to users on demand and over a communications network, such as the Internet, by a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from a customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Thus, customers can use cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system may host applications, and users can order and use the applications as needed over the Internet, without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, Java cloud services, etc., are offered by Oracle Corporation of Redwood Shores, California.

[0184] In one example, cloud infrastructure system 1102 can provide one or more cloud services using various models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including hybrid service models. Cloud infrastructure system 1102 may include a set of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0185] The SaaS model allows applications or software to be delivered to customers as a service over a communications network, such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 1102. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human capital / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, etc.

[0186] The IaaS model is generally used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) as cloud services to customers to provide elastic computing and storage capabilities. Various IaaS services are offered by Oracle Corporation.

[0187] The PaaS model is generally used to provide platform and environment resources as a service, allowing customers to develop, run, and manage applications and services without the customer having to acquire, build, or maintain such resources. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java® Cloud Services (JCS), Oracle Database Cloud Services (DBCS), data management cloud services, various application development solution services, etc.

[0188] Cloud services are generally provided in an on-demand, self-service manner, on a subscription basis, elastically scalable, and highly reliable, available, and secure. For example, a customer may order one or more services provided by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent), as described above, and / or provide the services of a chatbot system as described herein. Cloud infrastructure system 1102 may be configured to provide one or even more cloud services.

[0189] Cloud infrastructure system 1102 can provide cloud services through several different deployment models. In a public cloud model, cloud infrastructure system 1102 may be owned by a third-party cloud service provider, and cloud services are offered to any public customer, where the customer may be an individual or a business. In another example, under a private cloud model, cloud infrastructure system 1102 may be operated within an organization (e.g., within a corporate organization) and services offered to customers within the organization. For example, customers may be various departments of a company, such as a human resources department, a payroll department, etc., or even individuals within the company. In another example, under a community cloud model, cloud infrastructure system 1102 and the services offered may be shared by several organizations within an associated community. Various other models, such as hybrids of the above models, may also be used.

[0190] Client computing devices 1104, 1106, and 1108 may be of different types (e.g., client computing devices 1002, 1004, 1006, and 1008 shown in FIG. 10 ) and may be capable of running one or more client applications. Users may use the client devices to interact with cloud infrastructure system 1102, such as to request services provided by cloud infrastructure system 1102. For example, users may use the client devices to request information or actions from a chatbot, as described in this disclosure.

[0191] In some examples, the processing performed by cloud infrastructure system 1102 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing the data in parallel, performing simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 1102 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0192] 11 , cloud infrastructure system 1102 may include infrastructure resources 1130 utilized to facilitate the provision of various cloud services provided by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc. In one example, a storage virtual machine available to service storage requested by an application may be part of cloud infrastructure system 1102. In other examples, the storage virtual machine may be part of a different system.

[0193] In one example, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 1102 to different customers, resources can be bundled into sets of resources or resource modules (also referred to as “pods”). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In one example, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provided for database services, a second set of pods may be provided for Java services, and so on, which may include a different combination of resources than the pods in the first set of pods. For some services, resources allocated to provide the service may be shared between services.

[0194] Cloud infrastructure system 1102 may itself use services 1132 internally that are shared by different components of cloud infrastructure system 1102 and that facilitate the provision of services by cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and restore services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0195] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 11 , the subsystems may include a user interface subsystem 1112 that allows users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 through which cloud services offered by cloud infrastructure system 1102 are advertised and available for purchase by consumers, and other interfaces 1118. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 1102 (service request 1134) using one or more of interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 1102, and place a subscription order for one or more services offered by cloud infrastructure system 1102 to which the customer wishes to subscribe. The service request may include information identifying a customer and one or more services to which the customer wishes to subscribe. For example, a customer may place a subscription order for services provided by cloud infrastructure system 1102. As part of the order, the customer may provide information identifying a chatbot system to which the service will be provided, and optionally one or more authentication credentials for that chatbot system.

[0196] 11 , cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this processing, OMS 1120 may be configured to create an account for the customer if not already done so, receive billing and / or accounting information from the customer that will be used to bill the customer for providing the customer with the requested services, verify the customer information, and upon verification, register the customer's order and coordinate various workflows to prepare the order for delivery.

[0197] Upon proper validation, the OMS 1120 can then invoke an order fulfillment subsystem (OPS) 1124, which is configured to provide resources for the order, including processing, memory, and networking resources. The fulfillment may include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provided for the order and the type of resources provided may depend on the type of cloud service ordered by the customer. For example, according to one workflow, the OPS 1124 can be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for that specific cloud service. The number of pods allocated to the order may depend on the size / amount / level / scope of the service being requested. For example, the number of pods allocated may be determined based on the number of users to be supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized to provide the requested service for the specific requesting customer.

[0198] In one example, the setup phase process described above may be performed as part of the provisioning process by cloud infrastructure system 1102. Cloud infrastructure system 1102 may generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by cloud infrastructure system 1102 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 1102.

[0199] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service is ready for use at that time. In some cases, information (e.g., a link) may be sent to the customer to enable the customer to begin using and taking advantage of the benefits of the requested service. In one example, for the customer requesting the service, the response may include a chatbot system ID generated by cloud infrastructure system 1102 and information identifying the chatbot system selected by cloud infrastructure system 1102 for the chatbot system corresponding to the chatbot system ID.

[0200] Cloud infrastructure system 1102 may provide services to multiple customers. For each customer, cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1102 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0201] Cloud infrastructure system 1102 can provide services to multiple customers in parallel. Cloud infrastructure system 1102 can store information for these customers, possibly including sensitive information. In one example, cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage customer information and provide separation of managed information so that information associated with one customer is not accessible by another customer. IMS 1128 can be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0202] 12 illustrates an example of a computer system 1200. In some examples, the computer system 1200 can be used to run a digital assistant or chatbot system in a distributed environment, as well as any of the various servers and computer systems described above. As shown in FIG. 12, the computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 may include a non-transitory computer-readable storage medium, including a storage medium 1222, and a system memory 1210.

[0203] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. While bus subsystem 1202 is shown schematically as a single bus, alternative examples of a bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a Mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0204] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may be a single-core or multi-core processor. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1204 may include one or more special-purpose coprocessors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 1204 may be implemented using custom circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0205] In some examples, processing units within processing subsystem 1204 can execute instructions stored in system memory 1210 or on computer-readable storage medium 1222. In various examples, the processing units can execute various program or code instructions and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code being executed can reside in system memory 1210 and / or on computer-readable storage medium 1222, which in some cases includes one or more storage devices. Through appropriate programming, processing subsystem 1204 can provide the various functions described above. In instances where computer system 1200 is running one or more virtual machines, one or more processing units can be allocated to each virtual machine.

[0206] In one example, a processing acceleration unit 1206 may optionally be provided to perform customized processing or to offload portions of the processing performed by the processing subsystem 1204 so as to accelerate the overall processing performed by the computer system 1200.

[0207] I / O subsystem 1208 may include devices and mechanisms for inputting information into computer system 1200 and / or outputting information from or through computer system 1200. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, acoustic input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion-sensing and / or gesture-recognition devices such as a Microsoft Kinect® motion sensor, a Microsoft Xbox® 360 game controller, or devices that provide an interface for receiving input using gesture and voice commands, allowing a user to control and interact with the input device. The user interface input device may also include an eye gesture recognition device, such as a Google Glass® blink detector, that detects eye activity from the user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gesture as input to the input device (e.g., Google Glass®). Additionally, the user interface input device may include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., Siri® Navigator) through voice commands.

[0208] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, and eye-tracking devices. Additionally, user interface input devices may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0209] In general, use of the term output device(s) is intended to include all possible types of devices and mechanisms for outputting information from computer system 1200 to a user or to another computer. User interface output devices may also include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may be flat-panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touch screens, etc. For example, user interface output devices may include various display devices that visually convey textual, graphical, and audio / video information, such as, but not limited to, monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0210] The storage subsystem 1218 provides a repository or data store for storing information and data used by the computer system 1200. The storage subsystem 1218 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data structures that provide some example functionality. The storage subsystem 12112 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above. The software may be executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 can also provide authentication according to the teachings of the present disclosure.

[0211] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 12, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include several memories, including volatile main random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory, in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 1200, such as during start-up, may typically be stored in ROM. The RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 1204. In some embodiments, the system memory 1210 may include several different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0212] 12, system memory 1210 may load executing application programs 1212, program data 1214, and operating system 1216, which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc. By way of example, operating system 1216 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux® operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0213] The computer-readable storage medium 1222 can store programming and data structures that provide some example functionality. The computer-readable medium 1222 can provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1200. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above may be stored in the storage subsystem 1218. By way of example, the computer-readable storage medium 1222 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, and an optical disk drive, such as a Blu-ray® disk or other optical media. The computer-readable storage medium 1222 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash device, a Secure Digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, etc., DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0214] In some examples, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may be further connected to a computer-readable storage medium 1222. Reader 1220 may receive data from a memory device such as a disk, flash drive, or the like and may be configured to read the data.

[0215] In some examples, computer system 1200 may support virtualization technology, including, but not limited to, virtualization of processing and memory resources. For example, computer system 1200 may provide support for running one or more virtual machines. In some examples, computer system 1200 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may have allocated memory resources, computational resources (e.g., processors, cores), I / O resources, and networking resources. Each virtual machine generally operates independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines executed by computer system 1200. Thus, multiple operating systems may potentially be run simultaneously by computer system 1200.

[0216] The communications subsystem 1224 provides an interface to other computer systems and networks. The communications subsystem 1224 serves as an interface for receiving data from other systems and transmitting data from the computer system 1200 to other systems. For example, the communications subsystem 1224 can enable the computer system 1200 to establish a communications channel to one or more client devices over the Internet to receive and transmit information from the client devices. For example, when the computer system 1200 is used to execute the bot system 120 shown in FIG. 1, the communications subsystem can be used to communicate with a chatbot system selected for the application.

[0217] The communications subsystem 1224 can support both wired and / or wireless communications protocols. In certain examples, the communications subsystem 1224 can include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some examples, the communications subsystem 1224 can provide wired network connections (e.g., Ethernet) in addition to or instead of a wireless interface.

[0218] The communications subsystem 1224 can receive and transmit data in various formats. In some examples, in addition to other formats, the communications subsystem 1224 can receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. For example, the communications subsystem 1224 can be configured to receive (or transmit) data feeds 1226 in real time from users of social media networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0219] In one example, the communications subsystem 1224 can be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230, which may be continuous or unlimited in nature with no apparent end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0220] The communications subsystem 1224 may also be configured to communicate data from the computer system 1200 to other computer systems or networks. The data may be communicated in a variety of different forms, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1200.

[0221] Computer system 1200 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, the description of computer device 1200 shown in FIG. 12 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 12. It will be appreciated that there are other manners and / or ways to implement the various examples based on the present disclosure and the teachings provided herein.

[0222] While specific examples have been described, various modifications, variations, alternative configurations, and equivalents are possible. The examples are not limited to operation in one particular data processing environment, but can freely operate in multiple data processing environments. Additionally, while certain examples are described using a particular sequence of transactions and steps, those skilled in the art will appreciate that this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may have additional steps not included in the figures. Various features and aspects of the examples described above may be used individually or jointly.

[0223] Additionally, while certain examples are described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. An example may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or on multiple different processors in any combination.

[0224] Where a device, system, component, or module is described as being configured to perform a certain operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, such as by executing computer instructions or code, or by a processor or core that is programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0225] Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the examples. This description provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the preceding description of the examples provides those skilled in the art with an empowering description for practicing various examples. Various changes can be made in the function and arrangement of elements.

[0226] Accordingly, the specification and drawings should be regarded in an illustrative, and not a restrictive, sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the appended claims. Accordingly, while particular examples have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0227] In the foregoing specification, aspects of the present disclosure have been described with reference to particular examples thereof, but those skilled in the art will recognize that the present disclosure is not limited thereto. Various features and aspects of the above-described disclosure may be used individually or jointly. Moreover, the examples may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the specification and drawings should be considered illustrative and not restrictive.

[0228] In the above description, for purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative examples, the methods may be performed in an order different from that described. It should also be appreciated that the methods described above may be performed by hardware components or embodied in a series of machine-executable instructions that can be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuitry that has been programmed with the instructions, to perform the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, or any other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0229] Where an element is described as being configured to perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0230] Although illustrative examples of the present application have been described in detail herein, it is to be understood that the inventive concepts may be variously embodied and utilized in other manners, and that the appended claims are intended to be construed to include such modifications except insofar as limited by the prior art.

Claims

1. 1. A computer-implemented method comprising: obtaining one or more raw beam scores generated from one or more beam levels of a decoder of a machine learning model trained to convert natural language into a logical form, each of the one or more raw beam scores being a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more beam levels; The computer-implemented method comprises: further comprising classifying a logical form output by the machine learning model as accurate or inaccurate based on the one or more raw beam scores, wherein the classifying includes: inputting the one or more raw beam scores into a calibration model; predicting, by the calibration model, a correct or incorrect class of the logical form based on the one or more raw beam scores, wherein the correct or incorrect class represents a determination by the calibration model as to whether a subtree with a highest raw beam score at a final beam level of the decoder represents a correct or incorrect transformation of a natural language utterance, the subtree with the highest raw beam score representing the logical form; The classification may include: outputting a conditional probability score for whether the subtree with the highest raw beam score is correct or incorrect according to the calibration model; further comprising The computer-implemented method comprises: providing a confidence score to the logical form that is determined based on the conditional probability score; determining that the confidence score is above a predetermined threshold; responsive to determining that the confidence score exceeds the predetermined threshold, executing a query against a database based on the logical form; The computer-implemented method further comprising:

2. and further comprising converting the natural language utterance into the logical form by the machine learning model, the conversion comprising a bottom-up generation process performed by the heuristic search algorithm, the bottom-up generation process comprising: For a first beam level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form and selecting the top K subtrees to retain using a raw beam score associated with each subtree; for each beam level after the first beam level, generating the F possible subtrees of the natural language utterance based on the vocabulary or grammar associated with the logical form and the top K subtrees retained from the previous beam level, and selecting another top K subtrees to retain using a raw beam score associated with each subtree; determining, for the last beam level, the subtree with the highest raw beam score as the final output of the decoder; 2. The computer-implemented method of claim 1, comprising:

3. 3. The computer-implemented method of claim 2, wherein the one or more beam levels include the first beam level, the last beam level, one or more beam levels between the first beam level and the last beam level, or any combination thereof.

4. 4. The computer-implemented method of claim 3, wherein the one or more raw beam scores are obtained from the first beam level, the last beam level, the one or more beam levels between the first beam level and the last beam level, or any combination thereof.

5. 4. The computer-implemented method of claim 3, wherein the one or more raw beam scores are the raw beam scores of the top K subtrees retained from the first beam level, the last beam level, and one or more beam levels between the first beam level and the last beam level.

6. The computer-implemented method of claim 1 , wherein the conditional probability scores are modeled using a sigmoid function, and the conditional probability scores are used as the confidence scores of the logical forms.

7. 1. A system comprising: one or more processors; one or more non-transitory computer-readable media storing a program that, when executed by the one or more processors, causes the one or more processors to perform the method of any one of claims 1 to 6; A system comprising:

8. A computer program causing one or more data processors to carry out the method of any one of claims 1 to 6.

Images