Calibrating confidence scores for machine learning models trained as natural language interfaces

Abstract

Techniques are disclosed herein for calibrating a confidence score of a machine learning model trained to convert a natural language into a meaning representation language. The techniques include obtaining one or more original bundling scores generated from one or more bundling levels of a decoder trained as a machine learning model that translates natural language to logical form, each of the one or more original bundling scores is a conditional probability of a sub-tree determined by a heuristic search algorithm of the decoder at one of the one or more bundling levels; classifying, by the calibration model, the logical form output by the machine learning model as correct or incorrect based on the one or more raw bundling scores; and providing a confidence score for the logical form, the confidence score determined based on the classification of the logical form.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to non-provisional application No. 18 / 107,624, filed on February 9, 2023, which claims the benefit and priority of U.S. Provisional Application No. 63 / 400,037, filed on August 22, 2022, under 35 U.S.C. 119(e), the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] The present disclosure relates generally to transforming natural language into a meaning representation language, and more particularly to machine learning techniques for calibrating confidence scores of a machine learning model trained to transform natural language into a meaning representation language, such as Oracle Meaning Representation Language (OMRL). Background Art

[0004] In order to obtain an immediate response, many users around the world use instant messaging or chat platforms. Organizations often use these instant messaging or chat platforms to conduct real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots or robots have begun to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with robots through messaging applications that end users have installed and used. Intelligent robots (usually powered by artificial intelligence (AI)) can communicate more intelligently and contextually in real-time conversations, and therefore can allow more natural conversations between robots and end users to improve the conversation experience. Instead of the end user learning a fixed set of keywords or commands that the robot knows how to respond to, the intelligent robot can understand the end user's intentions based on the user's utterances in natural language and respond accordingly.

[0005] The chatbot can have both an analog (human) interface and a digital (machine) interface for interacting with humans and connecting to backend systems. Advantageously, it is possible to extract and analyze the meaning of an utterance (e.g., a request) when a human uses natural language, regardless of how the utterance will be processed by the backend system. As an example, a request may be for data that needs to be obtained from a relational database, or the requested data may need to be extracted from a knowledge graph. Meaning Representation Language (MRL) is a universal representation of natural language utterances that a chatbot can convert into any number of machine-oriented target languages. Therefore, a chatbot can utilize MRL to communicate interchangeably with both humans and various backend systems, including systems that communicate using structured query language (SQL), application programming interface (API), representational state transfer (REST), graph query language (GraphQL), property graph query language (PGQL), etc.

[0006] For example, SQL is a standard database management language for interacting with relational databases. SQL can be used to store, manipulate, obtain and / or otherwise manage data stored in a relational database management system (RDBMS) and / or for stream processing in a relational data stream management system (RDSMS). SQL includes statements or commands for interacting with relational databases. SQL statements or commands are particularly classified into data query language (DQL) statements, data definition language (DDL) statements, data control language (DCL) statements, and data manipulation language (DML) statements. To interact with relational databases using SQL, users must know how the database is structured (e.g., understand the rows and columns in the table and each table), SQL syntax, and how to associate syntax with database structure. Without this knowledge, users are usually difficult to interact with these relational databases using SQL.

[0007] Natural language interfaces (e.g., chatbots) to database systems (NLIDBs) such as RDBMS provide users with a method for interacting with these relational databases in an intuitive manner without knowledge of the database management language. For example, using natural language statements and queries (i.e., natural language queries), users can interact with these relational databases in a concise language via NLIDB. Recently, text-to-SQL systems have become popular, and deep learning methods for converting natural language queries into SQL queries have proven promising. Using semantic parsing, natural language statements, requests, and questions (i.e., sentences) can be transformed into machine-oriented languages ​​that can be executed by applications (e.g., chatbots, models, programs, machines, etc.). For example, semantic parsing can transform natural language sentences into general programming languages ​​such as Python, Java, and SQL. The process for transforming natural language sentences into SQL queries typically includes rule-based systems, statistical-based systems, and / or deep learning-based systems. Rule-based systems typically use a series of fixed rules to convert natural language sentences into SQL queries. These rule-based systems are typically domain-specific and are therefore considered to lack flexibility and cannot be well generalized to new use cases (e.g., across different domains). Statistics-based systems (such as slot filling) label tokens (i.e., words or phrases) in an input natural language sentence according to their semantic roles in the sentence, and use these labels to fill slots in SQL queries. Typically, these statistics-based systems have restrictions on the types of sentences that can be parsed (e.g., sentences must be able to be represented as parse trees). Deep learning-based systems (such as sequence-to-sequence models) involve training deep learning models that directly convert natural language sentences into machine-oriented language and have been shown to generalize across tasks, domains, and datasets. However, such deep learning systems require a large amount of training data for supervised learning, and obtaining labeled data (e.g., natural language query-SQL statement pairing) is challenging. Therefore, converting natural language sentences into machine-oriented language based on deep learning cannot avoid the need for a large amount of labeled training data. Summary of the invention

[0008] Machine learning techniques (e.g., a method, a system, a non-transitory computer-readable medium storing code or instructions executable by one or more processors) are provided for calibrating confidence scores of a machine learning model trained to convert natural language to a meaning representation language, such as Oracle Meaning Representation Language (OMRL).

[0009] In various embodiments, a computer-implemented method (e.g., for processing natural language) is provided, the method comprising: obtaining one or more raw bundle scores generated from one or more bundle levels of a decoder of a machine learning model trained to convert natural language into logical form, wherein each of the one or more raw bundle scores is a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more bundle levels; classifying a logical form output by the machine learning model as correct or incorrect based on the one or more raw bundle scores, wherein the classification comprises: inputting the one or more raw bundle scores into a calibration model; predicting, by the calibration model, a correct or incorrect category of the logical form based on the one or more raw bundle scores, wherein the correct or incorrect category represents a determination by the calibration model as to whether a subtree with the highest raw bundle score at a last bundle level of the decoder represents a correct or incorrect conversion of a natural language utterance, and the subtree with the highest raw bundle score represents a logical form; and outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest raw bundle score is correct or incorrect; and providing a confidence score for the logical form, the confidence score being determined based on the conditional probability score.

[0010] In some embodiments, the computer-implemented method further includes converting the natural language utterance into a logical form by a machine learning model, wherein the conversion includes a bottom-up generation process implemented by a heuristic search algorithm, and wherein the bottom-up generation process includes: for a first clustering level, generating F possible subtrees of the natural language utterance based on the vocabulary or grammar associated with the logical form, and selecting the top K subtrees to retain using the original clustering score associated with each subtree; for each clustering level after the first clustering level, generating F possible subtrees of the natural language utterance based on the vocabulary or grammar associated with the logical form and the top K subtrees retained from the previous clustering level, and selecting another top K subtrees to retain using the original clustering score associated with each subtree; and for the last clustering level, determining the subtree with the highest original clustering score as the final output of the decoder.

[0011] In some embodiments, the one or more bundling levels include a first bundling level, a last bundling level, one or more bundling levels between the first bundling level and the last bundling level, or any combination thereof.

[0012] In some embodiments, the one or more raw bundle scores are obtained from a first bundle level, a last bundle level, one or more bundle levels between the first bundle level and the last bundle level, or any combination thereof.

[0013] In some embodiments, the one or more raw bundling scores are the bundling scores of the first K subtrees retained from the first bundling level, the last bundling level, and one or more bundling levels between the first bundling level and the last bundling level.

[0014] In some embodiments, the conditional probability scores are modeled using a sigmoid function and used as confidence scores in logical form.

[0015] In some embodiments, the computer-implemented method further includes: determining that the confidence score is above a predetermined threshold; and in response to determining that the confidence score is above the predetermined threshold, performing a query on the database based on the logical form.

[0016] In various embodiments, a system is provided that includes one or more processors and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the system to perform part or all of one or more methods disclosed herein.

[0017] In various embodiments, a computer program product tangibly embodied in one or more non-transitory machine-readable media is provided, the computer program product comprising instructions executed by one or more processors to perform part or all of one or more methods disclosed herein.

[0018] In various embodiments, an apparatus is provided that includes means for implementing part or all of the operations and / or methods disclosed herein.

[0019] In various embodiments, a computer program product is provided that includes computer instructions that, when executed by one or more processors, implement some or all of the operations and / or methods disclosed herein.

[0020] The technology described above and below can be implemented in a variety of ways and in a variety of contexts. As described in more detail below, several example implementations and contexts are provided with reference to the following figures. However, the following implementations and contexts are only some of many implementations and contexts. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a simplified block diagram of a distributed environment incorporating an exemplary embodiment.

[0022] Figure 2 is a simplified block diagram of a computing system implementing a master robot according to some embodiments.

[0023] Figure 3is a simplified block diagram of a computing system that implements a skill robot in accordance with certain embodiments.

[0024] Figure 4A and Figure 4B An example reliability graph for a calibrated conversation-to-OMRL (C2OMRL) model is shown in accordance with various embodiments.

[0025] Figure 5 is a simplified block diagram of a C2OMRL architecture according to various embodiments.

[0026] Figure 6 The beam search process of a decoder according to various embodiments is illustrated.

[0027] Figure 7 A confidence score calibration system for confidence scores according to various embodiments is illustrated.

[0028] Figure 8 Illustrated is a model system for training and deploying models according to various embodiments.

[0029] Fig. 9 Illustrated is a process flow for calibrating confidence scores of a machine learning model trained to convert natural language to meaning-representing language, according to various embodiments.

[0030] Fig.10 Depicted is a simplified diagram of a distributed system for implementing various embodiments.

[0031] Fig.11 is a simplified block diagram of one or more components of a system environment according to various embodiments by which services provided by one or more components of an embodiment system may be provided as a cloud service.

[0032] Fig.12 An example computer system is illustrated that can be used to implement the various embodiments. DETAILED DESCRIPTION

[0033] In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments or designs.

[0034] introduction

[0035] In recent years, the amount of data supporting different industries and their systems has grown exponentially. Most business information is stored in the form of relational databases that store, process, and retrieve data. Databases support information systems across multiple industries, such as consumer technology (e.g., orders, cancellations, refunds), supply chain (e.g., raw materials, inventory, suppliers), healthcare (e.g., medical records), finance (e.g., financial business indicators), customer support, search engines, and more. Modern data-driven companies must track the real-time status of their business in order to quickly understand and diagnose any problems, trends, or anomalies that appear in the data and take immediate corrective actions. This work is usually performed manually by analysts who write complex queries in query languages ​​(e.g., database query languages ​​such as declarative query languages ​​such as SQL, PGQL, logical database queries, API query languages ​​such as GraphQL, REST, etc.). Writing such queries can be used to obtain in-depth information from data stored in multiple tables. These results are usually processed in the form of charts or graphs to enable users to quickly visualize the results and facilitate data-driven decision making.

[0036] Although common database queries (e.g., SQL queries) are often predefined and incorporated into commercial products, any new or subsequent queries still need to be manually coded by analysts. This static interaction between database queries and the consumption of corresponding results requires time-consuming manual intervention and results in slow feedback cycles. It is very efficient to enable non-technical users (e.g., business leaders, doctors, or other data users) to interact directly with analysis tables via natural language (NL) queries that abstract the underlying query language (e.g., SQL) code. Defining database queries requires a deep understanding of database schemas and query language syntax, which can quickly become overwhelming for beginners and non-technical stakeholders. To bridge this communication gap, efforts have been made to develop a new type of processing called NLIDB. In recent years, this natural search capability has become increasingly popular as companies are developing deep learning methods for natural language to logical form (NL2LF), such as natural language to SQL (NL2SQL). Logical form can refer to a meaning representation language and / or a machine-oriented language. NL2SQL seeks to transform natural language questions into SQL, allowing individuals to run unstructured queries against databases. The transformed SQL can also enable digital assistants (such as chatbots) to improve their responses when the answer can be found in a different database or table.

[0037] However, in addition to logical forms such as SQL or Oracle's own meaning representation language (Oracle Meaning Representation Language (OMRL)) similar to SQL, it is important to obtain confidence scores associated with the logical form output generated by an NL2LF model (e.g., an NL2SQL model or a conversational to OMRL (C2OMRL) model). A confidence score is an indicator, such as a number between 0 and 1, that represents the likelihood that the output of the machine learning model is correct and will satisfy the user's request. Confidence scores are important because the output from an NL2LF model can be difficult to interpret in some cases, and it is difficult to understand how confident the model is in the predicted logical form (e.g., OMRL) by looking at the rough output. Confidence scores will help to better explain responses, analyze model shortcomings, and present uncertainty to end users. In particular, when the C2OMRL model is used as a skill for a digital assistant (DA), a low confidence score can be used as a trigger to issue disambiguation prompts to the user. Confidence scores can also be used in internal evaluation studies to better understand the model's weaknesses on certain types of input examples. However, in order for confidence scores to be useful, it is crucial that they are calibrated, i.e., they can be directly interpreted as the probability that the model is correct.

[0038] Therefore, it is necessary to take measures to address these challenges and other challenges. The method described in this article includes training and deploying a confidence score calibration model, which uses the intermediate "raw" score generated at one or more steps in the decoding process of the NL2LF model to determine the calibrated confidence score of the NL2LF model. More specifically, the decoder of the NL2LF model is based on a bottom-up generation process, wherein the final generation output is a logical form tree (i.e., a tree-based structure representing a complete logical form). The generation process is performed in steps, also known as "bundle level". At each step (e.g., "step i"), the process generates all possible subtrees that can be obtained by expanding the current subtree (from step "i-1"), and then the process uses the original score associated with each subtree to retain only the first K subtrees. These scores are referred to as "original bundle scores" in this article, so the first K intermediate results (which will be taken into account in the next generation step) are the K intermediate results with the highest original bundle scores. The confidence score calibration model (also described herein as a calibration model) is a binary classifier that takes as input a vector of one or more raw bundle scores with one or more bundle levels, and uses the one or more raw bundle scores to predict whether the final output of the NL2LF model (e.g., the final ORML tree) is correct or incorrect. The probability scores from the calibration model (e.g., the output of the sigmoid function in logistic regression) become the calibrated NL2LF model confidence scores. Advantageously, the raw bundle scores of the candidate subtrees can reveal information about the uncertainty of the NL2LF model during the bottom-up generation process (e.g., caused by the model's weakness on certain types of examples or by ambiguous inputs), and thus they are a good source of information for the calibrated confidence scores.

[0039] In various embodiments, a computer-implemented method is provided, the computer-implemented method comprising obtaining one or more raw bundle scores generated from one or more bundle levels of a decoder of a machine learning model trained to convert natural language into logical form. Each of the one or more raw bundle scores is a conditional probability of a subtree determined by a heuristic search algorithm of the decoder in one of the one or more bundle levels. The computer-implemented method further comprises classifying a logical form output by the machine learning model as correct or incorrect based on the one or more raw bundle scores. The classification comprises: inputting the one or more raw bundle scores into a calibration model; predicting, by the calibration model, a correct or incorrect category of the logical form based on the one or more raw bundle scores, wherein the correct or incorrect category represents a determination by the calibration model whether the subtree with the highest raw bundle score at the last bundle level of the decoder represents a correct or incorrect conversion of the natural language utterance, and the subtree with the highest raw bundle score represents a logical form; and outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest raw bundle score is correct or incorrect. The computer-implemented method further comprises providing a confidence score for the logical form, the confidence score being determined based on the conditional probability score.

[0040] As used herein, when an action is "based on" something, it means that the action is based, at least in part, on at least a portion of that something. As used herein, the terms "similarly," "substantially," "approximately," and "about" are defined as being largely, but not necessarily entirely, what is specified (and entirely including what is specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "similarly," "substantially," "approximately," or "about" may be replaced with "within [percentage] of what is specified," where the percentage includes 0.1%, 1%, 5%, and 10%.

[0041] Robotics and analytics systems

[0042] A robot (also referred to as a skill, chatbot, conversational robot, or conversational robot) is a computer program that can perform a conversation with an end user. A robot can typically respond to a natural language message (e.g., a question or comment) through a messaging application that uses natural language messages. An enterprise can use one or more robot systems to communicate with end users through a messaging application. A messaging application (which may be referred to as a channel) may be a messaging application preferred by an end user that the end user has installed and is familiar with. Therefore, the end user does not need to download and install a new application in order to chat with the robot system. The messaging application may include, for example, an over-the-top (OTT) messaging channel (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), a virtual personal assistant (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), a mobile and web application extension of a local or hybrid extension / responsive mobile application or web application with a chat function, or a voice-based input (such as a device or application with other voice input using Siri, Microsoft Cortana, Google Voice, or other voice input for interaction).

[0043] In some examples, a robot system can be associated with a uniform resource identifier (URI). The URI can identify a robot system using a string of characters. The URI can be used as a webhook for one or more messaging application systems. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. The HTTP post call message can involve a URI from the messaging application system. In some embodiments, the message can be different from the HTTP post call message. For example, the robot system can receive a message from a short message service (SMS). Although the discussion herein may refer to the communication received by the robot system as a message, it should be understood that the message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0044] End users can interact with the robot system through conversational interactions (sometimes called conversational user interfaces (UIs)), just as people interact with each other. In some cases, the interaction can include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking the end user how the robot can help. In some cases, the interaction can also be a transactional interaction with, for example, a banking robot, such as transferring money from one account to another; an informational interaction with, for example, an HR robot, such as inquiring about vacation balances; or an interaction with, for example, a retail robot, such as discussing returning a purchased item or seeking technical support.

[0045] In some embodiments, the robot system can intelligently handle end-user interactions without interacting with the administrator or developer of the robot system. For example, the end user can send one or more messages to the robot system in order to achieve the desired goal. The message can include some content, such as text, emoticons, audio, images, videos, or other methods of conveying messages. In some embodiments, the robot system can convert the content into a standardized form (e.g., a representational state transfer (REST) ​​call for enterprise services with appropriate parameters) and generate a natural language response. The robot system can also prompt the end user to enter additional input parameters or request other additional information. In some embodiments, the robot system can also initiate communication with the end user instead of passively responding to the end user speech. Described herein are various technologies for identifying explicit calls to the robot system and determining the input of the called robot system. In some embodiments, the explicit call analysis is performed by the main robot based on the call name detected in the speech. In response to detecting the call name, the speech can be refined for input to the skill robot associated with the call name.

[0046] A conversation with a robot can follow a specific conversation flow including multiple states. The flow can define what happens next based on input. In some embodiments, a state machine including user-defined states (e.g., end-user intent) and actions to be taken in or between states can be used to implement a robot system. Conversations can take different paths based on end-user input, which may affect the decision made by the robot for the flow. For example, in each state, based on end-user input or speech, the robot can determine the end-user's intent in order to determine the next appropriate action to be taken. As used herein and in the context of speech, the term "intent" refers to the intention of the user providing the speech. For example, a user may intend to have the robot participate in a conversation for ordering pizza, so that the user's intent can be represented by the speech "Order pizza". User intent can relate to a specific task that the user wants the chatbot to perform on behalf of the user. Therefore, speech can be expressed as questions, commands, requests, etc. that reflect the user's intent. Intent can include goals that the end user wants to accomplish.

[0047] In the context of the configuration of the chatbot, the term "intent" used herein refers to configuration information for mapping the user's speech to a specific task / action or a specific type of task / action that the chatbot can perform. In order to distinguish the intent of the speech (i.e., the user's intent) from the intent of the chatbot, the latter is sometimes referred to as "robot intent" herein. The robot intent may include a set of one or more utterances associated with the intent. For example, the intent to order pizza may be conveyed by various utterance arrangements expressing the expectation of placing an order to buy pizza. These associated utterances may be used to train the chatbot's intent classifier so that the intent classifier can then determine whether the input utterance from the user matches the intent to order pizza. The robot intent may be associated with one or more dialog flows for initiating a conversation with the user in a certain state. For example, the first message for the intent to order pizza may be the question "What kind of pizza would you like?" In addition to the associated utterances, the robot intent may further include named entities related to the intent. For example, the intent to order pizza may include variables or parameters for performing the task of ordering pizza, such as filling 1, filling 2, pizza type, pizza size, pizza quantity, etc. The value of the entity is usually obtained by talking to the user.

[0048] Figure 11 is a simplified block diagram of an environment 100 incorporating a chatbot system according to some embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102 that enables users of the DABP 102 to create and deploy digital assistants or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, Figure 1 As shown, a user 104 representing a particular business can use DABP 102 to create and deploy a digital assistant 106 for the users of the particular business. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0049] For purposes of this disclosure, a "digital assistant" is an entity that assists the user of the digital assistant in completing various tasks through natural language conversation. A digital assistant may be implemented using software alone (e.g., a digital assistant is a digital entity implemented using a program, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in a variety of physical systems or devices such as computers, mobile phones, watches, appliances, vehicles, and the like. A digital assistant is also sometimes referred to as a chatbot system. Therefore, for purposes of this disclosure, the terms "digital assistant" and "chatbot system" are interchangeable.

[0050] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and obtain responses 112 in return from the digital assistant 106. The conversation can include one or more of the inputs 110 and the responses 112. Via these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond to the user with appropriate responses.

[0051] User input 110 is usually in the form of natural language and is referred to as utterance. User utterance 110 can be in text form, such as when a user types a sentence, question, text snippet, or even a single word and provides the text as input to the digital assistant 106. In some embodiments, user utterance 110 can be in the form of audio input or speech, such as when a user speaks or says something as input to the digital assistant 106. The utterance is usually in the form of a language spoken by the user 108. For example, the utterance can be English or some other language. When the utterance is in the form of speech, the speech input is converted into a textual utterance in the particular language, and the text utterance is then processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert the speech or audio input into a textual utterance, which is then processed by the digital assistant 106. In some embodiments, the conversion of speech to text can be done by the digital assistant 106 itself.

[0052] An utterance (which may be a text utterance or a voice utterance) may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the above types, and the like. The digital assistant 106 is configured to apply natural language understanding (NLU) technology to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing for understanding the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that these utterances are text utterances that have been directly provided by the user 108 of the digital assistant 106, or are the result of converting the input voice utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0053] For example, the input of user 108 can request to order pizza by providing an utterance such as "I want to order a pizza". After receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate actions. Appropriate actions may involve, for example, responding to the user with questions requesting user input regarding the type of pizza that the user desires to order, the size of the pizza, any toppings of the pizza, etc. The responses provided by the digital assistant 106 may also be in natural language form, and are typically in the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). In order for the user to order pizza via a conversation between the user and the digital assistant 106, the digital assistant may guide the user to provide all necessary information for pizza ordering, and then order the pizza at the end of the conversation. The digital assistant 106 may end the conversation by outputting information indicating that the pizza has been ordered to the user.

[0054] At a conceptual level, the digital assistant 106 performs various processing in response to utterances received from the user. In some embodiments, the processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to be performed in response to the utterance, causing the action to be performed where appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing can include grammatical analysis of the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better understandable form (e.g., logical form) or structure of the utterance. Generating a response can include using NLG technology.

[0055] The NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related processing such as sentence grammatical analysis (e.g., tokenization, classification by inflectional form, identification of part-of-speech tags of sentences, identification of named entities in sentences, generation of dependency trees to represent sentence structure, division of sentences into clauses, analysis of separate clauses, resolution of references, execution of chunks, etc.). In some embodiments, the NLU processing or part thereof is performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 may use other resources to perform part of the NLU processing. For example, the syntax and structure of the input utterance sentence can be identified by processing the sentence using a grammatical analyzer, a part-of-speech tagger, and / or a named entity recognizer. In one embodiment, for the English language, a grammatical analyzer, a part-of-speech tagger, and a named entity recognizer provided by the Stanford Natural Language Processing (NLP) group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0056] Although the various examples provided in this disclosure show utterances in the English language, this is meant only as an example. In some embodiments, the digital assistant 106 is also capable of handling utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that may be called from an NLU core server using service calls. This makes NLU processing flexible and extensible for each language, including allowing different processing orders. Language packs may be provided for separate languages, wherein the language pack may register a list of subsystems that may provide services from an NLU core server.

[0057] The digital assistant (e.g., a user) may be enabled through a variety of different channels, such as, but not limited to, via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. Figure 1 106) is available or accessible to its user 108. A single digital assistant can configure several channels for itself, so that a single digital assistant can run on and be accessed through different services at the same time.

[0058] A digital assistant or chatbot system typically contains or is associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill bots) that are configured to interact with a user and complete specific types of tasks (such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc.). For example, Figure 1 In the depicted embodiment, the digital assistant or chatbot system 106 includes skills 116 - 1 , 116 - 2 , etc. For purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skill bot” and “skill bots,” respectively.

[0059] Each skill associated with the digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, where the conversation can include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses can be in the form of text or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for selection.

[0060] There are various ways to associate skills or skill robots with digital assistants or add them to digital assistants. In some instances, skill robots can be developed by enterprises and then added to digital assistants using DABP 102. In other instances, skill robots can be developed and created using DABP 102 and then added to digital assistants created using DABP 102. In yet other instances, DABP 102 provides an online digital store (referred to as a "skill store") that provides multiple skills related to a variety of tasks. The skills provided by the skill store can also expose various cloud services. In order to add skills to a digital assistant generated using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skills, and indicate that the selected skills will be added to the digital assistant created using DABP 102. Skills from the skill store can be added to the digital assistant as is or in modified form (e.g., a user of DABP 102 can select and copy a particular skill bot provided by the skill store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0061] A digital assistant or chatbot system may be implemented using a variety of different architectures. For example, in some embodiments, a digital assistant created and deployed using DABP 102 may be implemented using a master robot / secondary (or child) robot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a master robot interacting with one or more secondary robots that are skill robots. For example, in Figure 1 In the depicted embodiment, the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are secondary robots to the master robot 114. In some embodiments, the digital assistant 106 itself is considered to act as a master robot.

[0062] A digital assistant implemented according to the master-sub robot architecture enables users of the digital assistant to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user interacts with the digital assistant, the master robot receives the user input. The master robot then performs processing to determine the meaning of the user input utterance. The master robot then determines whether the task requested by the user in the utterance can be handled by the master robot itself, otherwise the master robot selects an appropriate skill robot to handle the user request and routes the conversation to the selected skill robot. This enables users to have a conversation with the digital assistant through a public single interface and still have the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot of the digital assistant can be interfaced with a skill robot with a specific function (such as a CRM robot for performing functions related to customer relationship management (CRM), an ERP robot for performing functions related to enterprise resource planning (ERP), an HCM robot for performing functions related to human capital management (HCM), etc.). In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a public master robot interface, while multiple skill robots are provided in the background for handling user requests.

[0063] In some embodiments, in the master robot / secondary robot infrastructure, the master robot is configured to understand the list of available skill robots. The master robot can access metadata identifying various available skill robots, and for each skill robot, access the capabilities of the skill robot including tasks that can be performed by the skill robot. After receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot that can best serve or handle the user request from multiple available skill robots. The master robot then routes the utterance (or a portion of the utterance) to the specific skill robot for further handling. Thus, control flows from the master robot to the skill robot. The master robot can support multiple input channels and output channels. In some embodiments, routing can be performed with the aid of processing performed by one or more available skill robots. For example, as discussed below, a skill robot can be trained to infer the intent of an utterance and determine whether the inferred intent matches the intent configured for the skill robot. Therefore, routing performed by the master robot can involve the skill robot transmitting an indication to the master robot indicating whether the skill robot has been configured with an intent suitable for handling the utterance.

[0064] Although Figure 1The embodiment of the digital assistant 106 is shown to include a master robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an embodiment using a combination of software and hardware.

[0065] DABP 102 provides an infrastructure and various services and features that enable users of DABP 102 to create digital assistants (including one or more skill robots associated with the digital assistant). In some instances, skill robots can be created by cloning existing skill robots, for example, cloning skill robots provided by a skill store. As previously indicated, DABP 102 provides a skill store or skill catalog that provides multiple skill robots for performing various tasks. Users of DABP 102 can clone skill robots from the skill store. The cloned skill robots can be modified or customized as needed. In some other instances, users of DABP 102 create skill robots from scratch using the tools and services provided by DABP 102. As previously indicated, the skill store or skill catalog provided by DABP 102 can provide multiple skill robots for performing various tasks.

[0066] In some embodiments, at a high level, creating or customizing a skill bot involves the following steps:

[0067] (1) Configuring settings for new skill robots

[0068] (2) Configure one or more intents for the skill robot

[0069] (3) Configure one or more entities for one or more intents

[0070] (4) Training Skill Robots

[0071] (5) Create a dialogue flow for the skill bot

[0072] (6) Add custom parts to skill robots as needed

[0073] (7) Test and deploy skill robots

[0074] Each of these steps is briefly described below.

[0075] (1) Configuring settings for a new skill bot - Various settings can be configured for a skill bot. For example, a skill bot designer can specify one or more invocation names for the skill bot being created. Users of the digital assistant can then use these invocation names to explicitly invoke the skill bot. For example, a user can enter an invocation name in the user's utterance to explicitly invoke the corresponding skill bot.

[0076] (2) Configuring one or more intents and associated example utterances for the skill robot - The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classifications that the skill robot is trained to infer for the input utterance. After receiving the utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a set of predefined intents used to train the skill robot. The skill robot then takes appropriate actions in response to the utterance based on the intent inferred for the utterance. In some instances, the intent of the skill robot represents the tasks that the skill robot can perform for the user of the digital assistant. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot may include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0077] For each intent defined for the skill robot, the skill robot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent the utterances that a user can enter into the skill robot for that intent. For example, for the CheckBalance intent, example utterances can include "What's mysavings account balance?", "How much is in mychecking account?", "How much money do I have inmy account?", etc. Therefore, various arrangements of typical user utterances can be specified as example utterances for an intent.

[0078] These intents and their associated example utterances are used as training data for training the skill robot. A variety of different training techniques can be used. As a result of the training, a prediction model is generated that is configured to take an utterance as input and output the intent inferred by the prediction model for the utterance. In some instances, the input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer the intent of the input utterance. The skill robot can then take one or more actions based on the inferred intent.

[0079] (3) Configuring entities for one or more intents of a skill robot - In some instances, additional context may be required to enable a skill robot to respond appropriately to user utterances. For example, there may be situations where user input utterances resolve to the same intent in a skill robot. For example, in the above example, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests for different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill robot, an entity called AccountType (which defines values ​​called "checking" and "saving") can enable the skill robot to parse the user request and respond appropriately. In the above example, although these utterances resolve to the same intent, the values ​​associated with the AccountType entity for the two utterances are different. This enables the skill robot to perform potentially different actions for the two utterances, even though the two utterances resolve to the same intent. One or more entities can be specified for certain intents configured for a skill robot. Thus, entities are used to add context to the intent itself. Entities help more fully describe intent and enable skill bots to complete user requests.

[0080] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (2) custom entities that can be specified by a skill robot designer. Built-in entities are general entities that can be used with various robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by a skill robot designer to implement various banking transactions by checking user input keywords such as checking, saving, and credit card, etc.

[0081] (4) Training a Skill Robot - A skill robot is configured to receive user input in the form of speech, parse or otherwise process the received input and identify or select intents associated with the received user input. As indicated above, to do this, the skill robot must be trained. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example speech associated with the intents (collectively referred to as training data) so that the skill robot can parse the user input speech into one of the intents configured for it. In some embodiments, the skill robot uses a predictive model that is trained using the training data and allows the skill robot to discern what the user has said (or in some cases, is attempting to say). DABP 102 provides a variety of different training techniques that can be used by skill robot designers to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion (e.g., 80%) of the training data is used to train the skill robot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as a trained skill robot) can be used to handle user speech and respond to user speech. In some cases, the user's utterance may be a question that requires only a single answer and no additional conversation is required. To handle this situation, a Q&A (question answering) intent can be defined for the skill bot. This enables the skill bot to output a reply to the user's request without having to update the dialog definition. A Q&A intent is created in a similar manner to a regular intent. The dialog flow for a Q&A intent may be different from the dialog flow for a regular intent.

[0082] (5) Create a dialog flow for the skill robot - The dialog flow specified for the skill robot describes how the skill robot reacts when parsing different intents of the skill robot in response to received user input. The dialog flow defines the operations or actions that the skill robot will take, for example, how the skill robot responds to user utterances, how the skill robot prompts the user for input, and how the skill robot returns data. The dialog flow is like a flowchart followed by the skill robot. Skill robot designers use languages ​​such as markdown to specify the dialog flow. In some embodiments, a version of YAML called OBotML can be used to specify the dialog flow for the skill robot. The dialog flow definition of the skill robot acts as a model of the conversation itself, which is a model that enables the skill robot designer to choreograph the interaction between the skill robot and the user served by the skill robot.

[0083] In some embodiments, the dialog flow definition of a skill bot includes the following three parts:

[0084] (a) Context

[0085] (b) Default Transformation Section

[0086] (c) Status section

[0087] Context section - The skill bot designer can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, user variables that enable the skill bot to recognize and save user preferences, etc.

[0088] Default Transitions Section - Transitions for a skill bot can be defined in the Dialog Flow State section or in the Default Transitions section. Transitions defined in the Default Transitions section act as fallbacks and are triggered when no applicable transitions are defined within a state or when the conditions required to trigger a state transition cannot be met. The Default Transitions section can be used to define routes that allow a skill bot to gracefully handle unexpected user actions.

[0089] State Part - A dialog flow and its associated operations are defined as a sequence of transient states that govern the logic within the dialog flow. Each state node within a dialog flow definition names a component that provides the functionality required at that point in the dialog. States are thus built around components. States contain properties specific to a component and define transitions to other states that are triggered after the component executes.

[0090] Special case scenarios can be handled using the state section. For example, you may sometimes want to provide the user with the option to temporarily let the first skill with which the user is engaged do something in a second skill within the digital assistant. For example, if the user is busy in a conversation with a shopping skill (e.g., the user has made some purchase choices), the user may want to jump to the banking skill (e.g., the user may want to make sure he / she has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in the first skill can be configured to initiate an interaction with a second, different skill in the same digital assistant and then return to the original flow.

[0091] (6) Adding custom components to a skill bot - As described above, the states specified in the skill bot's dialog flow name the components that provide the desired functionality corresponding to the states. Components enable the skill bot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured components for performing a variety of functions. The skill bot designer can select one or more of these pre-configured components and associate them with states in the skill bot's dialog flow. The skill bot designer can also use the tools provided by DABP 102 to create custom or new components and associate the custom components with one or more states in the skill bot's dialog flow.

[0092] (7) Testing and deploying skill robots - DABP 102 provides several features that enable skill robot designers to test the skill robots they are developing. The skill robots can then be deployed and included in the digital assistant.

[0093] Although the above description describes how to create a skill robot, similar techniques can also be used to create a digital assistant (or master robot). At the master robot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master robot) can handle without invoking the skill robot associated with the digital assistant. Examples of system intents defined for the master robot include: (1) Exit: Applicable when the user signals that they want to exit the current session or context in the digital assistant; (2) Help: Applicable when the user requests help or directions; and (3) UnresolvedIntent: Applicable to user input that does not quite match the exit intent and help intent. The digital assistant also stores information about one or more skill robots associated with the digital assistant. This information enables the master robot to select a specific skill robot for handling the utterance.

[0094] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing for determining how to route the utterance and the associated conversation. The digital assistant determines this using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance is to be routed to a specific skill for handling, to be handled by the digital assistant or master robot itself according to built-in system intents, or to be handled as a different state in the current conversation flow.

[0095] In some embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill robot using its invocation name. If the invocation name is present in the user input, the invocation name is considered to be an explicit invocation of the skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill robot for further processing. In some embodiments, if there is no specific invocation or explicit invocation, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and skill robot associated with the digital assistant. The score calculated for the skill robot or system intent indicates how likely it is that the user input represents the task that the skill robot is configured to perform or represents the system intent. Any system intent or skill robot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill robot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill robots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Any intent with a confidence score exceeding a threshold (e.g., 70%) is typically considered a candidate intent. If a specific skill robot is selected, the user utterance is routed to that skill robot for further processing. If a system intent is selected, one or more actions are performed by the master robot itself based on the selected system intent.

[0096] Figure 2 2 is a simplified block diagram of a master robot (MB) system 200 according to some embodiments. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multi-intention subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data storage 250. Figure 2 The MB system 200 depicted in FIG. 1 is merely an example of a component arrangement in a master robot. A person skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have more than Figure 2 More or fewer systems or components than those shown, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0097] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language grammar analyzer 214. As indicated above, the utterance may be provided in various ways including audio or text. The utterance 202 may be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 may include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text.

[0098] Language detector 212 detects the language of utterance 202 based on the text of utterance 202. How utterance 202 is treated depends on the language, as each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of an utterance.

[0099] The language parser 214 performs grammatical analysis on the utterance 202 to extract part-of-speech (POS) tags for various language units (e.g., words) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 may also tokenize the language units of the utterance 202 (e.g., convert each word into a separate token) and classify the words by inflected form. A lemma is the primary form of a group of words as represented in a dictionary (e.g., "run" is a lemma of run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 may perform include chunking of compound expressions, e.g., combining "credit" and "card" into a single expression "credit_card". The language parser 214 may also identify relationships between words in the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a particular noun) is a direct object, which part of the utterance is a preposition, etc. The results of the processing performed by the language parser 214 form the extracted information 205 and are provided as input to the MIS 220 along with the utterance 202 itself.

[0100] As indicated above, the utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, the utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in some embodiments, pre-processing may be performed, for example by the pre-processing subsystem 210, to identify a single sentence among the multiple sentences for multiple intent analysis and explicit call analysis. In general, the results produced by the MIS 220 and the EIS 230 are substantially the same regardless of whether the utterance 202 is processed at the level of a single sentence or as a single unit including multiple sentences.

[0101] MIS 220 determines whether utterance 202 indicates multiple intents. Although MIS 220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS 220 does not involve determining whether the intent of utterance 202 matches any intent that has been configured for the robot. Instead, the processing for determining whether the intent of utterance 202 matches the robot's intent can be performed by intent classifier 242 of MB system 200 or an intent classifier of a skill robot (e.g., Figure 3 The processing performed by MIS 220 assumes that there is a robot (e.g., a specific skill robot or the master robot itself) that can handle the utterance 202. Therefore, the processing performed by MIS 220 does not need to know which robots are in the chatbot system (e.g., the identity of the skill robots registered with the master robot) or what intents have been configured for a specific robot.

[0102] To determine that the utterance 202 includes multiple intents, the MIS 220 applies one or more rules from a set of rules 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and may include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern may include a coordinating conjunction that connects two parts of a sentence (e.g., a conjunction), where the two parts correspond to different intents. If the utterance 202 matches the sentence pattern, it can be inferred that the utterance 202 represents multiple intents. It should be noted that an utterance with multiple intents does not necessarily have different intents (e.g., intents involving different robots or different intents within the same robot). Instead, an utterance can have different instances of the same intent, such as "Place a pizza order using payment account X, then place apizza orderusing payment account Y".

[0103] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines which parts of utterance 202 are associated with each intent. MIS 220 constructs a new utterance for separate processing to replace the original utterance for each intent represented in the utterance containing multiple intents, for example, Figure 2, utterance "B" 206 and utterance "C" 208 depicted in . Thus, the original utterance 202 can be split into two or more separate utterances that are handled one at a time. MIS 220 uses the extracted information 205 and / or from an analysis of the utterance 202 itself to determine which of the two or more utterances should be handled first. For example, MIS 220 can determine that utterance 202 contains a marker word indicating that a particular intent should be handled first. The newly formed utterance corresponding to the particular intent (e.g., one of utterance 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (or has been temporarily paused), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) can then be sent to EIS 230 for processing.

[0104] EIS230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) contains a call name for a skill robot. In some embodiments, each skill robot in the chatbot system is assigned a unique call name that distinguishes the skill robot from other skill robots in the chatbot system. The list of call names can be saved in the data store 250 as part of the skill robot information 254. When the utterance contains a word that matches the call name, the utterance is considered an explicit call. If the robot is not explicitly called, the utterance received by EIS230 is considered a non-explicit call utterance 234 and is input to the intent classifier of the main robot (e.g., intent classifier 242) to determine which robot to use to handle the utterance. In some instances, the intent classifier 242 will determine that the main robot should handle the non-explicit call utterance. In other instances, the intent classifier 242 will determine the skill robot to which the utterance is routed for handling.

[0105] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that the master robot has to perform. For example, when there is an explicit call, the master robot may not have to perform any intent classification analysis (e.g., using intent classifier 242), or may have to perform a simplified intent classification analysis to select a skill robot. Thus, the explicit call analysis can enable the selection of a particular skill robot without resorting to intent classification analysis.

[0106] Furthermore, there may be cases where there is functional overlap between multiple skill robots. This may happen, for example, if the intents handled by two skill robots overlap or are very close to each other. In this case, it may be difficult for the master robot to identify which of the multiple skill robots to select based solely on the intent classification analysis. In this scenario, explicit calls make it clear that there is no ambiguity as to the specific skill robot to use.

[0107] In addition to determining that the utterance is an explicit call, EIS230 is also responsible for determining whether any part of the utterance should be used as input to the skill robot that is explicitly called. In particular, EIS230 can determine whether a part of the utterance is irrelevant to the call. EIS230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. EIS230 can send the part of the utterance that is irrelevant to the call to the called skill robot instead of sending the entire utterance received by EIS230. In some instances, the input of the called skill robot is simply formed by deleting any part of the utterance associated with the call. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order a pizza" because "using PizzaBot" is related to the call of the pizza robot, but is not related to any processing to be performed by the pizza robot. In some instances, EIS230 can reformat the part to be sent to the called robot, for example to form a complete sentence. Thus, EIS 230 not only determines that there is an explicit call, but also determines what to send to the skill robot when there is an explicit call. In some instances, there may not be any text that can be entered into the robot being called. For example, if the utterance is "Pizza Bot", EIS 230 may determine that the pizza robot is being called, but there is no text to be processed by the pizza robot. In this scenario, EIS 230 may indicate to skill robot invoker 240 that there is nothing to send.

[0108] The skill robot caller 240 calls a skill robot in various ways. For example, the skill robot caller 240 may call a robot in response to receiving an indication 235 that a particular skill robot has been selected as a result of an explicit call. The indication 235 may be sent by the EIS 230 along with the input for the skill robot that was explicitly called. In this scenario, the skill robot caller 240 will hand over control of the session to the skill robot that was explicitly called. The skill robot that was explicitly called will determine an appropriate response to the input from the EIS 230 by treating it as an independent utterance. For example, the response may be to perform a specific action or to start a new session in a specific state, where the initial state of the new session depends on the input sent from the EIS 230.

[0109] Another way that the skill robot invoker 240 can invoke a skill robot is implicitly by using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different categories, one category for each skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. Parameters generated as a result of this training (e.g., a set of parameter values ​​for a machine learning model) can be stored as part of the skill robot information 254.

[0110] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training of the machine learning model can involve inputting at least a subset of utterances from example utterances associated with various skill robots to generate, as output of the machine learning model, inferences about which robot is the correct robot for handling any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., by backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0111] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot that indicates the likelihood that the skill robot can handle an utterance (e.g., a non-explicit call utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each system-level intent (e.g., help, exit) that has been configured. If a particular confidence score satisfies one or more conditions, the skill robot caller 240 will call the robot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must also exceed the next highest confidence score by a certain margin. When the confidence scores of multiple skill robots all exceed the threshold confidence score value, applying such a condition will enable routing to a particular skill robot.

[0112] After identifying the robot based on the evaluation of the confidence score, the skill robot invoker 240 hands over the processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is a skill robot. Further, the skill robot invoker 240 will determine what to provide as input 247 to the identified robot. As indicated above, in the case of an explicit invocation, the input 247 can be based on a portion of the utterance that is not relevant to the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invocation, the input 247 can be the entire utterance.

[0113] The data store 250 includes one or more computing devices that store data used by various subsystems of the master robot system 200. As explained above, the data store 250 includes rules 252 and skill robot information 254. The rules 252 include, for example, rules for determining by the MIS 220 when an utterance represents multiple intents and how to split utterances representing multiple intents. The rules 252 further include rules for determining by the EIS 230 which parts of an utterance that explicitly calls a skill robot are to be sent to the skill robot. The skill robot information 254 includes the calling names of the skill robots in the chatbot system, for example, a list of the calling names of all skill robots registered with a particular master robot. The skill robot information 254 may also include information used by the intent classifier 242 to determine the confidence score of each skill robot in the chatbot system, for example, parameters of a machine learning model.

[0114] Figure 3 is a simplified block diagram of a skill robot system 300 according to some embodiments. The skill robot system 300 is a computing system that can be implemented in software only, hardware only, or a combination of hardware and software. In some embodiments, as Figure 1 In the depicted embodiment, the skill bot system 300 can be used to implement one or more skill bots within a digital assistant.

[0115] The skill robot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to Figure 2 The MIS 220 in FIG. 310 and provides similar functionality, including being operable to use rules 352 in data store 350 to determine: (1) whether an utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules for detecting multiple intents and for splitting utterances applied by MIS 310 are the same as the rules applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to Figure 1The extracted information 205 in the skill robot system 300 may be generated using the language parser 214 or a language parser local to the skill robot system 300.

[0116] The intent classifier 320 can be combined with the above Figure 2 In some embodiments, the intent classifier 320 is trained in a similar manner to the intent classifier 242 discussed in the embodiments of the present invention and described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. For a specific skill robot, at least a subset of example utterances associated with the specific skill robot is used as training utterances to train the machine learning model of the intent classifier 320. The ground truth of each training utterance will be the specific robot intent associated with the training utterance.

[0117] The speech 302 may be received directly from the user or provided by the main robot. When the speech 302 is provided by the main robot, for example, as Figure 2 As a result of the processing by MIS 220 and EIS 230 in the embodiment depicted in , MIS 310 may be bypassed to avoid duplicating processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a conversation that occurs after being routed to a skill robot, MIS 310 may process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0118] The intent classifier 320 is configured to match a received utterance (e.g., utterance 306 or 308) with an intent associated with the skill robot system 300. As explained above, a skill robot may be configured with one or more intents, each of which includes at least one example utterance associated with the intent and used to train the classifier. Figure 2In an embodiment, the intent classifier 242 of the master robot system 200 is trained to determine confidence scores for individual skill robots and confidence scores for system intents. Similarly, the intent classifier 320 can be trained to determine confidence scores for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and is therefore more fine-grained. The intent classifier 320 can access intent information 354. For each intent associated with the skill robot system 300, the intent information 354 includes a list of utterances that represent the intent and explain the meaning of the intent and are typically associated with tasks that can be performed by the intent. The intent information 354 can further include parameters generated as a result of training on the list of utterances.

[0119] The conversation manager 330 receives, as an output of the intent classifier 320, an indication 322 of a particular intent that is identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some instances, the intent classifier 320 is unable to determine any match. For example, if the utterance involves a system intent or an intent of a different skill robot, the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill robot system 300 may submit the utterance to the master robot for disposal, e.g., for routing to a different skill robot. However, if the intent classifier 320 successfully identifies the intent within the skill robot, the conversation manager 330 will initiate a conversation with the user.

[0120] The session initiated by the session manager 330 is a session specific to the intent identified by the intent classifier 320. For example, the session manager 330 can be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine can include a default starting state (e.g., when the intent is invoked without any additional input) and one or more additional states, wherein each state is associated with an action to be performed by the skill robot (e.g., perform a purchase transaction) and / or a dialog to be presented to the user (e.g., a question, a response). Thus, the session manager 330 can determine an action / dialog 335 upon receiving an indication 322 that an intent is recognized, and can determine additional actions or dialogs in response to subsequent utterances received during the session.

[0121] The data storage 350 includes one or more computing devices that store data used by the various subsystems of the skill robot system 300. Figure 3 As depicted, data store 350 includes rules 352 and intent information 354. In some embodiments, data store 350 may be integrated into the data store of a host robot or digital assistant, e.g. Figure 2The data is stored in 250.

[0122] Confidence score calibration

[0123] Confidence calibration is defined as the ability of a model to provide an accurate probability of correctness for any of its predictions. For example, suppose is the model prediction output (in the OMRL logic form in the case of C2OMRL) and is the associated confidence score. Represents the true probability of the model being correct. In other words, if the model makes 100 predictions with the same confidence level of 0.9, then 90 predictions are expected to be correct (out of the 100 predictions). A poorly calibrated model has “underconfidence” if the accuracy exceeds the confidence level, or “overconfidence” if the model accuracy is less than the confidence level. In this case, the model’s confidence score is well calibrated. Such calibrated confidence scores have historically been important in various “high-risk” applications (e.g., self-driving cars, medical diagnostics, etc.) where incorrect predictions can be extremely problematic, but have become important for other applications such as data analytics and customer satisfaction as well, as the calibrated probability score associated with each prediction allows low-quality predictions to be identified and discarded. Therefore, even if the output of a machine learning model cannot yet be fully explained, confidence calibration provides a practical way to avoid errors in practice by associating an accurate uncertainty / confidence score with each prediction.

[0124] Although confidence calibration is a useful property, it is not directly measurable in contrast to specific performance metrics such as accuracy or loss. Therefore, various metrics for confidence calibration have been proposed over time, each with its own pros and cons. For example, the Brier score is an appropriate scoring rule that measures the squared error between the predicted probability vector and the one-hot encoded true label. Lower scores correspond to more accurate calibration. While the Brier score is a good metric for measuring the calibration of network predictions, it is not sensitive to the probabilities associated with infrequent events. Therefore, using multiple metrics in addition to the Brier score to measure calibration often provides useful insights. The negative log-likelihood is also an appropriate scoring rule that can be used to assess model uncertainty on held-out data, where lower scores correspond to better calibration. Although commonly used as an objective function for training, the negative log-likelihood characterizes the difference between the predicted confidence of the true label and the actual confidence, reaching a perfect score of zero when all data are correctly predicted with 100% confidence. However, the negative log-likelihood overemphasizes the tail probabilities, which means that a slight deviation from the correct fully confident prediction can lead to a large increase in the negative log-likelihood.

[0125] The expected calibration error (ECE) calculates the expected difference between confidence and accuracy. This expected difference can be calculated as shown below.

[0126]

[0127] Category Prediction

[0128] Confidence prediction

[0129] Y True confidence

[0130] p True Accuracy

[0131] Although the above expression cannot be evaluated in closed form, it can be approximated by partitioning the model predictions into separate bins based on their associated confidence scores. The difference between the average confidence and accuracy can then be calculated within each bin. Thereafter, ECE calculates the difference between the average confidence and accuracy within each bin and takes a weighted average of these values ​​based on the relative size of each bin.

[0132] The maximum calibration error (MCE) follows this same process, but is equal to the maximum difference between the average confidence and accuracy across bins. For both ECE and MCE, lower scores correspond to better calibrated predictions. Neither ECE nor MCE is an appropriate scoring rule, which means that there is a trivial solution with the best calibration error (e.g., predicting a uniform random probability). In addition, due to the binning procedure, neither of these indicators decreases monotonically with the improvement of the prediction. Nevertheless, these indicators can be used because they can provide simple and interpretable estimates of model calibration. MCE is often used in applications where a reliable confidence measure is absolutely required (i.e., large errors in calibration are harmful), while ECE provides a more comprehensive average indicator across bins.

[0133] If the user is interested in obtaining a graph that characterizes the calibration error rather than a scalar metric such as ECE or MCE, the per-bin accuracy and confidence measures can be converted into a reliability graph. A reliability graph plots accuracy on the y-axis and average confidence on the x-axis. The average confidence measure and the accuracy measure within each bin are plotted on the graph, forming a line graph. Perfect calibration would produce a diagonal line on the reliability graph where the confidence is equal to the accuracy within each individual bin. For example, perfect calibration means:

[0134]

[0135] The main idea of ​​a reliability plot is to plot the observed model accuracy against the expected confidence level p, i.e.,

[0136]

[0137] For finite samples, this is achieved by grouping the predictions into M bins of equal size 1 / M. Consider the confidence level falling into the interval Prediction warehouse B M , and calculates its observed accuracy by:

[0138]

[0139] And its expected confidence is calculated as follows:

[0140]

[0141] If the model is well calibrated, its observed accuracy equals its expected confidence, that is,

[0142] acc(B M )=conf(B M )

[0143] For all warehouses B M .

[0144] Figure 4A and Figure 4B An example reliability plot of a C2OMRL model calibrated according to the calibration technique disclosed herein is shown. The calibration results are measured in ECE (the average of the gap bars in the reality plot). The desired calibration target is ECE < 0.1. Figure 4A In , ECE = 0.29, and thus the C2OMRL model does not meet the calibration target. Figure 4B , ECE = 0.05, and thus the C2OMRL model does meet the calibration target.

[0145] The x-axis represents the bin position B M .

[0146] Black bars: Plotting the observed accuracy acc(B M ).

[0147] Gray bar: plotting the expected confidence conf (B M ).

[0148] The top of the grey bar should follow the identity function (dashed diagonal line).

[0149] • Therefore, the gap bar (ie, the grey box shown) is the difference between the expected confidence and the observed accuracy.

[0150] · Figure 4AThe reliability plot in illustrates a poorly calibrated model, where the bars vary in size between different confidence values. The confidence in this model is too high.

[0151] · Figure 4B The reliability plot in illustrates an overall well-calibrated model, where the gap bars are relatively small, i.e., acc(B M )≈conf(B M ).

[0152] ·exist Figure 4A and Figure 4B In the case where the black bars cover the gray bars, the observed accuracy acc(B M )>Expected confidence conf(B M ). The model does not have enough confidence. \

[0153] Although reliability plots can be used to visualize how calibrated a model's confidence scores are, scalar metrics such as ECE or MCE are also important for quantitatively comparing different models and calibration methods. Calibrated scalar metrics allow reliability plots to be compressed into a single number for comparison. ECE can be defined as:

[0154]

[0155] Among them, B M is the bin position, acc(B M ) is the observed accuracy, conf(B M ) is the expected confidence, and n is the total number of predictions. The ECE can therefore be interpreted as the average of the gap bars in the reliability plot.

[0156] Several conventional methods have been proposed to improve deep learning model (e.g., neural network) confidence calibration (e.g., temperature scaling, Pratt scaling, etc.). Some of these methods are simple processes that can be added to the training process, while others require a post-processing stage that retains a validation set or even requires significant changes to the network architecture. However, existing calibration methods cannot be applied to NL2LF because they are only applicable to classification models, for example, Pratt scaling is the preferred calibration method for binary classifiers, while temperature scaling is the standard method for calibrating multi-class classifiers. This is because these methods directly derive confidence scores from the final "activation layer" of the classification model (e.g., "Softmax" for multi-class and "S-shaped" for binary classification).

[0157] Calibration of deep learning based semantic parsers (NL2LF models belong to this category) has hardly been explored. The only related work on deep learning based semantic parsers is described in "Confidence Modeling for Neural Semantic Parsing" by Dong, Li et al., Annual Conference of the Association for Computational Linguistics (2018). This paper explores confidence by identifying the most "uncertain" part of the output generated by the model and then "tracing back" it to a specific part of the model input. However, the described method is specific only to a specific encoder-decoder architecture (LSTM, where the encoder and decoder are two different recurrent neural networks with long short-term memory units), and therefore cannot be generally applied to most NL2LF models, such as the C2OMRL model described in this paper. To address these and other deficiencies, this paper describes calibration techniques that utilize the intermediate "raw" scores generated at each step in the decoding process of the NL2LF model and combine them to obtain a calibrated confidence score.

[0158] An overview of systems for converting natural language into meaning-representing language

[0159] Figure 5 The C2OMRL system 500 is shown to be powered by a machine learning model to be able to translate natural language (NL) utterances (e.g., Figures 1 to 3 The C2OMRL model 510 is a machine learning model that converts natural language utterances (such as utterances within the described digital assistant platform) into logical form (LF) statements (such as OMRL queries or commands), which in turn can be executed to query existing systems (such as relational databases). The machine learning model (referred to herein as the "C2OMRL semantic parser" or "C2OMRL model") is trained on hundreds to thousands of annotated example pairs (natural language and logical form pairs) for converting natural language utterances into logical form statements. As shown, example 505 (a concatenation of natural language utterances and database schemas, such as a sequence of table names and column names) is input into the C2OMRL model 510. Example 505 is first processed by an encoder 515, which captures the representation of natural language utterances and database schemas according to the context. Then, a decoder 520 receives the encoded input and predicts a logical form 525 (e.g., OMRL, which is a SQL-like query) based on the captured representation of the natural language utterances and database schema.

[0160] In the C2OMRL model 510, the encoder 515 includes two main components: (1) a pre-trained language model (PLM) 530; and (2) a relation-aware transformer (RAT) 535. The PLM 530 is used to embed natural language utterances and database schemas because it captures representations of natural language utterances and database schemas according to context. In some instances, a transformer-based PLM called Decoding Enhanced BERT with Disentangled Attention (DeBERTa) is used as the PLM 530. Transformer-based PLMs use self-supervised learning to learn general language representations from large amounts of text data and pass this knowledge to downstream tasks. These models provide good background knowledge for downstream tasks, avoiding the need to train downstream models from scratch. The RAT 535 encodes the relationships between entities in the database schema and words in the natural language utterances (these relationships are called "schema link" relationships).

[0161] Decoder 520 is based on a bottom-up generation process (i.e., the bottom-up generation process generates a tree from left to right), where the final generation output is an OMRL tree (i.e., a tree-based structure representing the complete OMRL logical form) that can be mapped to the final OMRL logical form 525. The bottom-up generation process is implemented using beam search, which is a heuristic search algorithm that explores a graph by expanding the most promising nodes in a finite set. Figure 6 As shown, the beam search 600 is performed in steps (e.g., about 10 steps), also referred to as "beam levels". At each step (e.g., "step i"), the beam search algorithm generates F possible subtrees for the input sequence that can be obtained by expanding the current subtree (from step "i-1"), and then selects the top K subtrees (referred to as the beam width) to retain using the conditional probability associated with each subtree. The conditional probability is referred to as the "original beam score" in this article, so the top K intermediate results (which will be taken into account in the next generation step) are the K intermediate results with the highest original beam score. For more information on the bottom-up generation process, see Ohad Rubin and Jonathan Berant, "SmBoP: Semi-autoregressive Bottom-up Semantic Parsing", 2021. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 311-324, online. Association for Computational Linguistics, the entire contents of which are incorporated herein by reference for all purposes. The output of the final decoder 520 is the subtree with the highest original cluster score at the last step N.

[0162] For example, in the first step (bunch level 1), the encoded input utterance and database pattern are input to the decoder 520, and the decoder 520 applies a softmax function to all tokens in the vocabulary or grammar to find the best alternative for the first subtree (e.g., the first token or node of the tree). In order to generate F possible subtrees (referred to as the frontier), the decoder 520 makes predictions that represent the conditional probability of each token in the vocabulary or grammar appearing next in the sequence (the possible value of yi+1 under the condition of the previous token y1, ..., yi and the context variable c (generated by the encoder to represent the input sequence)). The vocabulary or grammar is obtained from a corpus of words or terms including the target logical form (e.g., OMRL). In some instances, the corpus further includes rules for the words or terms of the target logical form. These rules define how words or terms can be used to create appropriate phrases or operations in the target logical form (e.g., term combinations that are used together for appropriate OMRL queries). The beam search algorithm then selects the top K subtrees with the highest conditional probabilities or original bundle scores as the most likely choices for the time step. In this example, it is assumed that the top K subtrees or cluster width is 2, and the subtrees with the highest conditional probability P(y1|c) in the first step are sub-tree_1 and sub-tree_12. These top K results can be selectable and / or optimizable hyperparameters. sub-tree_1 and sub-tree_12 and the corresponding conditional probabilities or raw cluster scores are saved in memory.

[0163] In the second step (bunch level 2), the two selected trees (sub-tree_1 and sub-tree_12) from the first step are input to the decoder 520, and the decoder 520 will apply the softmax function to all tokens in the vocabulary or grammar to find the two best alternatives for the second subtree (e.g., the first token or node and the second token or node of the tree). While doing this, the beam search algorithm will use conditional probabilities to determine the combination of the first token or node and the second token or node that is most likely to form a pair or second subtree. In other words, for all y2∈Y, the beam search algorithm calculates P(sub-tree_1,y2|c)=P(sub-tree_1|c)P(y2|sub-tree_1,c), P(sub-tree_12,y2|c)=P(sub-tree_12|c)P(y2|sub-tree_12,c), and selects the largest two values ​​among these values, such as P(sub-tree_22|c) and P(sub-tree_37|c). Sub-tree_22 and sub-tree_37 and the corresponding conditional probabilities or raw beam scores are saved in memory.

[0164] In the third step (clustering level 3), the two selected trees (sub-tree_22 and sub-tree_37) from the second step are input to the decoder 520, and the decoder 520 will apply the softmax function to all tokens in the vocabulary or grammar to find the two best alternatives for the third subtree (e.g., the first token or node, the second token or node, and the third token or node of the tree). While doing this, the beam search algorithm will use conditional probabilities to determine the combination of the first token or node, the second token or node, and the third token or node that is most likely to form a string or a third subtree. In other words, for all y3∈Y, the beam search algorithm calculates P(sub-tree_22,y3|c)=P(sub-tree_22|c)P(y3|sub-tree_22,c), P(sub-tree_37,y3|c)=P(sub-tree_37|c)P(y3|sub-tree_37,c), and selects the top K subtrees. The first K subtrees and the corresponding conditional probabilities or raw cluster scores are saved in memory. This process continues until N cluster levels are completed (this can be an optimized or selected hyperparameter). The final model output is the subtree with the highest conditional probability or raw cluster score at the last step N (cluster level N). The tokens or nodes of this final subtree can then be mapped to the final logical form, such as OMRL logical form statement 525.

[0165] The predicted OMRL logical form statements 525 (i.e., the OMRL tree with the highest original cluster score at the last step N) can then be input into a language converter 540 (e.g., (OMRL2SQL)) to convert the meaning representation language into a system language query or command, such as SQL, API, REST, GraphQL, PGQL, etc. The system language query or command can then be used to query or perform an operation on a system 545 (e.g., a relational database) and obtain an output 550 as a result of the query or command. Although systems and models for natural language to meaning representation language are described herein with respect to C2OMRL, it should be understood that the techniques described herein are applicable to any system and / or model (i.e., any NL2LF model, such as NL2SQL) that converts natural language to meaning representation language or logical form using a bottom-up generation process for a decoder.

[0166] Confidence score calibration system and technology

[0167] The disclosed confidence score calibration systems and techniques utilize a calibration model, an auxiliary model of the C2OMRL model, which takes as input the raw bundle scores of one or more bundle levels of the decoder of the C2OMRL model and outputs a confidence score associated with the C2OMRL output. In various embodiments, the calibration model is a binary classifier, such as a logistic regression binary classifier, which takes as input a vector of one or more raw bundle scores (e.g., the first K raw bundle scores) with one or more bundle levels, and the calibration model uses the information in the vector to predict whether the final result (i.e., the OMRL tree with the highest raw bundle score at the last step N) is correct or incorrect. The probability score from the calibration model (e.g., the output of the sigmoid function in logistic regression) becomes the calibrated C2OMRL confidence score. The theory behind this approach is that the raw bundle scores of subtree candidates can reveal information about model uncertainty in the generation process (e.g., caused by the model's weakness on a certain type of example or caused by ambiguous input), and therefore these raw scores are a good source of information for the calibrated confidence score.

[0168] Figure 7 The confidence score calibration system 700 using the calibration model 705 is shown. The input 710 of the calibration model 705 is obtained from the NL2LF machine learning model (e.g., Figure 5 and Figure 6The decoder 715 of the C2OMRL model 510 described in the embodiment of the present invention obtains the raw bundle scores of one or more bundle levels. The raw bundle scores of one or more bundle levels are preprocessed to generate a vector of raw bundle scores of one or more bundle levels. The vector includes a stack or sequence arrangement of the raw bundle scores. In some instances, the order of the raw bundle scores from the bundle levels is maintained in the vector (i.e., the raw bundle scores from the first bundle level are ranked first in the vector, followed by the raw bundle scores from the second bundle level, followed by the raw bundle scores from the third bundle level, and so on). In some instances, the raw bundle scores of all bundle levels or a subset of the raw bundle scores are vectorized. In some instances, the first K raw bundle scores of all bundle levels are vectorized. In other instances, a subset of the first K raw bundle scores of all bundle levels are vectorized. In some instances, the raw bundle scores of some bundle levels or a subset of the raw bundle scores are vectorized (e.g., the last bundle level N or the last three bundle levels). In some instances, the first K raw beam scores of some beam levels are vectorized. In other instances, a subset of the first K raw beam scores of some beam levels are vectorized. For example, by using only the first K raw beam scores of some beam levels (fewer features available for classification), training efficiency can be improved and processing time can be shortened; while the accuracy of the calibration model can be improved by using more raw beam scores from more beam levels (such as all beam levels) (more features available for classification).

[0169] The calibration model 705 is trained as a binary classifier that takes input 710 and uses the information in the vector to predict whether the final result (i.e., the OMRL tree with the highest original cluster score at the last step N) is correct or incorrect. The class of the correct state can be assigned a class label of 0, and the class with the incorrect state can be assigned a class label of 1. The algorithm trained for the calibration model 705 can be a logistic regression, K nearest neighbor, decision tree, support vector machine, naive Bayes, or a similar algorithm commonly used for binary classification problems. In some instances, the algorithm trained for the calibration model 705 is a logistic regression. Logistic regression is characterized by a logistic function to model the conditional probability of the label Y variable X. In the calibration model 705, Y represents whether the state is correct or incorrect, and X will be a feature of the model uncertainty in the generation process (i.e., the original cluster score). Then, the conditional probability of whether the output of the machine learning model trained to convert natural language into logical form is correct or incorrect given the observation value X can be modeled as a sigmoid function. The sigmoid function maps a continuous variable to a closed set [0,1], which can then be interpreted as a probability. Each data point to the right of the split value is interpreted as y=1, while each data point to the left of the split value is inferred to be y=0.

[0170] The training of the calibration model 705 can be performed before, after, or simultaneously with the training of the NL2LF machine learning model for converting natural language into logical form. In any case, the training of the calibration model 705 does not change any weights / parameters in the NL2LF machine learning model. In other words, the calibration process has no effect on the predicted LF (e.g., OMRL) output of the NL2LF machine learning model. In order to avoid overfitting of the calibration model 705, the calibration model 705 can be trained using a training example set different from the training example set used to train the NL2LF machine learning model. The training example set used to train the calibration model 705 is annotated with a basic fact label to determine whether the prediction of the LF by the NL2LF machine learning model is correct or incorrect given a specific NL utterance. The label can be generated by comparing the LF output from the NL2LF machine learning model with the LF basic facts established for the utterance, and determining whether the LF output is the same or substantially the same as the LF basic facts established for the utterance.

[0171] The calibration model 705 generates two outputs 720: (i) a binary classification indicating whether the subtree with the highest conditional probability or original cluster score at the last step N is correct or incorrect; and (ii) a conditional probability score as to whether the subtree with the highest conditional probability or original cluster score at the last step N is correct or incorrect. In some instances, the conditional probability score is the output of the sigmoid function. The conditional probability score is used as a calibrated confidence score for the NL2LF machine learning model.

[0172] Figure 8 A block diagram is shown illustrating an example system configured for training and deploying a machine learning model (e.g., Figures 5 to 7The present invention provides various aspects of a model system 800 for a machine learning model (NL2LF machine learning model and calibration model described in the present invention). In this example, the model system 800 includes various stages: a training stage 805 for training a machine learning model; an NL2LF conversion stage 810 for converting a natural language text (e.g., an utterance) into a logical form, such as OMRL; a confidence score calibration stage 815 for obtaining a confidence score associated with the logical form output from the NL2LF conversion stage 810; and a query execution stage 820 for executing the logical form on a system such as a database to obtain a result (e.g., an answer to a query within the utterance). The training stage 805 establishes and trains one or more machine learning models 425a-425n ('n' represents any natural number) for use in other stages (these machine learning models may be referred to as models 825 individually or collectively herein). For example, the model 825 may include a first model for converting a natural language utterance into a logical form and a second model for predicting whether the logical form output from the first model is correct or incorrect. Other types of prediction models (e.g., intent classifiers) may be implemented in other examples according to the present disclosure.

[0173] Model 825 can be a deep learning model, such as a convolutional neural network ("CNN"), for example, an initial neural network, a residual neural network ("Resnet"); or a recursive neural network, for example, a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model; other variants of a deep neural network ("DNN") (e.g., a multi-label n-point DNN classifier or a multi-class DNN classifier). Model 825 can also be any other suitable machine learning model trained for classification or natural language processing, understanding or generation, such as a logistic regression classifier, a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, a transformer (such as a bidirectional encoder representation from a transformer (BERT)), or a combination of one or more of these technologies - for example, a CNN-HMM or an MCNN (multi-scale convolutional neural network). The model system 800 can use the same type of model or different types of models to complete various tasks, such as converting natural language into logical form and / or binary classification.

[0174] In some embodiments, model 825 is configured as a semantic parser or NL2LF model, such as RAT-SQL, DuoRAT, Grammar Augmented Pre-Trained Model for Table Semantic Parsing (GRAPPA), RAT-SQL+GRAPPA, RoBERTa, Semi-Autoregressive Bottom-Up Semantic Parsing Model (SMBOP), etc. For more information about the RAT-SQL model, see Wang et al., "RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers," Proceedings of the 58th Annual Conference of the Association for Computational Linguistics, the entire contents of which are incorporated herein by reference for all purposes. For more information about the DuoRAT model, see Scholak et al., "DuoRAT: Towards Simpler Text-to-SQL Models," Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, the entire contents of which are incorporated herein by reference for all purposes. For more information on the SMBOP model, see Ohad Rubin and Jonathan Berant, “SmBoP: Semi-autoregressive Bottom-up SemanticParsing,” 2021. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 311–324, online. Association for Computational Linguistics, which is incorporated herein by reference in its entirety for all purposes.

[0175] To train the various models 825, the training phase 805 consists of two main subsystems or services: a dataset preparer 830 and a model trainer 840. The dataset preparer 830 facilitates the process of loading data assets 845, splitting the data assets 845 into training and validation sets (845-an) so that the system can train and test the models 825, and performing basic natural language preprocessing (e.g., standardizing, normalizing, tokenizing data, annotating, enhancing, embedding, etc.). The data assets 845 include natural language utterances (e.g., natural language questions / requests) obtained from one or more sources, such as from human annotators (e.g., Spider, SParC, and / or CoSQL datasets), databases (not shown), computing systems (e.g., data preprocessing subsystems), etc. In some instances, the utterances are provided by a client or customer. In other instances, the utterances are automatically generated and / or obtained from a repository of utterances (e.g., identifying utterances from the repository that are specific to the task to be learned by the model). Data assets 845 may include input text or audio (or input features of text or audio frames) and labels 850 corresponding to the input text or audio (or input features) as a matrix or value table. For example, for each utterance, the corresponding label may include an indication of the corresponding logical form (e.g., a statement / query, such as an OMRL query) and / or an indication of whether the logical form is correct or incorrect for the utterance provided as the basic factual information of the utterance. The behavior of the model may then be adapted (e.g., by backpropagation) to minimize the difference between the generated inference and the basic factual information. For example, data assets 845 may include a natural language utterance "What is the average life expectancy in the United States of America?" and a label containing the corresponding logical form "SELECT AVG(life_expectancy)FROM country=United States of America" ​​(and in some instances, whether the predicted logical form is "correct" or "incorrect").

[0176] In some instances, data assets 845 also include database schema information. A database schema defines how data is organized within a database such as a relational database; this includes logical constraints such as table names, fields, data types, and relationships between these entities. A relational database may be formed of one or more tables, wherein each of the one or more tables includes one or more columns, wherein each of the one or more columns includes one or more values. Each table and column of a relational database may be named with a unique identifier, each of which may include one or more words. In some instances, one or more columns of a relational database may be used as a primary key, wherein each of the values ​​of the one or more columns used as a primary key is unique to each other. In some instances, one or more columns of a relational database may be used as a foreign key, which is used to link a table including the one or more columns to another table in the relational database. In some instances, the database schema information includes one or more data structures for storing unique identifiers of one or more tables, unique identifiers of one or more columns, and values ​​of each relational database. These unique identifiers and values ​​may be stored in one or more vectors and / or matrices by the data set preparer 830. In some embodiments, the data structure storing the schema information of the relational database may store a directed graph representing unique identifiers and values.

[0177] In some examples, data asset 845 also includes cluster score information. Cluster score information for one or more cluster levels can be obtained from a trained NL2LF machine learning model (e.g., Figure 5 and Figure 6 The dataset preparer 830 preprocesses the raw cluster scores of one or more cluster levels to generate a vector of raw cluster scores of one or more cluster levels. The vector includes a stack or sequence arrangement of the raw cluster scores. The cluster score information can be annotated with a basic fact label to determine whether the prediction of the LF by the trained NL2LF machine learning model is correct or incorrect given a specific NL utterance. The label can be generated by comparing the LF output from the trained NL2LF machine learning model with the LF basic facts established for the utterance, and determining whether the LF output is the same or substantially the same as the LF basic facts established for the utterance.

[0178] Once the data asset 845 is obtained, the data set can be split into a training data set and a validation data set. The split can be performed randomly (e.g., 90% / 10% or 70% / 30%), or the split can be performed according to more complex validation techniques (such as K-fold cross validation, leave one cross validation, leave one group cross validation, nested cross validation, etc.) to minimize sampling bias and overfitting. Before or after the split, basic natural language preprocessing can be performed on the data asset 845. In some instances, the preprocessing includes tokenizing the utterances of the data asset 845. Tokenization is the splitting of phrases, sentences, paragraphs, or entire text documents into smaller units, such as individual words or terms. Each of these smaller units is called a word unit. Smaller units are created by locating boundaries (such as word boundaries), which are the end points of a word and the beginning of the next word. For example, the text "How many employees work for company X (how many employees work for company X)" can be tokenized into 'How', 'many', 'employees', 'work', 'for', 'company', 'X'. These tokens help the model understand the context and develop a model for a given task. There are various tokenization techniques that can be used to perform tokenization based on language and modeling tasks. For example, tokenization can be performed using natural language toolkit, space tokenization, dictionary-based tokenization, rule-based tokenization, Keras tokenization, Penn tree-based tokenization, spaCy tokenization, Moses tokenization, subword tokenization, etc.

[0179] In some instances, the word-grams of data asset 845 can then be embedded into word embeddings (e.g., contextualized word embeddings). Word embeddings are learned representations of text in which words with the same meaning have similar representations. Word embeddings are generated by embedding techniques in which individual words are represented as real-valued vectors in a predefined vector space so that they can be understood by deep learning algorithms. The embedding techniques can be joint or separate embedding techniques, such as including an embedding layer within a deep learning algorithm, or using a separate model, such as a BERT-based pre-trained language model (e.g., BERT, RoBERTa, and DeDERTa). BERT-based models are pre-trained language models that use self-supervised learning to learn the deep meaning of words and context. The embedding layer is a word embedding learned jointly with a neural network model on a specific natural language processing task, such as the conversion of natural language to logical form (e.g., (multiple) natural language-logical form (NL-LF) algorithms). In some instances, other embedding techniques, such as Word2Vec or GloVe, can be used. Word2Vec is a statistical technique that uses models such as the Continuous Bag of Words model or the Continuous Skip-Gram model to learn individual word embeddings from a text corpus. GloVe (which stands for Global Vectors) is a model used to create word embeddings based on global corpus statistics. It is trained on the non-zero entries of the global word-word co-occurrence matrix, which lists the co-occurrence frequencies between words in a given corpus.

[0180] Model training typically includes selecting hyperparameters of model 825 and using an optimization algorithm (e.g., a stochastic gradient descent algorithm or a variant thereof, such as batch gradient descent or mini-batch gradient descent) to find the model parameters that correspond to the best fit between the predicted output and the actual output. Hyperparameters are settings that can be adjusted or optimized to control the behavior of model 825. Most models explicitly define hyperparameters that control different aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined and optimized to adapt the model to specific scenarios. For example, hyperparameters can include the number of hidden units of the model, the learning rate of the model, the convolution kernel width, the number of kernels of the model, the top K results, the N clustering levels, etc.

[0181] During training by the model trainer 840, the error is calculated as the difference between the actual output (e.g., as indicated by the label 850) and the predicted output (e.g., as determined by the model 825). The function used to calculate the error is called the objective function (e.g., loss function or cost function). The error is a function of the internal parameters of the model (e.g., weights and biases). In order to accurately predict, the error needs to be minimized. In order to minimize the error, the model parameters are gradually updated by minimizing the objective function on the training examples from the preprocessed data assets 845. The objective function can be constructed to measure the difference between the output inferred using the model and the basic facts annotated to the sample using the label. For example, for a model based on supervised learning, the goal of training is to learn a function "h()" (sometimes also called a hypothesis function) (h:X→Y) that maps the training input space X to the target value space Y, so that h(x) is a good predictor of the corresponding value y. A variety of different techniques can be used to learn the hypothesis function. In some machine learning algorithms such as neural networks, this is done using backpropagation. The current error is usually propagated back to the previous layer, where it is used to modify the weights and bias values ​​in a way that minimizes the error. An optimization function is used to modify the weights. The optimization function usually calculates the error gradient, that is, the partial derivative of the objective function with respect to the weight, and the weight is modified in the opposite direction of the calculated error gradient. For example, techniques such as backpropagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), Hebbian learning, etc. are used to update the model parameters in a way that minimizes or maximizes the objective function. This cycle is repeated until the minimum value of the objective function is reached.

[0182] In some instances, the model trainer 840 uses the preprocessed data assets 845 (e.g., the data assets after word segmentation) to train (multiple) natural language-logical form (NL2LF) algorithms. In some instances, (multiple) NL2LF algorithms include encoder-decoder neural networks. The encoder consists of an input layer and one or more encoding layers. The one or more encoding layers may include multiple recurrent units, such as long short-term memory (LSTM), wherein each recurrent unit obtains input in the form of a single element of the input sequence, thereby collecting data for the specific element and generating it forward. The encoder follows an embedding procedure to transform the relevant text (and optionally the database pattern) into a digital / vector representation to preserve the conditions and connections between words and sentences, so that the machine can understand the patterns associated with any text, find the context of the sentence, and optionally learn the relationship between words and given database patterns. The result of the encoder will be a state vector or context vector. The state vector will be the input of the decoder. The decoder consists of an input layer, one or more decoding layers, a dense layer, and an output layer (e.g., a layer with a softmax function). One or more decoding layers may include multiple recurrent units, such as LSTMs, in which the output is predicted at each time step, as in Figure 5 and Figure 6 The current recurrent unit accepts the hidden state from the previous recurrent unit. The result of the decoder will be in logical form, such as an OMRL query converted from the utterances in the preprocessed data asset 845.

[0183] In some instances, the preprocessed data assets 845 (e.g., vectorized data assets) are used by the model trainer 840 to train (multiple) binary classifier algorithms. In some instances, the (multiple) binary classifier algorithms include logistic regression. The logic behind the logistic regression algorithm is that it finds a certain relationship and the probability of a certain outcome. In some instances, the binary classifier is trained to predict the logical form of the NL2LF model output as correct or incorrect. The data assets 845 include features of model uncertainty in the generation process of the NL2LF model (i.e., the original bundle scores), and the target / response variable has two values, correct or incorrect / 1 or 0. This type of task is called binomial logistic regression because the response / target variable has only two values, 0 and 1 or true / false. The input is fed into the (multiple) binary classifier algorithm and returns 1 or 0. As described with respect to Figure 7 As discussed, the results of logistic regression can include two outputs: (i) a binary classification indicating whether the subtree with the highest conditional probability or original cluster score at the last step N is correct or incorrect (i.e., the target / response variable); and (ii) a conditional probability score as to whether the subtree with the highest conditional probability or original cluster score at the last step N is correct or incorrect.

[0184] Once the model trainer 840 identifies the model parameter set, the model 825 has been trained, and the validator is configured to validate the model 825 using the validation data set. The validation process performed by the validator includes the following iterative operations: inputting the validation data set into the trained model 825, using validation techniques (such as K-fold cross validation, leave one cross validation, leave one group cross validation, nested cross validation, etc.) to adjust the model parameters and ultimately find the best model parameter set. Once the best model parameter set is obtained, a reserved test data set from the validation data set is input into the trained model 825 to obtain the output, and the output is evaluated relative to the ground truth value using correlation techniques such as the Bland-Altman method and the Spearman rank correlation coefficient, and performance indicators such as error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc. are calculated. In some instances, the acquisition, training and validation data process in the model system 800 can be repeatedly performed (adjusted) by the model trainer 840 until the predetermined conditions are met and the model trainer 840 can provide the final model parameter set.

[0185] It should be understood that other training / validation mechanisms are envisioned and can be implemented within the model system 800. For example, the model 825 can be trained and model parameters can be adjusted on a dataset from a subset of the obtained or filtered dataset, and the dataset from the subset of the obtained or filtered dataset can be used only to test and evaluate the performance of the model 825. In addition, although the training mechanisms described herein focus on training new models 825, these training mechanisms can also be used to fine-tune existing models trained from other datasets. For example, in some instances, the model 825 may have been pre-trained using datasets from one or more different modalities or tasks. In these cases, the model 825 can be used for transfer learning and retrained / validated using the training data and validation data as described above.

[0186] The training phase 805 outputs a trained model 825 with an optimized set of model parameters for use in the NL2LF conversion phase 810 or the confidence score calibration phase 815. The NL2LF conversion phase 810 includes a predictor for converting input data 855 (e.g., natural language utterances) into a logical form 860 (e.g., an OMRL query). For example, the predictor performs a process for inputting the input data 855 into the trained NL2LF model 865 and generating the logical form 860 based on features within the input data 855 using the trained NL2LF model 865. In some instances, the trained NL2LF model 865 performs one or more semantic parsing tasks to generate predictions based on features extracted from the input data 855. The NL2LF conversion phase 810 outputs a logical form 860, which can be used in the confidence score calibration phase 815 and the query execution phase 820. The NL2LF conversion stage 810 also outputs one or more raw cluster scores for one or more cluster levels of the trained NL2LF model 865, which can be used in the confidence score calibration stage 815, as described herein with respect to Figure 6 and Figure 7 Described in detail.

[0187] The confidence score calibration stage 815 includes a predictor for classifying a logical form 860 (e.g., an OMRL query) as correct or incorrect. For example, the predictor performs the following process: input one or more raw bundle scores of one or more bundle levels of the trained NL2LF model 865 into the trained calibration model 870, and use the trained calibration model 870 to generate a binary classification based on features within the one or more raw bundle scores. In some instances, the trained calibration model 870 performs a logistic regression task to generate a binary classification based on features extracted from the calibration model 870. The confidence score calibration stage 815 outputs a binary classification and a conditional probability score 875, which can be used as a confidence score for the logical form 860 in the query execution stage 820.

[0188] The query execution stage 820 includes one or more executors configured to execute the logical form 860 on a system such as a database 880 to obtain a result 885 (e.g., an answer to a query within (multiple) natural language utterances). For example, one or more executors may be configured to translate or convert the meaning representation language of the logical form 860 into a system language query or command, such as SQL, API, REST, GraphQL, PGQL, etc., and execute the system language query or command on the relational database to obtain an answer to the query posed in (multiple) natural language utterances. The confidence scores and conditional probability scores 875 obtained from the binary classification will allow for better interpretation of the logical form 860 and the result 885, analyze the shortcomings of the trained NL2LF model 865, and present uncertainty to the end user. In particular, when C2OMRL is used as a skill for a digital assistant (DA), a low confidence score can be used as a trigger to issue an ambiguity elimination prompt to the user. The confidence score can also be used for internal evaluation studies to better understand the weaknesses of the trained NL2LF model 865 on a certain class of input examples.

[0189] Although not explicitly shown, it should be understood that the model system 800 can further include a developer device associated with a developer. Communications from the developer device to components of the model system 800 can indicate what types of input data, utterances, and / or database schemas are to be used for the models, the number and type of models to be used, the hyperparameters for each model (e.g., learning rate and number of hidden layers), how data requests are formatted, which training data to use (e.g., and how to obtain the training data), and which validation techniques to use, and / or how the controller process is configured.

[0190] Fig. 9 is a flow diagram illustrating a process 900 for calibrating confidence scores of a machine learning model trained to convert natural language to meaning-representing language, according to various embodiments. Fig. 9 The depicted processes may be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software may be stored on a non-transitory storage medium (e.g., a memory device). Fig. 9 The methods presented and described below are intended to be illustrative rather than limiting. Fig. 9 Various processing steps are depicted as occurring in a particular order or sequence, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order or some steps may be performed in parallel. In some embodiments, such as in Figures 1 to 8 In the embodiment depicted in Fig. 9The processing depicted in can be performed by a model system (e.g., model system 800) to create, train, update, and deploy one or more machine learning models that are configured to convert natural language into logical form and use the raw bundle scores to determine calibrated confidence scores for the logical form.

[0191] At step 905, one or more raw cluster scores are obtained. The one or more raw cluster scores are generated from one or more cluster levels of a decoder of a machine learning model trained to convert natural language into logical form. Each of the one or more raw cluster scores is a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more cluster levels. Natural language can be expressed as natural language utterances (e.g., natural language queries), and logical form can be expressed as logical form statements, such as queries or operations in a meaning representation language (e.g., OMRL).

[0192] In some instances, the machine learning model converts the natural language utterance into a logical form using a bottom-up generation process implemented by a heuristic search algorithm. The bottom-up generation process may include: (i) for a first clustering level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form, and selecting the top K subtrees to retain using the original clustering score associated with each subtree; (ii) for each clustering level after the first clustering level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form and the top K subtrees retained from the previous clustering level, and selecting another top K subtrees to retain using the original clustering score associated with each subtree; and (iii) for the last clustering level, determining the subtree with the highest original clustering score as the final output of the decoder. The subtree with the highest original clustering score is converted to a logical form.

[0193] In some instances, the one or more bundling levels include a first bundling level, a last bundling level, one or more bundling levels between the first bundling level and the last bundling level, or any combination thereof. In some instances, the one or more raw bundling scores are obtained from the first bundling level, the last bundling level, one or more bundling levels between the first bundling level and the last bundling level, or any combination thereof. In certain instances, the one or more raw bundling scores are the bundling scores of the first K subtrees retained from the first bundling level, the last bundling level, and one or more bundling levels between the first bundling level and the last bundling level.

[0194] In some instances, one or more original bundle scores are preprocessed to generate a vector. The vector includes a stack or sequence arrangement of one or more original bundle scores. In some instances, the order of one or more original bundle scores from the bundle level is maintained in the vector (i.e., the original bundle score from the first bundle level is ranked first in the vector, followed by the original bundle score from the second bundle level, followed by the original bundle score from the third bundle level, and so on). In some instances, one or more original bundle scores of all bundle levels or a subset of one or more original bundle scores are vectorized. In some instances, the first K original bundle scores of all bundle levels are vectorized. In other instances, a subset of the first K original bundle scores of all bundle levels is vectorized. In some instances, the original bundle scores of some bundle levels or a subset of the original bundle scores are vectorized (e.g., the last bundle level or the last three bundle levels). In some instances, the first K original bundle scores of some bundle levels are vectorized. In other examples, a subset of the top K raw bundle scores for some bundle levels are vectorized.

[0195] At step 910, the logical form output by the machine learning model is classified as correct or incorrect based on one or more raw bundle scores. The classification includes: inputting one or more raw bundle scores into the calibration model, and predicting the correct or incorrect category of the logical form by the calibration model based on the one or more raw bundle scores. The correct or incorrect category represents the calibration model's determination of whether the subtree with the highest raw bundle score at the last bundle level of the decoder represents a correct or incorrect transformation of the natural language utterance, and the subtree with the highest raw bundle score represents the logical form. In some instances, the conditional probability score is modeled using a sigmoid function. The classification further includes outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest raw bundle score is correct or incorrect. In some instances, the calibration model also outputs the correct or incorrect category of the logical form.

[0196] At step 915, a confidence score is provided for the logical form, the confidence score being determined based on the conditional probability score. In some instances, the conditional probability score is used as the confidence score for the logical form. In some instances, a correct or incorrect category of the logical form is also provided.

[0197] At optional step 920, a process can be controlled based on the confidence score in the logical form. In some instances, the training or retraining of the machine learning model can be controlled or optimized based on the confidence score in the logical form. In some instances, it can be determined whether the confidence score is above a predetermined threshold, and in response to determining that the confidence score is above the predetermined threshold, a query can be performed on a database based on the logical form. In some instances, the confidence score in the logical form is reported to an end user.

[0198] Descriptive System

[0199] Fig.10 Depicted is a simplified diagram of a distributed system 1000. In the illustrated example, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 may be configured to execute one or more applications.

[0200] In various examples, server 1012 can be adapted to run one or more services or software applications that implement one or more embodiments described in the present disclosure. In some examples, server 1012 can also provide other services or software applications that can include non-virtual environments and virtual environments. In some examples, these services can be provided to users of client computing devices 1002, 1004, 1006, and / or 1008 as web-based services or cloud services (such as under a software as a service (SaaS) model). Users operating client computing devices 1002, 1004, 1006, and / or 1008 can then utilize one or more client applications to interact with server 1012 to utilize the services provided by these components.

[0201] exist Fig.10 In the depicted configuration, the server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by the server 1012. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be appreciated that a variety of different system configurations that may differ from the distributed system 1000 are possible. Thus, Fig.10 The illustrated example is one example of a distributed system for implementing the example system and is not intended to be limiting.

[0202] A user may use client computing devices 1002, 1004, 1006, and / or 1008 to execute one or more applications, models, or chatbots that may generate one or more events or models that may then be implemented or serviced according to the teachings of the present disclosure. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although Fig.10 Only four client computing devices are depicted, but any number of client computing devices may be supported.

[0203] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome TM OS), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM , Palm Portable handheld devices may include cellular phones, smart phones (e.g., ), tablet computers (e.g. ), personal digital assistants (PDAs), etc. Wearable devices can include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Microsoft Gesture Input Devices Game consoles, Sony System, by Various game systems provided, and others), etc. The client device may be able to execute various different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications) and may use various communication protocols.

[0204] The network(s) 1010 may be any type of network familiar to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), By way of example only, the network(s) 1010 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., in accordance with the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, and / or any other wireless protocol) and / or any combination of these networks and / or other networks.

[0205] The server 1012 may be composed of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, The server 1012 may include a server, a mid-range server, a mainframe computer, a rack server, etc.), a server farm, a server cluster, or any other suitable arrangement and / or combination. The server 1012 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices for the server. In various examples, the server 1012 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0206] The computing system in the server 1012 can run one or more operating systems, including any of those discussed above and any commercially available server operating system. The server 1012 can also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transport Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines Corporation), etc.

[0207] In some implementations, server 1012 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. As examples, data feeds and / or event updates may include, but are not limited to: feed, The server 1012 may also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client computing devices 1002, 1004, 1006, and 1008.

[0208] The distributed system 1000 may also include one or more data repositories 1014, 1016. In some examples, these data repositories can be used to store data and other information. For example, one or more of the data repositories 1014, 1016 can be used to store information (such as information related to chatbot performance or generated models) for use by the chatbot used by the server 1012 when performing various functions according to various embodiments. The data repositories 1014, 1016 can reside in various locations. For example, the data repository used by the server 1012 can be local to the server 1012 or can be remote from the server 1012 and communicate with the server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 can be of different types. In some examples, the data repository used by the server 1012 can be a database, such as a relational database, such as Oracle And other suppliers of databases. One or more of these databases can be adapted to respond to SQL format commands to implement data storage, update and retrieval to and from the database.

[0209] In some examples, one or more of data repositories 1014, 1016 may also be used by applications to store application data. The data repositories used by applications may be of different types, such as, for example, key-value storage repositories, object storage repositories, or general storage repositories backed by a file system.

[0210] In some examples, the functionality described in this disclosure may be provided as a service via a cloud environment. Fig.11 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services according to certain examples. Fig.11 In the depicted example, cloud infrastructure system 1102 may provide one or more cloud services that may be requested by a user using one or more client computing devices 1104, 1106, and 1108. Cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above with respect to server 1012. The computers in cloud infrastructure system 1102 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0211] The network(s) 1110 can facilitate data communication and exchange between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network(s) 1110 can include one or more networks. The networks can be of the same or different types. The network(s) 1110 can support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0212] Fig.11 The depicted example is only one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in some other examples, the cloud infrastructure system 1102 may have more Fig.11 Those components depicted may have more or fewer components, may combine two or more components, or may have a different configuration or arrangement of components. Fig.11 Three client computing devices are depicted, but in alternative examples any number of client computing devices may be supported.

[0213] The term cloud service is generally used to refer to services that are made available to users on demand and via a communication network such as the Internet through a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can make use of the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can subscribe and use the applications on demand via the Internet without having to purchase infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle of Redwood Shores, California, offers cloud services. It provides several cloud services such as middleware services, database services, Java cloud services and other services.

[0214] In some examples, cloud infrastructure system 1102 may provide one or more cloud services using different models, such as software as a service (SaaS) model, platform as a service (PaaS) model, infrastructure as a service (IaaS) model, and other models (including hybrid service models). Cloud infrastructure system 1102 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0215] The SaaS model enables applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase the underlying application hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by the cloud infrastructure system 1102. Oracle Examples of SaaS services provided include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and others.

[0216] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services to provide elastic computing and storage capabilities. Provides various IaaS services.

[0217] The PaaS model is typically used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without the customer having to purchase, build, or maintain such resources. Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

[0218] Cloud services are typically provided on an on-demand self-service basis, subscription-based, elastically scalable, reliable, highly available, and secure. For example, a customer may order one or more services provided by the cloud infrastructure system 1102 via a subscription order. The cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use speech to request that the cloud infrastructure system take a certain action (e.g., intent) as described above and / or provide services to a chatbot system as described herein. The cloud infrastructure system 1102 may be configured to provide one or even multiple cloud services.

[0219] The cloud infrastructure system 1102 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 1102 can be owned by a third-party cloud service provider, and the cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In some other examples, under a private cloud model, the cloud infrastructure system 1102 can operate within an organization (e.g., within an enterprise organization) and services are provided to customers within the organization. For example, customers can be various departments of an enterprise such as a human resources department, a payroll department, or even individuals within an enterprise. In some other examples, under a community cloud model, the cloud infrastructure system 1102 and the services provided can be shared by several organizations in the relevant community. Various other models such as a mix of the models mentioned above can also be used.

[0220] Client computing devices 1104, 1106, and 1108 may be of different types (eg, Fig.10 The depicted client computing devices 1002, 1004, 1006, and 1008) may be capable of operating one or more client applications. A user may use a client device to interact with the cloud infrastructure system 1102, such as requesting services provided by the cloud infrastructure system 1102. For example, a user may use a client device to request information or actions from a chatbot as described in the present disclosure.

[0221] In some examples, the processing performed by the cloud infrastructure system 1102 for providing services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating data sets to train and deploy one or more models. The analysis may be performed by one or more processors, thereby potentially processing data in parallel, performing simulations using data, and the like. For example, big data analysis may be performed by the cloud infrastructure system 1102 for generating and training one or more models for a chatbot system. The data used for the analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0222] like Fig.11 , cloud infrastructure system 1102 may include infrastructure resources 1130 used to facilitate provisioning of various cloud services provided by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that may be used to service storage requested from an application may be part of cloud infrastructure system 1102. In other examples, the storage virtual machine may be part of a different system.

[0223] In some examples, to facilitate efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 1102 for different customers, resources can be bundled into resource groups or resource modules (also referred to as "pods"). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods can be pre-provisioned for different types of cloud services. For example, a first group of pods can be provisioned for database services, a second group of pods can be provisioned for Java services (the second group of pods can include a different combination of resources than the pods in the first group of pods), and so on. For some services, the resources allocated for provisioning the services can be shared between services.

[0224] The cloud infrastructure system 1102 itself may internally use services 1132 that are shared by different components of the cloud infrastructure system 1102 and that facilitate the provision of services by the cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for implementing cloud support, email services, notification services, file transfer services, and the like.

[0225] The cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. Fig.11 As depicted, the subsystems may include a user interface subsystem 1112 that enables users or customers of the cloud infrastructure system 1102 to interact with the cloud infrastructure system 1102. The user interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 (where cloud services provided by the cloud infrastructure system 1102 are advertised and can be purchased by consumers), and other interfaces 1118. For example, a customer may use a client device to request (service request 1134) one or more services provided by the cloud infrastructure system 1102 using one or more of the interfaces 1114, 1116, and 1118. For example, a customer may visit an online store, browse cloud services provided by the cloud infrastructure system 1102, and place a subscription order for one or more services provided by the cloud infrastructure system 1102 that the customer wishes to subscribe to. The service request may include information identifying the customer and the one or more services that the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 1102. As part of the order, the customer may provide information identifying a chatbot system for which the services are to be provided and optionally provide one or more credentials for the chatbot system.

[0226] In some examples (such as Fig.11In the depicted example), cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this processing, OMS 1120 may be configured to: create an account for the customer (if not already created); receive billing and / or billing information from the customer to be used to bill the customer for providing the requested service to the customer; verify the customer information; after verification, book the order for the customer; and orchestrate various workflows to prepare the order for fulfillment.

[0227] Once properly authenticated, the OMS 1120 may then call an order provisioning subsystem (OPS) 1124 configured to provision resources (including processing resources, memory resources, and networking resources) for the order. Provisioning may include allocating resources to an order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for an order and the type of resources provisioned may depend on the type of cloud service that the customer has ordered. For example, according to a workflow, the OPS 1124 may be configured to determine a specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated to an order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users to be supported by the service, the duration of the service being requested, and the like. The allocated groups may then be customized for the specific requesting customer to provide the requested service.

[0228] In some examples, the setup phase processing as described above can be performed as part of the provisioning process by cloud infrastructure system 1102. Cloud infrastructure system 1102 can generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by cloud infrastructure system 1102 itself or from storage virtual machines provided by systems other than cloud infrastructure system 1102.

[0229] The cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) may be sent to the customer that enables the customer to begin using and utilizing the benefits of the requested service. In some examples, for the customer requesting the service, the response may include a chatbot system ID generated by the cloud infrastructure system 1102 and information identifying the chatbot system selected by the cloud infrastructure system 1102 as the chatbot system corresponding to the chatbot system ID.

[0230] The cloud infrastructure system 1102 can provide services to multiple customers. For each customer, the cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the order, and providing the requested services to the customer. The cloud infrastructure system 1102 can also collect usage statistics about the customer's use of the subscribed services. For example, statistics can be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system up time and system down time. The usage information can be used to bill the customer. Billing can be done, for example, on a monthly basis.

[0231] The cloud infrastructure system 1102 can provide services to multiple customers in parallel. The cloud infrastructure system 1102 can store information (possibly including proprietary information) of these customers. In some examples, the cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage customer information and provide separation of managed information so that information related to one customer cannot be accessed by another customer. The IMS 1128 can be configured to provide various security-related services such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles and related functions, etc.

[0232] Fig.12 An example of a computer system 1200 is illustrated. In some examples, the computer system 1200 can be used to implement any digital assistant or chatbot system in a distributed environment and the various servers and computer systems described above. Fig.12 As shown, computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. Storage subsystem 1218 may include non-transitory computer-readable storage media, including storage media 1222 and system memory 1210.

[0233] The bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of the computer system 1200 to communicate with each other as expected. Although the bus subsystem 1202 is schematically shown as a single bus, alternative examples of the bus subsystem can utilize multiple buses. The bus subsystem 1202 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such an architecture can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus (which can be implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0234] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core processor or a multi-core processor. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1204 may include one or more dedicated coprocessors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 1204 may be implemented using custom circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0235] In some examples, the processing unit in the processing subsystem 1204 can execute instructions stored in the system memory 1210 or on the computer-readable storage medium 1222. In various examples, the processing unit can execute various programs or code instructions and can maintain multiple programs or processes that are executed simultaneously. At any given time, some or all of the program code to be executed can reside in the system memory 1210 and / or on the computer-readable storage medium 1222 (potentially including residing on one or more storage devices). Through suitable programming, the processing subsystem 1204 can provide the various functions described above. In the instance where the computer system 1200 executes one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0236] In some examples, a processing acceleration unit 1206 may optionally be provided for performing custom processing or for offloading some of the processing performed by processing subsystem 1204 , thereby speeding up the overall processing performed by computer system 1200 .

[0237] I / O subsystem 1208 may include devices and mechanisms for inputting information to computer system 1200 and / or for outputting information from or via computer system 1200. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information to computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as a mouse or trackball, touch pads or touch screens incorporated into a display, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Windows, which enable a user to control and interact with input devices. Motion Sensor, Microsoft 360 game controllers, devices that provide an interface for receiving input using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as detecting eye activity from the user (e.g., "blinking" when taking a picture and / or making a menu selection) and translating eye gestures into input devices (such as Google ) Blink detector. In addition, the user interface input device may include an interface that enables the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0238] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing rods, game controllers and graphics boards, and audio / visual devices (such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, eye tracking devices). In addition, user interface input devices may include, for example, medical imaging input devices, such as computer tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices, such as MIDI keyboards, digital musical instruments, etc.

[0239] In general, the term output device is used to include all possible types of devices and mechanisms for outputting information from the computer system 1200 to a user or other computers. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. The display subsystem may be a cathode ray tube (CRT), a flat panel device (such as a flat panel device using a liquid crystal display (LCD) or a plasma display), a projection device, a touch screen, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0240] The storage subsystem 1218 provides a repository or data storage for storing information and data used by the computer system 1200. The storage subsystem 1218 provides a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide some of the example functionality. The storage subsystem 12112 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 can also provide authentication in accordance with the teachings of the present disclosure.

[0241] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. Fig.12 As shown, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include multiple memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines that help transfer information between elements within the computer system 1200 during startup may typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by the processing subsystem 1204. In some embodiments, the system memory 1210 may include a variety of different types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0242] By way of example and not limitation, Fig.12As depicted, system memory 1210 may load executing applications 1212 (which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc.), program data 1214, and an operating system 1216. By way of example, operating system 1216 may include various versions of Microsoft Apple and / or Linux operating systems, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or iOS, Telephone, OS, OS, OS operating system and other mobile operating systems, as well as other operating systems.

[0243] Computer-readable storage media 1222 may store programming and data structures that provide some example functionality. Computer-readable media 1222 may provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 1200. Software (programs, code modules, instructions) that provide the functionality described above when executed by processing subsystem 1204 may be stored in storage subsystem 1218. By way of example, computer-readable storage media 1222 may include, for example, a hard drive, a magnetic disk drive, an optical disk drive (e.g., a CD ROM, a DVD, Computer readable storage media 1222 may include, but is not limited to, The computer readable storage medium 1222 may include a solid-state drive (SSD) based on flash memory, an enterprise-class flash memory drive, a solid-state ROM, and the like, and an SSD based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSD, magnetoresistive RAM (MRAM) SSD, and a hybrid SSD using a combination of DRAM and flash-based SSD.

[0244] In some examples, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may be further connected to computer-readable storage medium 1222. Reader 1220 may receive data from a storage device such as a disk, flash memory drive, etc. and is configured to read data from the storage device.

[0245] In some examples, computer system 1200 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, computer system 1200 can provide support for executing one or more virtual machines. In some examples, computer system 1200 can execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine can be allocated with memory, computing (e.g., processor, core), I / O and networking resources. Each virtual machine usually runs independently of other virtual machines. Virtual machines usually run their own operating system, which can be the same or different from the operating system executed by other virtual machines executed by computer system 1200. Therefore, multiple operating systems can potentially be run simultaneously by computer system 1200.

[0246] The communication subsystem 1224 provides an interface to other computer systems and networks. The communication subsystem 1224 is used as an interface for receiving data from other systems and transmitting data from the computer system 1200 to other systems. For example, the communication subsystem 1224 can enable the computer system 1200 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and sending information to the client devices. For example, when the computer system 1200 is used to implement Figure 1 When the robot system 120 is depicted, the communication subsystem can be used to communicate with the chat robot system selected for the application.

[0247] The communication subsystem 1224 can support both wired communication protocols and / or wireless communication protocols. In some examples, the communication subsystem 1224 may include a radio frequency (RF) transceiver component for accessing a wireless voice and / or data network (e.g., using cellular phone technology, advanced data network technologies such as 3G, 4G or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX Family Standards, or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, in addition to or in lieu of a wireless interface, the communication subsystem 1224 may provide wired network connectivity (e.g., Ethernet).

[0248] The communication subsystem 1224 can receive and transmit various forms of data. In some examples, the communication subsystem 1224 can receive input communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., among other forms. For example, the communication subsystem 1224 can be configured to receive (or send) data feeds 1226 in real time from users of social media networks and / or other communication services, such as feed, Updates, web feeds (such as Rich Site Summary (RSS) feeds), and / or real-time updates from one or more third-party information sources.

[0249] In some examples, the communication subsystem 1224 may be configured to receive data in the form of continuous data streams, which may include event streams 1228 of real-time events (which may be continuous or unbounded in nature without explicit termination) and / or event updates 1230. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0250] The communication subsystem 1224 may also be configured to transmit data from the computer system 1200 to other computer systems or networks. The data may be transmitted in various forms such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. to one or more databases that may be in computer communication with one or more stream data source computers coupled to the computer system 1200.

[0251] Computer system 1200 may be one of various types, including a handheld portable device (e.g., Cellular phone, computing tablets, PDAs), wearable devices (e.g., Google head mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, the Fig.12 The description of the computer system 1200 depicted is intended to be a specific example only. Fig.12 Many other configurations of more or fewer components of the depicted systems are possible.Based on the disclosure and the teachings provided herein, it should be appreciated that there are other ways and / or methods to implement the various examples.

[0252] Although specific examples have been described, various modifications, changes, alternative constructions and equivalents are possible. Examples are not limited to operations in certain specific data processing environments, but are freely operated in multiple data processing environments. In addition, although certain examples have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be restrictive. Although some flow charts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. The process can have additional steps not included in the figure. The various features and aspects of the examples described above can be used individually or jointly.

[0253] Further, although certain examples have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are possible. Certain examples may be implemented in hardware only or software only or using a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors.

[0254] Where a device, system, component or module is described as being configured to perform certain operations or functions, such configuration may be accomplished, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0255] Specific details are given in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures and techniques have been shown without unnecessary details to avoid obscuring the examples. This description only provides illustrative examples and is not intended to limit the scope, applicability or configuration of other examples. Instead, the previous description of the examples will provide enabling descriptions for implementing various examples for those skilled in the art. Various changes can be made to the functions and arrangements of the elements.

[0256] Therefore, the present specification and drawings should be viewed in an illustrative rather than a restrictive sense. However, it will be apparent that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, although specific examples have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0257] In the foregoing description, various aspects of the present disclosure are described with reference to specific examples of the present disclosure, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used individually or in combination. Further, the examples may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0258] In the foregoing description, for the purpose of illustration, the method is described in a specific order. It should be appreciated that in an alternative example, the method can be performed in an order different from the described order. It should also be appreciated that the method described above can be performed by hardware components or can be embodied in a sequence of machine executable instructions, which can be used to make a machine (such as a general or special processor or logic circuit programmed with the instruction) perform the method. These machine executable instructions can be stored on one or more machine-readable media, such as CD-ROM or other types of optical disks, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic cards or optical cards, flash memory or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the method can be performed by a combination of hardware and software.

[0259] Where a component is described as being configured to perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operations, or any combination thereof.

[0260] While illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be interpreted to include such variations, except to the extent limited by the prior art.

Claims

1. A computer-implemented method comprising: obtaining one or more raw bundle scores generated from one or more bundle levels of a decoder of a machine learning model trained to convert natural language into logical form, wherein each of the one or more raw bundle scores is a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more bundle levels; Classifying the logical form output by the machine learning model as correct or incorrect based on the one or more raw bundle scores, wherein the classification comprises: inputting the one or more raw bundle scores into a calibration model; predicting, by the calibration model, a correct or incorrect category of the logical form based on the one or more raw clustering scores, wherein the correct or incorrect category represents a determination by the calibration model as to whether a subtree with a highest raw clustering score at a last clustering level of the decoder represents a correct or incorrect transformation of a natural language utterance, and the subtree with the highest raw clustering score represents the logical form; and Outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest original bundle score is correct or incorrect; and A confidence score is provided for the logical form, the confidence score being determined based on the conditional probability score.

2. The computer-implemented method of claim 1 , further comprising converting, by the machine learning model, the natural language utterance into the logical form, wherein: The conversion comprises a bottom-up generation process implemented by the heuristic search algorithm, and wherein the bottom-up generation process comprises: For a first clustering level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form, and selecting top K subtrees to retain using an original clustering score associated with each subtree; for each bundling level after the first bundling level, generating F possible subtrees of the natural language utterance based on the vocabulary or grammar associated with the logical form and the top K subtrees retained from the previous bundling level, and selecting additional top K subtrees to retain using the original bundling scores associated with each subtree; and For the last clustering level, the subtree with the highest original clustering score is determined as the final output of the decoder.

3. The computer-implemented method of claim 2, wherein: The one or more bundling levels include the first bundling level, the last bundling level, one or more bundling levels between the first bundling level and the last bundling level, or any combination thereof.

4. The computer-implemented method of claim 3, wherein: The one or more raw bundle scores are obtained from the first bundle level, the last bundle level, the one or more bundle levels between the first bundle level and the last bundle level, or any combination thereof.

5. The computer-implemented method of claim 3, wherein: The one or more original clustering scores are clustering scores of the first K subtrees retained from the first clustering level, the last clustering level, and the one or more clustering levels between the first clustering level and the last clustering level.

6. The computer-implemented method of any one of claims 1 to 5, wherein: The conditional probability score is modeled using a sigmoid function and used as the confidence score of the logical form.

7. The computer-implemented method of any one of claims 1 to 5, further comprising: determining that the confidence score is above a predetermined threshold; as well as In response to determining that the confidence score is above the predetermined threshold, a query is performed on a database based on the logical form.

8. A system comprising: one or more processors; as well as One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations including: obtaining one or more raw bundle scores generated from one or more bundle levels of a decoder of a machine learning model trained to convert natural language into logical form, wherein each of the one or more raw bundle scores is a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more bundle levels; Classifying the logical form output by the machine learning model as correct or incorrect based on the one or more raw bundle scores, wherein the classification comprises: inputting the one or more raw bundle scores into a calibration model; predicting, by the calibration model, a correct or incorrect category of the logical form based on the one or more raw clustering scores, wherein the correct or incorrect category represents a determination by the calibration model as to whether a subtree with a highest raw clustering score at a last clustering level of the decoder represents a correct or incorrect transformation of a natural language utterance, and the subtree with the highest raw clustering score represents the logical form; and Outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest original bundle score is correct or incorrect; and A confidence score is provided for the logical form, the confidence score being determined based on the conditional probability score.

9. The system of claim 8, wherein: The operations further include converting, by the machine learning model, the natural language utterance into the logical form, wherein the converting includes a bottom-up generation process implemented by the heuristic search algorithm, and wherein the bottom-up generation process includes: For a first clustering level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form, and selecting top K subtrees to retain using an original clustering score associated with each subtree; for each bundling level after the first bundling level, generating F possible subtrees of the natural language utterance based on the vocabulary or grammar associated with the logical form and the top K subtrees retained from the previous bundling level, and selecting additional top K subtrees to retain using the original bundling scores associated with each subtree; and For the last clustering level, the subtree with the highest original clustering score is determined as the final output of the decoder.

10. The system of claim 9, wherein: The one or more bundling levels include the first bundling level, the last bundling level, one or more bundling levels between the first bundling level and the last bundling level, or any combination thereof.

11. The system of claim 10, wherein: The one or more raw bundle scores are obtained from the first bundle level, the last bundle level, the one or more bundle levels between the first bundle level and the last bundle level, or any combination thereof.

12. The system of claim 10, wherein: The one or more original clustering scores are clustering scores of the first K subtrees retained from the first clustering level, the last clustering level, and the one or more clustering levels between the first clustering level and the last clustering level.

13. The system of any one of claims 8 to 12, wherein: The conditional probability score is modeled using a sigmoid function and used as the confidence score of the logical form.

14. The system of any one of claims 8 to 12, wherein: The operations further include: determining that the confidence score is above a predetermined threshold; and In response to determining that the confidence score is above the predetermined threshold, a query is performed on a database based on the logical form.

15. A computer program product tangibly embodied in one or more non-transitory machine-readable media, the computer program product comprising instructions executable by one or more processors to: obtaining one or more raw cluster scores generated from one or more cluster levels of a decoder of a machine learning model trained to convert natural language into logical form, wherein Each of the one or more raw cluster scores is a conditional probability of a subtree determined by a heuristic search algorithm of the decoder at one of the one or more cluster levels; Classifying the logical form output by the machine learning model as correct or incorrect based on the one or more raw bundle scores, wherein the classification comprises: inputting the one or more raw bundle scores into a calibration model; predicting, by the calibration model, a correct or incorrect category of the logical form based on the one or more raw clustering scores, wherein the correct or incorrect category represents a determination by the calibration model as to whether a subtree with a highest raw clustering score at a last clustering level of the decoder represents a correct or incorrect transformation of a natural language utterance, and the subtree with the highest raw clustering score represents the logical form; and Outputting, by the calibration model, a conditional probability score as to whether the subtree with the highest original bundle score is correct or incorrect; and A confidence score is provided for the logical form, the confidence score being determined based on the conditional probability score.

16. The computer program product of claim 15, wherein: The operations further include converting, by the machine learning model, the natural language utterance into the logical form, wherein the converting includes a bottom-up generation process implemented by the heuristic search algorithm, and wherein the bottom-up generation process includes: For a first clustering level, generating F possible subtrees of the natural language utterance based on a vocabulary or grammar associated with the logical form, and selecting top K subtrees to retain using an original clustering score associated with each subtree; for each bundling level after the first bundling level, generating F possible subtrees of the natural language utterance based on the vocabulary or grammar associated with the logical form and the top K subtrees retained from the previous bundling level, and selecting additional top K subtrees to retain using the original bundling scores associated with each subtree; and For the last clustering level, the subtree with the highest original clustering score is determined as the final output of the decoder.

17. The computer program product of claim 16, wherein: The one or more bundling levels include the first bundling level, the last bundling level, one or more bundling levels between the first bundling level and the last bundling level, or any combination thereof.

18. The computer program product of claim 17, wherein: The one or more raw bundle scores are obtained from the first bundle level, the last bundle level, the one or more bundle levels between the first bundle level and the last bundle level, or any combination thereof.

19. The computer program product according to any one of claims 15 to 18, wherein: The conditional probability score is modeled using a sigmoid function and used as the confidence score of the logical form.

20. The computer program product of any one of claims 15 to 18, wherein: The operations further include: determining that the confidence score is above a predetermined threshold; and In response to determining that the confidence score is above the predetermined threshold, a query is performed on a database based on the logical form.

21. An apparatus comprising: Means for carrying out the operations of the method according to any one of claims 1 to 7.

22. A computer program product comprising computer instructions which, when executed by one or more processors, implement the operations of the method of any one of claims 1 to 7.

Images