Techniques for converting natural language utterances into intermediate database query representations

Abstract

[0003] A technique for converting a natural language utterance into an intermediate database query representation is disclosed herein. An input string is generated by concatenating the natural language utterance with a database schema representation of a database. Based on the input string, a first encoder generates one or more embeddings of the natural language utterance and the database schema representation. A second encoder encodes associations between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings. A grammar-based decoder generates an intermediate database query representation based on the encoded associations and the one or more embeddings. Based on the intermediate database query representation and the interface specification, a database query is generated in a database query language.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application of and claims the benefit of and priority under 35 U.S.C. §119(e) to U.S. Patent Application No. 18 / 209,844, filed June 14, 2023, and U.S. Provisional Patent Application No. 63 / 400,037, filed August 22, 2022, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] Field The present disclosure relates generally to converting natural language into semantic representation languages, and more particularly to machine learning techniques for converting natural language utterances into intermediate database query representation languages ​​such as the Oracle Semantic Representation Language (OMRL). [Background technology]

[0003] background Artificial intelligence has many applications. For example, many users around the world engage with instant messaging or chat platforms for instant responses. Organizations often use these instant messaging or chat platforms to engage with customers (or end users) in live conversations. However, it can be very costly for organizations to employ service personnel to participate in live communications with customers or end users. Chatbots or bots are beginning to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with bots through messaging apps that the end users already have installed and use. Intelligent bots, typically equipped with artificial intelligence (AI), can communicate more intelligently and contextually in live conversations, thus enabling more natural conversations between bots and end users for an improved conversational experience. Instead of end users learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots may be able to understand the end user's intent based on user utterances in natural language and respond accordingly.

[0004] Artificial intelligence-based solutions, such as chatbots, can have both analog (human) and digital (machine) interfaces for interacting with humans and connecting to back-end systems. When humans speak using natural language, it is advantageous to be able to extract and parse the meaning of the utterance (e.g., a request) regardless of how the back-end system addresses the utterance. As an example, the request may be for data that needs to be retrieved from a relational database, or the requested data may need to be extracted from a knowledge graph. A semantic representation language (MRL) is a generic representation of natural language utterances that a chatbot can translate into any number of target machine-oriented languages. Thus, MRL can be utilized by chatbots to communicate interchangeably with both humans and a variety of back-end systems, including systems that communicate using Structured Query Language (SQL), Application Programming Interfaces (APIs), Representational State Transfer (REST), Graph Query Language (GraphQL), Property Graph Query Language (PGQL), etc. Summary of the Invention [Problem to be solved by the invention]

[0005] For example, SQL is a standard database management language for interacting with relational databases. SQL can be used to store, manipulate, retrieve, and / or otherwise manage data held in a relational database management system (RDBMS) and / or for stream processing in a relational data stream management system (RDSMS). SQL includes statements or commands used to interact with relational databases. SQL statements or commands are categorized into, among others, Data Query Language (DQL) statements, Data Definition Language (DDL) statements, Data Control Language (DCL) statements, and Data Manipulation Language (DML) statements. To interact with a relational database using SQL, a user must know how the database is structured (e.g., knowledge of the tables and the rows and columns within each table), SQL syntax, and how the syntax relates to the database structure. Without this knowledge, it is often difficult for a user to interact with these relational databases using SQL.

[0006] Natural language interfaces (e.g., chatbots) to database systems (NLIDBs) such as RDBMSs provide users with an intuitive means of interacting with these relational databases without requiring knowledge of database management languages. For example, using natural language statements and queries (i.e., natural language queries), users can interact with these relational databases through NLIDBs using easy-to-understand language. Recently, text-to-SQL systems have become popular, and deep learning techniques for converting natural language queries into SQL queries have proven promising. Semantic parsing can be used to convert natural language statements, requests, and questions (i.e., sentences) into machine-oriented language that can be executed by applications (e.g., chatbots, models, programs, machines, etc.). For example, semantic parsing can convert natural language sentences into general-purpose programming languages ​​such as Python, Java, and SQL. Processes for converting natural language sentences into SQL queries typically involve rule-based, statistical-based, and / or deep learning-based systems. Rule-based systems typically use a set of fixed rules to convert natural language sentences into SQL queries. These rule-based systems are generally domain-specific and therefore considered inelastic, and do not generalize well to novel use cases (e.g., across different domains). Statistical-based systems, such as slot-filling, label tokens (i.e., words or phrases) in an input natural language sentence according to their semantic role within the sentence and use the labels to fill slots in an SQL query. These statistical-based systems are generally limited in the types of sentences they can parse (e.g., the sentence must be capable of being represented as a parse tree). Deep learning-based systems, such as sequence-to-sequence models, involve training deep learning models that directly convert natural language sentences into machine-oriented language and have been shown to generalize across tasks, domains, and datasets.However, such deep learning systems require a large amount of training data for supervised learning, and it is difficult to obtain labeled data (e.g., natural language query-SQL statement pairs). Therefore, deep learning-based conversion of natural language sentences into machine-oriented language may inevitably require a large amount of labeled training data. [Means for solving the problem]

[0007] Quick Overview Machine learning techniques (e.g., methods, systems, non-transitory computer-readable media storing code or instructions executable by one or more processors) are provided for techniques for using machine learning models that have been trained to convert natural language into a semantic representation language, such as Oracle Semantic Representation Language (OMRL).

[0008] In various embodiments, a computer-implemented method is provided that includes generating an input string by concatenating a natural language utterance with a database schema representation of a database; generating, by a first encoder, one or more embeddings of the natural language utterance and the database schema representation based on the input string; encoding, by a second encoder, associations between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings; generating, by a grammar-based decoder, an intermediate database query representation based on the encoded associations and the one or more embeddings; and generating a database query in a database query language based on the intermediate database query representation and an interface specification.

[0009] In some embodiments, the computer-implemented method further includes providing to the second encoder schema link associations linking elements in the database schema representation with words in the natural language utterance, and the embeddings are further generated based on the schema link associations. In some embodiments, the schema link associations include metadata specifying synonyms for the words.

[0010] In some embodiments, the method further includes providing a relational algebra grammar to the grammar-based decoder, the relational algebra grammar representing the intermediate database query expression as a tree, the intermediate database query expression being further based on the relational algebra grammar.

[0011] In some embodiments, the first encoder is a pre-trained language model (PLM) and the second encoder is a relationship-aware transformer (RAT).

[0012] In some embodiments, the method further includes performing a database query on a database to retrieve data in response to the natural language utterance.

[0013] In some embodiments, the database schema representation of the database includes link attributes that reference entries in a table without referencing the name of the table.

[0014] In various embodiments, a system is provided that includes one or more processors and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0015] In various embodiments, one or more non-transitory computer-readable media are provided for storing instructions that, when executed by one or more processors, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0016] In various embodiments, an apparatus is provided, the apparatus comprising means for performing some or all of the operations and / or methods disclosed herein.

[0017] In various embodiments, a computer program product is provided that includes instructions that, when executed by a processor, perform some or all of the operations and / or methods disclosed herein.

[0018] The techniques described above and below can be implemented in multiple ways and in multiple contexts. Some exemplary embodiments and contexts are provided with reference to the accompanying drawings, as described in more detail below. However, the following embodiments and contexts are only a few of many. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a simplified block diagram of a distributed environment incorporating an illustrative embodiment. [Figure 2] FIG. 1 is a simplified block diagram of a computing system implementing a masterbot according to one embodiment. [Figure 3] FIG. 1 is a simplified block diagram of a computing system implementing a skillbot according to one embodiment. [Figure 4] FIG. 1 is a block diagram illustrating an overview of a C2OMRL architecture and process for generating queries to a backend interface starting with a natural language utterance, according to various embodiments. [Figure 5]FIG. 1 is a simplified block diagram of a C2OMRL architecture in accordance with various embodiments. [Figure 6] FIG. 1 illustrates a model system for training and deploying models according to various embodiments. [Figure 7] FIG. 1 illustrates a process flow for converting a natural language utterance into an intermediate database query representation according to various embodiments. [Figure 8] FIG. 1 illustrates an example RA tree illustrating link functionality according to various embodiments. [Figure 9A] FIG. 1 illustrates an exemplary RA tree with Like_start according to various embodiments. [Figure 9B] FIG. 1 illustrates an exemplary RA tree with Like_end in accordance with various embodiments. [Figure 10] FIG. 1 is a simplified diagram of a distributed system for implementing various embodiments. [Figure 11] FIG. 1 is a simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment system can be provided as cloud services, according to various embodiments. [Figure 12] FIG. 1 illustrates an exemplary computer system that can be used to implement various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0020] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of certain embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0021] introduction In recent years, the amount of data feeding into various industries and their systems has grown exponentially. Most business information is stored in the form of relational databases, which store, process, and retrieve data. Databases power information systems across multiple industries, such as consumer technology (e.g., orders, cancellations, refunds), supply chains (e.g., raw materials, inventory, suppliers), healthcare (e.g., medical records), finance (e.g., financial performance indicators), customer support, and search engines. For modern data-driven enterprises, tracking the real-time state of their business is essential to quickly understand and diagnose any emerging issues, trends, or anomalies in the data and take immediate corrective action. This task is typically performed manually by analysts, who write complex queries in query languages ​​such as SQL, PGQL, logical database queries (e.g., database query languages ​​such as declarative query languages), and API query languages ​​such as GraphQL and REST. Writing such queries can be used to derive insightful information from data stored in multiple tables. These results are typically processed in the form of charts or graphs to allow users to quickly visualize the results and facilitate data-driven decision making.

[0022] While common database queries (e.g., SQL queries) are often pre-defined and built into products, any new or later-created queries still need to be manually coded by analysts. Such static interaction between database queries and the consumption of corresponding results requires time-consuming manual intervention, resulting in a slow feedback cycle. It is far more efficient to allow non-technical users (e.g., business leaders, physicians, or other users of data) to interact directly with analytical tables through natural language (NL) queries that abstract the underlying query language (e.g., SQL) code. Defining database queries requires a deep understanding of database schemas and query language syntax, which can quickly become overwhelming for novices and non-technical stakeholders. Efforts to bridge this communication gap have resulted in the development of a new type of processing called NLIDB. This natural search functionality has become more popular in recent years as companies develop deep learning methods for natural language to logical form (NL2LF) translation, such as natural language to SQL (NL2SQL). Logical forms may refer to semantic and / or machine-oriented languages. NL2SQL seeks to convert natural language questions into SQL, allowing individuals to run unstructured queries against databases. Converted SQL may also enable digital assistants, such as chatbots, to improve their responses when answers can be found in multiple different databases or tables.

[0023] However, a major problem with deep learning methods for NL2SQL is the need for vast amounts of training data to build accurate models. Conventional methods typically ignore this issue and assume the availability of large, manually supervised training datasets (e.g., by using crowdsourcing to generate NL and SQL pairs of data). However, in most cases, collecting and cleaning the data virtually guarantees significant amounts of time, effort, and money. Furthermore, existing methods for NL2SQL translation attempt to build models that generalize to novel and unknown databases, resulting in generally adequate performance that performs poorly when running new queries against the database used for training. That is, training data used to translate queries for one specific database, such as queries containing words and phrases related to food orders, does not always enable the model to generalize to queries in other domains, such as weather or stock quote databases. Furthermore, models for generating database queries from natural language input typically have millions of parameters, given the complexity of the task, making loading and training the model a time-consuming task that uses significant computational resources.

[0024] As mentioned above, multiple database query languages ​​are used, and requests for data may need to be retrieved from a knowledge graph or different types of relational databases. In addition to NL2SQL for generating SQL queries, other models may be developed for generating queries in other languages. However, the significant amount of time, effort, and computational resources required to build a model for converting natural language into a given database query language is multiplied when multiple database query languages ​​are taken into account. For example, traditionally, building a system that can convert natural language input into both SQL and PGQL requires generating a model for generating SQL queries through the time-consuming model building and training process described above, and then starting over again to generate a model for generating PGQL queries through the time-consuming model building and training process. Each additional database query language doubles the time and resources required to develop an appropriate model.

[0025] Therefore, a different approach is needed to address these and other challenges. The approach described herein generates versatile representations that can be used to communicate interchangeably with both humans and a variety of backend systems, including logical, SQL, APIs, REST, GraphQL, PGQL, etc.

[0026] The technology described herein solves these and other problems using a model configured to generate logical forms in an intermediate database query language based on natural language utterances. The training and development of this model does not need to be replicated to adapt to different backends. Rather, the logical forms in the intermediate database query language are translated into the appropriate database query language using an interface specification that specifies the translation from the intermediate database query language to the target database query language (e.g., SQL, PGQL, etc.). Unlike translation models, the interface specification does not require training, and therefore the model can be adapted to a variety of languages ​​with much less computational time and manual data preparation than would be required to generate individual models for multiple different database query languages.

[0027] The conversion of the natural language utterance into a logical form in the intermediate database query language is performed using a specialized machine learning model. The machine learning model is configured to receive as input a concatenation of the natural language utterance and a database schema representation (e.g., specifying tables, columns, etc. in the database schema). The machine learning model includes two encoders and a grammar-based decoder. The first encoder generates embeddings of the natural language utterance and the database schema representation. The second encoder encodes associations between elements in the database schema representation and words in the natural language utterance. Based on the encoded associations and the one or more embeddings, the grammar-based decoder generates an intermediate database query representation. The intermediate database query representation can then be converted into a database query in the target database query language using an interface specification specific to the target database query language.

[0028] As used herein, when an action is "based on" something, this means that the action is at least partially based on at least a portion of that something. As used herein, the terms "similarly," "substantially," "approximately," and "about" are defined as being loosely specified (and being fully specified), as understood by those of ordinary skill in the art. In any disclosed embodiment, the terms "similarly," "substantially," "approximately," or "about" may be substituted for "within [a percentage] of" what is specified, where percentage includes 0.1, 1, 5, and 10 percent.

[0029] Bots and Analytics Systems A bot (also referred to as a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. A bot can generally respond to natural language messages (e.g., questions or comments) through a messaging application using natural language messages. A business can use one or more bot systems to communicate with end users through messaging applications. The messaging application, sometimes referred to as a channel, can be an end user's preferred messaging application that the end user already has installed and is familiar with. Thus, an end user does not need to download and install a new application to chat with a bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat functionality or voice-based input (such as devices or apps with interfaces that use Siri, Cortana, Google Voice, or other voice input for interaction).

[0030] In some examples, a bot system may be associated with a Uniform Resource Identifier (URI). The URI may identify the bot system using a string of characters. The URI may be used as a webhook for one or more messaging application systems. The URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The bot system may be designed to receive a message (e.g., a Hypertext Transfer Protocol (HTTP) POST call message) from the messaging application system. The HTTP POST call message may be directed from the messaging application system to the URI. In some embodiments, the message may be different from an HTTP POST call message. For example, the bot system may receive a message via Short Message Service (SMS). While discussions herein may refer to communications received by the bot system as a message, it should be understood that the message may be an HTTP POST call message, an SMS message, or any other type of communication between two systems.

[0031] End users can interact with bot systems through conversational interactions (sometimes referred to as a conversational user interface (UI)), just like interactions between humans. In some cases, an interaction may involve the end user saying "Hello" to the bot, and the bot responding "Hi" and asking the end user how it can help. In some cases, an interaction may also be a transactional interaction, for example with a banking bot, such as transferring money from one account to another, an informational interaction, for example with an HR bot, such as checking remaining vacation days, or an interaction, for example with a retail bot, such as discussing a purchase return or requesting technical support.

[0032] In some embodiments, the bot system can intelligently handle end-user interactions without interaction with an administrator or developer of the bot system. For example, an end user may send one or more messages to the bot system to achieve a desired goal. The messages may include content, such as text, emojis, audio, images, videos, or other methods of conveying a message. In some embodiments, the bot system can convert the content into a standardized format (e.g., a Representational State Transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system can also initiate communication with the end user rather than passively responding to end-user utterances. Various techniques for identifying explicit invocations of a bot system and determining inputs to the bot system being invoked are described herein. In some embodiments, explicit invocation analysis is performed by a master bot based on detection of an invocation name in the utterance. In response to detection of an invocation name, the utterance can be refined for inputs to a skill bot associated with the invocation name.

[0033] A conversation with a bot may follow a specific conversational flow that includes multiple states. The flow can define what happens next based on input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intent) and actions to take in the states or from state to state. The conversation can take different paths based on end-user input, which can influence the decisions the bot makes for the flow. For example, in each state, based on the end-user input or utterance, the bot can determine the end-user's intent to determine the appropriate action to take next. As used herein and in the context of utterances, the term "intent" refers to the intent of the user who provided the utterance. For example, a user may intend to engage a bot in a conversation to order a pizza, and thus the user's intent may be expressed through the utterance "Order pizza." The user's intent may be directed toward a specific task the user wants the chatbot to perform on their behalf. Thus, an utterance may be phrased as a question, command, request, etc. that reflects the user's intent. Intent may include a goal that the end user wishes to achieve.

[0034] In the context of configuring a chatbot, the term “intent” is used herein to refer to configuration information for mapping a user utterance to a specific task / action or category of task / action that the chatbot can perform. To distinguish between the intent of an utterance (e.g., a user's intent) and the intent of a chatbot, the latter may be referred to herein as a “bot intent.” A bot intent may include a set of one or more utterances associated with the intent. For example, the intent to order a pizza may be communicated by various combinations of utterances expressing a desire to place a pizza order. These associated utterances can be used to train the chatbot's intent classifier, enabling the intent classifier to subsequently determine whether an input utterance from a user matches the intent to order a pizza. A bot intent may be associated with one or more dialog flows to initiate a conversation with a user in a state. For example, the first message of an intent to order a pizza may be the question, “What kind of pizza would you like?” In addition to the associated utterance, a bot intent may further include a named entity associated with the intent. For example, an intent to order a pizza may include variables or parameters used to perform the task of ordering a pizza, such as, for example, topping 1, topping 2, pizza type, pizza size, quantity of pizza, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0035] FIG. 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to one embodiment. The environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as shown in FIG. 1, a user 104 representing a particular business can use the DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, the DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform can be used by multiple businesses to create digital assistants. As another example, a restaurant (e.g., a pizza shop) owner can use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0036] For purposes of this disclosure, a "digital assistant" is an entity that helps a user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity that runs using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or executed in various physical systems or devices, such as a computer, a mobile phone, a watch, an appliance, a vehicle, etc. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0037] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and get responses 112 back from the digital assistant 106. The conversations can include one or more of the inputs 110 and responses 112. Through these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested task and respond to the user with an appropriate response.

[0038] User input 110 is generally in the form of natural language and is referred to as speech. User utterance 110 can be in text form, such as when a user types in a sentence, a question, a text fragment, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user utterance 110 can be in the form of acoustic input or speech, such as when a user says or speaks something that is provided to the digital assistant 106 as input. The speech is typically in the language that the user 108 speaks. For example, the speech may be in English or some other language. When the utterance is in the form of speech, the speech input is converted into textual speech in that particular language, and the textual speech is then processed by the digital assistant 106. Various utterance-to-text processing techniques can be used to convert the utterance or acoustic input into textual speech that is then processed by the digital assistant 106. In some embodiments, the utterance-to-text conversion can be performed by the digital assistant 106 itself.

[0039] The utterance, which may be a text utterance or a spoken utterance, can be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing of the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, including identifying one or more intents and one or more entities that correspond to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 can perform one or more actions or behaviors in response to the understood meaning or intent. For purposes of this disclosure, it is assumed that the utterance is either provided directly by a user 108 of the digital assistant 106 or is a text utterance that is the result of converting an input spoken utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0040] For example, user 108 input can request that a pizza be ordered by providing an utterance such as, "I want to order a pizza." Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action can include, for example, responding to the user with a question requesting user input regarding the type of pizza the user desires to order, the size of the pizza, any toppings on the pizza, etc. The responses provided by digital assistant 106 can also be in natural language format and typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 can perform natural language generation (NLG). For a user ordering a pizza, through a conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the information required for the pizza order so that the pizza is ordered at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0041] At a conceptual level, digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a sequence or pipeline of processing steps including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to be performed in response to the utterance, causing the action to be performed, if appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing can include analyzing the received input utterance to understand the structure and meaning of the utterance, refining and reshaping the utterance to develop a better understandable form (e.g., logical form) or structure of the utterance, etc. Generating a response can include using NLG techniques.

[0042] NLU processing performed by a digital assistant such as digital assistant 106 may include various NLP-related processes such as sentence analysis (e.g., tokenization, lemmatization, identifying utterance part tags for sentences, identifying named entities within sentences, generating dependency trees to represent sentence structure, dividing sentences into clauses, analyzing individual clauses, resolving anaphora, performing chunking, etc.). In some embodiments, the NLU processing, or portions thereof, is performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of input spoken sentences can be identified by processing the sentences using a parser, utterance part tagger, and / or named entity recognizer. In one embodiment, for the English language, parsers, utterance part taggers, and named entity recognizers, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0043] Although the various examples provided in this disclosure show speech in the English language, this is intended as an example only. In some embodiments, the digital assistant 106 is also capable of handling speech in languages ​​other than English. The digital assistant 106 can provide subsystems (e.g., components that perform NLU functions) configured to perform processing for multiple different languages. These subsystems can be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes NLU processing flexible and extensible for each language, including allowing for different orders of processing. Language packs can be provided for individual languages, and the language packs can register a list of subsystems that can be serviced by the NLU core server.

[0044] 1 can be available or accessible to its user 108 through a variety of different channels, including, but not limited to, through an application, through a social media platform, through various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for it, such that the digital assistant can be powered by and accessed on multiple different services simultaneously.

[0045] A digital assistant or chatbot system generally encompasses or is associated with one or more skills. In some embodiments, these skills are individual chatbots (referred to as skillbots) that are configured to interact with a user and perform specific types of tasks, such as tracking inventory, submitting a timecard, creating an expense report, ordering food, checking a bank account, making a reservation, purchasing a widget, etc. For example, in the embodiment shown in FIG. 1 , the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skillbot” and “skillbots,” respectively.

[0046] Each skill associated with a digital assistant helps a user of the digital assistant complete a task through a conversation with the user, where the conversation may include a combination of text or audio input provided by the user and responses provided by a skill bot. These responses may be in the form of text or audio messages to the user and / or may use simple user interface elements (e.g., selection lists) presented to the user for the user to make a selection.

[0047] There are various ways in which skills or skill bots can be associated with or added to a digital assistant. In some cases, skill bots can be developed by a company and then added to a digital assistant using DABP 102. In some cases, skill bots can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In still other cases, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills targeted at a wide range of tasks. Skills offered through the skill store may also expose various cloud services. To add a skill to a digital assistant being created using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and indicate that the selected skill should be added to the digital assistant being created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in a modified form (e.g., a user of DABP102 may select and clone a particular skillbot provided by the skill store, make customizations or modifications to the selected skillbot, and then add the modified skillbot to a digital assistant created using DABP102).

[0048] A variety of different architectures can be used to implement a digital assistant or chatbot system. For example, in one embodiment, a digital assistant created and deployed using DABP 102 can be implemented using a masterbot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a masterbot that interacts with one or more child bots, which are skillbots. For example, in the embodiment shown in FIG. 1, the digital assistant 106 comprises a masterbot 114 and skillbots 116-1, 116-2, e.g., that are child bots of the masterbot 114. In one embodiment, the digital assistant 106 itself can be considered to act as the masterbot.

[0049] A digital assistant implemented according to a master-child bot architecture allows a user of the digital assistant to interact with multiple skills through a unified user interface, i.e., via a master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself. If not, the master bot selects a suitable skill bot to handle the user request and routes the conversation to the selected skill bot. This allows a user to interact with the digital assistant through a common, single interface while still providing the ability to use several skill bots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the digital assistant's master bot may interface with skill bots having specific functions, such as a CRM bot for performing functions related to customer relationship management (CRM), an ERP bot for performing functions related to enterprise resource planning (ERP), an HCM bot for performing functions related to human capital management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master bot interface, and behind the scenes, multiple skill bots are provided to handle user requests.

[0050] In one embodiment, in a masterbot / childbot infrastructure, the masterbot is configured to recognize an available list of skillbots. The masterbot has access to various available skillbots and, for each skillbot, metadata identifying the skillbot's capabilities, including tasks that can be performed by the skillbot. Upon receiving a user request in the form of an utterance, the masterbot is configured to identify or predict a specific skillbot from multiple available skillbots that can best service or address the user request. The masterbot then routes the utterance (or a portion of the utterance) to that specific skillbot for further action. Thus, control flows from the masterbot to the skillbot. The masterbot can support multiple input and output channels. In one embodiment, routing can be performed with the assistance of processing performed by one or more available skillbots. For example, as described below, a skillbot can be trained to infer the intent of an utterance and determine whether the inferred intent matches an intent for which the skillbot is configured. Thus, routing performed by the masterbot may involve the skillbot communicating to the masterbot an indication of whether the skillbot is configured with the appropriate intent to address the utterance.

[0051] 1 illustrates a digital assistant 106 with a masterbot 114 and skillbots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in embodiments using a combination of software and hardware.

[0052] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants, including one or more skillbots associated with the digital assistant. In some cases, a skillbot can be created by cloning an existing skillbot, for example, by cloning a skillbot provided by a skill store. As indicated above, DABP 102 provides a skill store or skill catalog that provides multiple skillbots for performing various tasks. Users of DABP 102 can clone skillbots from the skill store. Modifications or customizations can be made to the cloned skillbot as needed. In some other cases, users of DABP 102 created skillbots from scratch using tools and services provided by DABP 102. As indicated above, the skill store or skill catalog provided by DABP 102 can provide multiple skillbots for performing various tasks.

[0053] In one embodiment, at a high level, creating or customizing a skillbot involves the following steps: (1) Configure the settings for the new skill bot. (2) Configure one or more intents for the skillbot. (3) Constitute one or more entities of one or more intentions. (4) Train the skill bot. (5) Create a dialog flow for the skill bot. (6) Add custom components to your skill bot as needed. (7) Test and deploy the skill bot. Each of the above steps is briefly described below.

[0054] (1) Configuring Settings for a New Skillbot—Various settings can be configured for a skillbot. For example, a skillbot designer can specify one or more invocation names for the skillbot being created. These invocation names can then be used by a user of the digital assistant to explicitly invoke the skillbot. For example, a user can enter the invocation name in the user's utterance to explicitly invoke the corresponding skillbot.

[0055] (2) Configuring One or More Intents and Associated Example Utterances for the Skillbot—A skillbot designer specifies one or more intents (also referred to as bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes that the skillbot is trained to infer for input utterances. Upon receiving an utterance, the trained skillbot infers the intent of the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes an appropriate action to respond to the utterance based on the inferred intent for that utterance. In some cases, the intents of the skillbot represent tasks that the skillbot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot being trained for banking, the intents specified for the skillbot may include "CheckBalance," "TransferMoney," "DepositCheck," etc.

[0056] For each intent defined for a skillbot, the skillbot designer may also provide one or more example utterances that express and illustrate the intent. These example utterances are intended to represent utterances a user might input into the skillbot for that intent. For example, for a CheckBalance intent, example utterances might include, "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account," etc. Thus, various combinations of typical user utterances may be specified as example utterances for an intent.

[0057] The intents and their associated example utterances are used as training data to train the skill bot. A variety of different training techniques may be used. As a result of this training, a predictive model is generated that is configured to receive an utterance as input and output an intent inferred by the predictive model for the utterance. In some cases, the input utterance is provided to an intent analysis engine that is configured to predict or infer the intent of the input utterance using the trained model. The skill bot can then take one or more actions based on the inferred intent.

[0058] (3) Configuring One or More Intent Entities for a Skill Bot—In some cases, additional context may be required for a skill bot to respond appropriately to a user utterance. For example, there may be situations in which a user input utterance resolves to the same intent within a skill bot. For example, in the above example, "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests asking about different things. To disambiguate such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "savings" can enable the skill bot to parse the user request and respond appropriately. In the above example, the utterances resolve to the same intent, but the values ​​associated with the AccountType entity are different for the two utterances. This allows the skill bot to perform potentially different actions for the two utterances, even though they resolve to the same intent. One or more entities may be specified for an intent configured for a skill bot. Thus, entities are used to add context to the intent itself: they help to more fully describe the intent and enable the skill bot to complete the user request.

[0059] In one embodiment, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by a skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, repeating time period, currency, phone number, URL, etc. Custom entities are used for more customized purposes. For example, for a banking skill, an AccountType entity can be defined by the skill bot designer to enable various banking transactions by checking user input for keywords such as checking, savings, and credit cards.

[0060] (4) Train the Skillbot—The skillbot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select an intent associated with the received user input. As indicated above, the skillbot must be trained for this. In one embodiment, the skillbot is trained based on the intents configured for it and example utterances associated with the intents (collectively, training data) so that the skillbot can resolve user input utterances into one of its configured intents. In one embodiment, the skillbot is trained using the training data and uses a predictive model that enables the skillbot to determine what a user says (or in some cases, what they intend to say). DABP 102 provides a variety of different training techniques that can be used by the skillbot designer to train the skillbot, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In one embodiment, a portion (e.g., 80%) of the training data is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as a trained skill bot) can then be used to address and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and no further dialogue. To address such situations, a Q&A (question and answer) intent may be defined for a skill bot. This allows the skill bot to output a response to a user request without having to update the dialog definition. A Q&A intent is created similarly to a regular intent. The dialog flow of a Q&A intent may differ from that of a regular intent.

[0061] (5) Create a Dialog Flow for the Skill Bot—The dialog flow specified for a skill bot describes how the skill bot reacts as its different intents are resolved in response to incoming user input. Dialog flow defines the actions or behaviors the skill bot performs, such as how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. Dialog flow is like a flowchart that the skill bot follows. Skill bot designers specify the dialog flow using a language such as Markdown. In one embodiment, a version of YAML called OBotML can be used to specify the dialog flow for a skill bot. The dialog flow definition for a skill bot acts as a model of the conversation itself, i.e., allowing the skill bot designer to orchestrate the interaction between the skill bot and the user it serves.

[0062] In one embodiment, a skill bot's dialog flow definition contains the following three sections: (a) Context Section (b) Default transition section (c) Status section Context Section - The context section allows the skill bot designer to define variables that will be used within the conversation flow. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to recognize and assert user preferences, etc.

[0063] Default Transition Section - Transitions for a skill bot can be defined in the dialog flow state section or the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there are no applicable transitions defined in a state or when the conditions required to trigger a state transition cannot be met. The default transition section can be used to define routing that allows the skill bot to gracefully handle unexpected user behavior.

[0064] State Section - A dialog flow and its associated behavior are defined as a series of transition states that govern the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality needed at that point in the dialog. Thus, states are built around components. States encapsulate the characteristics specific to the component and define transitions to other states that are triggered after the component executes.

[0065] The state section can be used to address special case scenarios. For example, there may be times within a digital assistant when a user wants to temporarily exit a first skill they're engaged in and provide the user with the option to do something in a second skill. For example, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), the user may want to jump to a banking skill (e.g., the user may want to ensure they have enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action within a first skill can be configured to initiate an interaction with a second, different skill within the same digital assistant and then return to the original flow.

[0066] (6) Adding Custom Components to a Skillbot—As described above, states specified in a skillbot's dialog flow nominate components that correspond to the state and provide the required functionality. Components enable a skillbot to perform its functions. In one embodiment, DABP 102 provides a set of pre-configured components for performing a wide range of functions. A skillbot designer can select one or more of these pre-configured components and associate them with states in the skillbot's dialog flow. A skillbot designer can also create custom or new components using tools provided by DABP 102 and associate the custom components with one or more states in the skillbot's dialog flow.

[0067] (7) Testing and Deploying Skillbots—DABP 102 provides several features that allow skillbot designers to test the skillbots being developed, which can then be deployed and included within a digital assistant.

[0068] While the above description describes how to create a skillbot, similar techniques can also be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without invoking a skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit, which applies when a user signals a desire to exit the current conversation or context within the digital assistant; (2) Help, which applies when a user asks for help or orientation; and (3) UnresolvedIntent, which applies to user input that does not match well with the Exit and Help intents. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information allows the masterbot to select a specific skillbot to handle an utterance.

[0069] At the MasterBot or Digital Assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and associated conversation. The digital assistant makes this determination using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance should be routed to a specific skill for action, addressed by the digital assistant or MasterBot itself via a built-in system intent, or addressed as a different state within the current conversation flow.

[0070] In one embodiment, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skill bot corresponding to the invocation name. In such a scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further action. In the absence of a specific or explicit invocation, in one embodiment, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and the skill bot associated with the digital assistant. The calculated score for the skill bot or system intent represents how likely the user input represents the task the skill bot is configured to perform or represents the system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further action of the user input utterance. In one embodiment, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model of each skill), and a confidence score is determined for each intent. Generally, any intent with a confidence score above a threshold (e.g., 70%) is treated as a candidate intent. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0071] FIG. 2 is a simplified block diagram of a MasterBot (MB) system 200 according to one embodiment. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skillbot invoker 240, and a data store 250. The MB system 200 shown in FIG. 2 is only one example of an arrangement of components within a MasterBot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some other embodiments, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0072] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in a variety of modalities, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 may include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the audio to text using an utterance-to-text converter (not shown) that inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text.

[0073] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. The manner in which the utterance 202 is addressed depends on the language, as each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of the utterance.

[0074] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (e.g., VB), etc. The language parser 214 can also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and lemmatize the words. A lemma is the primary form of a set of words as represented in a dictionary (e.g., "run" is a lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking of compound expressions, such as combining "credit" and "card" into a single expression, "credit_card." The language parser 214 can also identify relationships between words within the utterance 202. For example, in some embodiments, language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., particular nouns) are direct objects, which parts of the utterance are prepositions, etc. The results of the processing performed by language parser 214 form extracted information 205, which, along with utterance 202 itself, is provided as input to MIS 220.

[0075] As indicated above, utterance 202 may contain two or more sentences. For purposes of multiple-intent detection and explicit invocation, utterance 202 may be treated as a single unit even if it contains multiple sentences. However, in some embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210, to identify single sentences among multiple sentences for purposes of multiple-intent analysis and explicit invocation analysis. In general, the results produced by MIS 220 and EIS 230 are substantially the same whether utterance 202 is processed at the individual sentence level or as a single unit containing multiple sentences.

[0076] The MIS 220 determines whether the utterance 202 expresses multiple intents. While the MIS 220 can detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not involve determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the processing to determine whether the intent of the utterance 202 matches a bot intent can be performed by the intent classifier 242 of the MB system 200 or by an intent classifier of a skill bot (e.g., as shown in the embodiment of FIG. 3). The processing performed by the MIS 220 assumes that a bot exists that can address the utterance 202 (e.g., a particular skill bot or the master bot itself). Thus, the processing performed by the MIS 220 does not require knowledge of which bots are in the chatbot system (e.g., the identities of the skill bots registered with the master bot) or which intents are configured for a particular bot.

[0077] To determine that utterance 202 contains multiple intents, MIS 220 applies one or more rules from a set of rules 252 in data store 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include a sentence pattern that indicates the presence of multiple intents. For example, the sentence pattern may include a coordinating conjunction joining two parts of a sentence (e.g., a conjunction), where both parts correspond to separate intents. If utterance 202 matches the sentence pattern, it can be inferred that utterance 202 expresses multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents targeted at different bots or different intents within the same bot). Instead, the utterance may have separate instances of the same intent, such as, for example, "Place a pizza order using payment account X, then place a pizza order using payment account Y."

[0078] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines which portions of utterance 202 are associated with each intent. For each intent that represents an utterance encompassing multiple intents, MIS 220 constructs a new utterance to process separately in place of the original utterance, such as utterance “B” 206 and utterance “C” 208 as shown in FIG. 2 . Thus, original utterance 202 can be split into two or more separate utterances to be addressed at one time. MIS 220 determines which of the two or more utterances should be addressed first using extracted information 205 and / or from an analysis of utterance 202 itself. For example, MIS 220 may determine that utterance 202 includes a marker word indicating that a particular intent should be addressed first. The newly formed utterance corresponding to this particular intent (e.g., one of utterance 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (e.g., or has been temporarily put on hold), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) can then be sent to EIS 230 for processing.

[0079] The EIS 230 determines whether an utterance it receives (e.g., utterance 206 or utterance 208) includes a skillbot's invocation name. In one embodiment, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes the skillbot from other skillbots in the chatbot system. A list of invocation names may be maintained in the data store 250 as part of the skillbot information 254. When an utterance includes words that match the invocation name, the utterance is considered to be an explicit invocation. If the bot is not explicitly invoked, the utterance received by the EIS 230 is considered an implicit invocation utterance 234 and is input to the masterbot's intent classifier (e.g., intent classifier 242) to determine which bot to use to address the utterance. In some cases, the intent classifier 242 determines that the masterbot should address the implicit invocation utterance. In other cases, the intent classifier 242 determines which skillbot to route the utterance to for handling.

[0080] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that a masterbot needs to perform. For example, when there is an explicit call, the masterbot may not need to perform any intent classification analysis (e.g., using intent classifier 242) or a reduced intent classification analysis to select a skillbot. Thus, the explicit call analysis can enable the selection of a particular skillbot without using intent classification analysis.

[0081] There may also be situations where there is overlap in functionality among multiple skill bots. This can occur, for example, when the intents addressed by two skill bots overlap or are very close to each other. In such situations, it may be difficult for the master bot to identify which of the multiple skill bots to select based solely on intent classification analysis. In such scenarios, explicit invocation eliminates ambiguity about the specific skill bot to be used.

[0082] In addition to determining that an utterance is an explicit invocation, EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the skill bot being explicitly invoked. In particular, EIS 230 can determine that a portion of the utterance is not associated with an invocation. EIS 230 can make this determination through analysis of the utterance and / or analysis of extracted information 205. EIS 230 can send the portion of the utterance that is not associated with an invocation to the skill bot being invoked instead of sending the entire sentence received by EIS 230. In some cases, the input to the skill bot being invoked is formed simply by removing any portion of the utterance that is associated with an invocation. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using Pizza Bot" is related to the invocation of Pizza Bot but not to any processing that will be performed by Pizza Bot. In some cases, the EIS 230 can reformat the portion to be sent to the bot being invoked, for example, to form a complete sentence. Thus, the EIS 230 determines not only that there is an explicit invocation, but also what to send to the skill bot when there is an explicit invocation. In some cases, there may not be any text to be entered into the bot being invoked. For example, if the utterance was "Pizza Bot," the EIS 230 may determine that the Pizza Bot is being invoked, but there is no text to be processed by the Pizza Bot. In such a scenario, the EIS 230 can instruct the skill bot invoker 240 that there is nothing to send.

[0083] The skillbot invoker 240 invokes a skillbot in various manners. For example, the skillbot invoker 240 may invoke a bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invocation. The indication 235 may be sent by the EIS 230 along with input for the skillbot being explicitly invoked. In this scenario, the skillbot invoker 240 hands over control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response may be to perform a particular action or to start a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0084] Another manner in which the skillbot invoker 240 may invoke a skillbot is through implicit invocation using an intent classifier 242. The intent classifier 242 may be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 may be trained for multiple different classes, one class for each skillbot. For example, whenever a new skillbot is registered with a masterbot, a list of example utterances associated with the new skillbot may be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. Parameters (e.g., a set of values ​​for the parameters of a machine learning model) produced as a result of this training may be stored as part of the skillbot information 254.

[0085] In one embodiment, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training the machine learning model may involve inputting at least a subset of utterances from example utterances associated with various skill bots and generating, as the output of the machine learning model, a guess as to which bot is the correct bot to address any particular training utterance. For each training utterance, an indication of the correct bot to use for the training utterance may be provided as ground truth information. The behavior of the machine learning model may then be adapted (e.g., through backpropagation) to minimize the difference between the guesses generated and the ground truth information.

[0086] In one embodiment, the intent classifier 242 determines a confidence score for each skill bot registered with the master bot, indicating the skill bot's likelihood of handling an utterance (e.g., an implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skill bot invoker 240 invokes the bot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either the identity of the system intent or the identity of a particular skill bot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain win margin. Imposing such a condition enables routing to a particular skill bot when the confidence scores of multiple skill bots each exceed the threshold confidence score value.

[0087] After a bot is identified based on the evaluation of the confidence score, the skillbot invoker 240 hands over processing to the identified bot. In the case of system intent, the identified bot is a masterbot. Otherwise, the identified bot is a skillbot. Furthermore, the skillbot invoker 240 determines what to provide as input 247 to the identified bot. As indicated above, in the case of explicit invocation, the input 247 can be based on a portion of the utterance that is not associated with the invocation, or the input 247 can be none (e.g., an empty string). In the case of implicit invocation, the input 247 can be the entire utterance.

[0088] The data store 250 includes one or more computing devices that store data used by various subsystems of the masterbot system 200. As described above, the data store 250 includes rules 252 and skillbot information 254. The rules 252 include, for example, rules for determining, by the MIS 220, when an utterance expresses multiple intents and how to split an utterance expressing multiple intents. The rules 252 further include rules for determining, by the EIS 230, which portions of an utterance that explicitly invokes a skillbot to send to the skillbot. The skillbot information 254 includes, for example, invocation names of skillbots in the chatbot system, such as a list of invocation names of all skillbots registered with a particular masterbot. The skillbot information 254 may also include information used by the intent classifier 242 to determine a confidence score for each skillbot in the chatbot system, such as, for example, parameters of a machine learning model.

[0089] 3 is a simplified block diagram of a Skillbot system 300 according to one embodiment. The Skillbot system 300 is a computing system that can run solely in software, solely in hardware, or in a combination of hardware and software. In one embodiment, such as the embodiment shown in FIG. 1, the Skillbot system 300 can be used to run one or more Skillbots within a digital assistant.

[0090] Skillbot system 300 includes MIS 310, intent classifier 320, and conversation manager 330. MIS 310 is similar to MIS 220 of FIG. 2 and provides similar functionality, including being operable to use rules 352 in data store 350 to (1) determine whether an utterance expresses multiple intents, and if so, (2) determine how to split the utterance into separate utterances for each of the multiple intents. In one embodiment, the rules applied by MIS 310 to detect multiple intents and split the utterance are the same as those applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. Extracted information 304 is similar to extracted information 205 of FIG. 1 and can be generated using language parser 214 or a language parser local to Skillbot system 300.

[0091] The intent classifier 320 can be trained similarly to the intent classifier 242 described above in connection with the embodiment of FIG. 2 and as described in more detail herein. For example, in one embodiment, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of example utterances associated with that particular skill bot as training utterances. The ground truth for each training utterance is the particular bot intent associated with the training utterance.

[0092] The utterance 302 can be received directly from a user or can be provided through a masterbot. For example, if the utterance 302 is provided through a masterbot as a result of processing through the MIS 220 and the EIS 230 in the embodiment shown in FIG. 2 , the MIS 310 can be bypassed to avoid repeating processing already performed by the MIS 220. On the other hand, if the utterance 302 is received directly from a user, for example, during a conversation that occurs after routing to a skillbot, the MIS 310 can process the utterance 302 to determine whether the utterance 302 represents multiple intents. If so, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, such as, for example, utterance “D” 306 and utterance “E” 308. If the utterance 302 does not express multiple intents, the MIS 310 forwards the utterance 302 to the intent classifier 320 for intent classification without splitting the utterance 302 .

[0093] The intent classifier 320 is configured to match an incoming utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As explained above, a skillbot can be configured with one or more intents, each including at least one example utterance associated with the intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the masterbot system 200 is trained to determine a confidence score for each individual skillbot and a confidence score for the system intent. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skillbot system 300. While the classification performed by the intent classifier 242 is at the bot level, the classification performed by the intent classifier 320 is at the intent level and is therefore more granular. The intent classifier 320 can have access to intent information 354. For each intent associated with the skillbot system 300, the intent information 354 includes a list of utterances that represent the intent, indicate its meaning, and are typically associated with tasks that can be performed by the intent. The intent information 354 may further include parameters that result from training on this list of utterances.

[0094] The conversation manager 330 receives as an output of the intent classifier 320 an indication 322 of the particular intent identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some cases, the intent classifier 320 is unable to determine any match. For example, if the utterance is targeted to the system intent or the intent of a different skill bot, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value. When this occurs, the skill bot system 300 may refer the utterance to the master bot for action, for example, to route it to a different skill bot. However, if the intent classifier 320 is successful in identifying the intent within the skill bot, the conversation manager 330 begins a conversation with the user.

[0095] The conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute the dialog flow of the identified intent. The state machine may include a default starting state (e.g., the intent is invoked without any additional input) and one or more additional states, each associated with an action to be performed by the skill bot (e.g., completing a purchase transaction) and / or a dialog to be presented to the user (e.g., question, response). Thus, the conversation manager 330 may determine an action / dialog 335 upon receiving an instruction 322 identifying the intent, and may determine the additional action or dialog in response to subsequent utterances received during the conversation.

[0096] Data store 350 includes one or more computing devices that store data used by various subsystems of skillbot system 300. As shown in Figure 3, data store 350 includes rules 352 and intent information 354. In an embodiment, data store 350 can be integrated with a masterbot or digital assistant data store, such as, for example, data store 250 of Figure 2.

[0097] Transforming natural language utterances into an intermediate database query language The C2OMRL system comprises a deep learning model configured to convert natural language (NL) utterances (or conversations within the Oracle Digital Assistant platform) into a logical form in an intermediate database query language, such as Oracle Semantic Representation Language (OMRL). The logical form can be used to generate queries in a particular database query language, which can then be executed to query an existing database. This deep learning model (referred to as the "C2OMRL semantic parser" or "C2OMRL model") is trained using thousands of example pairs (natural language-logical form).

[0098] 4 is a block diagram 400 illustrating an overview of a C2OMRL architecture with an overview of a process for generating a query for a backend interface 406 starting with a natural language (NL) utterance 408, for example, as received through a human interface 402. For example, the human interface 402 can be a chatbot system that receives a spoken utterance and converts it into a text utterance, as described above, or a system where a user types a request in natural language or other suitable interface. The NL utterance 408 can be in the form of part of a conversation (e.g., "Hello, can you tell me how many orders we need to send out tomorrow?" or "Search for all employees with first name starting with 'S' and living in California.").

[0099] The NL utterance 408 is provided to an NL2LF model 410, which converts the NL utterance 408 into an intermediate representation 412. The NL2LF model 410 is a machine learning model that is trained to generate the intermediate representation 412 from the NL utterance 408. The NL2LF model 410 may include multiple layers.

[0100] 4, the NL2LF model 410 is a conversational to oracle semantic representation language (C2OMRL) model for converting conversational utterances into an intermediate oracle semantic representation language (OMRL) representation 412. The NL2LF model 410 may be referred to interchangeably herein as C2OMRL, although it should be understood that the techniques described herein may be applied to models configured to generate other intermediate representation 412 formats.

[0101] The intermediate representation 412 is a logical representation of the utterance that is configured to be convertible into a particular database query language. In some examples, the intermediate representation 412 is in the Oracle Semantic Representation Language (OMRL), which is an intermediate database query language with specialized schema and interface specifications as described herein. While the intermediate representation 412 may be referred to interchangeably herein as OMRL, it should be understood that the techniques described herein can be applied to other intermediate representation 412 formats.

[0102] The intermediate representation 412 can then be transformed into one or more desired query formats, such as SQL 416 using an OMRL2SQL 414 transformation process, or PGQL 420 using an OMRL2PGQL 418 transformation process. The transformed query (e.g., SQL 416 or PGQL 420) represents the concepts presented in the intermediate representation 412 in a manner that conforms to the requirements of the applicable query language. As discussed above, the process of generating a model for generating a back-end interface 406 that interfaces with queries (e.g., SQL 416 or PGQL 420) is time-consuming and involves processing vast amounts of data. By comparison, OMRL2SQL 414, OMRL2PGQL 418, and similar frameworks for transforming the intermediate representation 412 into queries in languages ​​such as SQL 416, PGQL 420, etc. can be constructed and executed relatively quickly. Because such conversion starts from an intermediate representation that is configured to be converted into a database query language, models for performing the conversion 414, 418 do not need to be trained on a large corpus of natural language training samples. Thus, the architecture and techniques described herein can significantly reduce training time for producing a platform for converting natural language utterances into multiple query languages.

[0103] An overview of systems for converting natural language into semantic expression language. FIG. 5 illustrates a C2OMRL system 500 equipped with a machine learning model to enable the conversion of natural language (NL) utterances (e.g., utterances within a digital assistant platform such as those described in connection with FIGS. 1-3) into logical form (LF) descriptions, such as OMRL queries or commands, that can then be executed to query existing systems, such as relational databases. This machine learning model (referred to herein as a “C2OMRL semantic parser” or “C2OMRL model”) is trained with hundreds to thousands of annotated example pairs (natural language-logical form pairs) to convert natural language utterances into logical form descriptions. As shown, an example 505 (a concatenation of a natural language utterance with a database schema, e.g., a set of table and column names) is input into the C2OMRL model 510. The example 505 is first processed by an encoder component 515, which contextually captures a representation of the natural language utterance and the database schema. A decoder 520 then receives the encoded input and predicts a logical form 525 (eg, OMRL, which is an SQL-like query) based on the captured representation of the natural language utterance and the database schema.

[0104] In the C2OMRL model 510, the encoder component 515 includes two encoders: (1) a pre-trained language model (PLM) 530, which is the first encoder, and (2) a relation-aware transformer (RAT) 535. The PLM 530 is used to embed natural language utterances and database schemas when contextually capturing their representations. In one example, a transformer-based PLM called Decoding Enhanced BERT with Separated Attention (DeBERTa) is used as the PLM 530. (See He et al., DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (2021)). Transformer-based PLMs use self-supervised learning to learn universal language representations from large amounts of text data and transfer this knowledge to downstream tasks. These models provide good background knowledge for downstream tasks, avoiding the need to train downstream models from scratch. The RAT 535 encodes associations between so-called entities (table and column words) in the database schema and words in the natural language utterance (these associations are called "schema-linked" associations).

[0105] The decoder 520 is based on a bottom-up generation process (i.e., the bottom-up generation process generates a tree from bottom (leaf nodes) to top (root node)), and the final generation output is an OMRL tree (i.e., a tree-based structure representing the entire OMRL logical form) that can be reconstructed into the OMRL logical form 525 output. The bottom-up generation process is performed using beam search, a heuristic search algorithm that explores a graph by expanding the most promising nodes in a restricted set. The beam search operates in steps (e.g., approximately 10 steps), also called "beam levels." At each step (e.g., "step i"), the beam search algorithm generates F possible subtrees for the input sequence that can be obtained by expanding the current subtree (from step "i-1"), and then selects the top K subtrees to retain (known as the beamwidth) using a conditional probability associated with each subtree. The conditional probabilities are referred to herein as "raw beam scores," and thus the top K intermediate results (considered in the next generation step) are the K intermediate results with the highest raw beam scores. Additional information for the bottom-up generation process can be found in "Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressive Bottom-up Semantic Parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311-324, Online. Association for Computational Linguistics," the entire contents of which are incorporated herein by reference for all purposes. The final decoder 520 output is the subtree with the highest raw beam score in the last step N.

[0106] The predicted OMRL logical form description 525 (i.e., the OMRL tree with the highest raw beam score in the last step N) can then be input to a language converter 540, such as (OMRL2SQL), to convert the semantic expression language into a system language query or command, such as SQL, API, REST, GraphQL, PGQL, etc. The system language query or command can then be used to query or perform operations on a system 545 (e.g., a relational database) and obtain an output 550 as a result of the query or command.

[0107] FIG. 6 shows a block diagram illustrating an embodiment of a model system 600 configured to train and deploy machine learning models (e.g., the NL2LF machine learning models and calibration models described in connection with FIGS. 4-5 ). The model system 600 in this example includes various stages: a training stage 605 for training machine learning models; an NL2LF conversion stage 610 for converting natural language text (e.g., utterances) into a logical form such as OMRL; and a query execution stage 620 for executing the logical form on a system such as a database to obtain results (e.g., answers to queries in the utterance). The training stage 605 builds and trains one or more machine learning models 625a-625n (where “n” represents any natural number) used by other stages (which may be referred to herein individually as models 625 or collectively as models 625). For example, the models 625 may include a first model for converting natural language utterances into a logical form in an intermediate database query language and a second model for converting the logical form into a particular database query language. Still other types of predictive models (e.g., intent classifiers) may be implemented in other examples in accordance with this disclosure.

[0108] In some embodiments, model 625 is configured as an alert parser or NL2LF model such as RAT-SQL (as the encoder) combined with a semi-autoregressive bottom-up semantic parsing model (SMBOP) (as the decoder). Additional information on the RAT-SQL model can be found in "RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers," Wang et al., published in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, the contents of which are incorporated herein by reference in their entirety for all purposes. Additional information for the SMBOP model can be found in Ohad Rubin and Jonathan Berant. 2021. "SmBoP: Semi-autoregressive Bottom-up Semantic Parsing." In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311-324, Online. Association for Computational Linguistics, the entire contents of which are incorporated herein by reference for all purposes.

[0109] To train various models 625, the training stage 605 consists of two main subsystems or services: a dataset preparer 630 and a model trainer 640. The dataset preparer 630 facilitates the process of loading data assets 645, splitting the data assets 645 into training and validation sets (645-a-n) so that the system can train and test the models 625, and performing basic natural language preprocessing (e.g., standardization, normalization, data tokenization, augmentation, etc.). The data assets 645 include natural language utterances (e.g., natural language questions / requests) as input and OMRL logical forms as output obtained from one or more sources, such as from human annotators and / or databases (not shown). In some cases, the utterances are provided by a client or customer. In other cases, the utterances are automatically generated and / or retrieved from a library of utterances (e.g., identifying utterances from a library that are specific to the task the model should learn). The data assets 645 may include input text or acoustics (or input features of text or acoustic frames) and labels 650 corresponding to the input text or acoustics (or input features) as a matrix or table of values. For example, for each utterance, the corresponding label may include an indication of the corresponding logical form (e.g., a description / query such as an OMRL query) that serves as ground truth information for the utterance, and / or an indication of whether the logical form is accurate or inaccurate for the utterance. The behavior of the model may then be adapted (e.g., through backpropagation) to minimize the difference between the generated guesses and the ground truth information.For example, data asset 845 may include the natural language utterance "What is the average life expectancy in the United States of America?" and a label containing the corresponding logical form "SELECT AVG(life_expectancy) FROM country = United States of America" ​​(and possibly whether the predicted logical form was "correct" or "incorrect").

[0110] In some cases, data assets 645 also include database schema information. A database schema defines how data is organized in a database, such as a relational database, and includes logical constraints such as table names, fields, data types, and relationships between these entities. A relational database can be formed from one or more tables, each of which includes one or more columns, and each of which includes one or more values. Each table and column in a relational database can be named by a unique identifier, each of which may include one or more words. In some cases, one or more columns in a relational database can serve as a primary key, and each of the values ​​in the one or more columns serving as a primary key are unique to one another. In some cases, one or more columns in a relational database can serve as a foreign key, which serves to link a table containing one or more columns with another table in the relational database. In some cases, the database schema information includes a unique identifier for one or more tables, a unique identifier for one or more columns, and one or more data structures for storing each relational database value. The unique identifiers and values ​​may be stored in one or more vectors and / or matrices by the dataset preparer 630. In some embodiments, the data structure that stores schema information for the relational database may store a directed graph that represents the unique identifiers and values.

[0111] Once the data asset 645 is obtained, the dataset can be split into a training dataset and a validation dataset. The splitting can be performed randomly (e.g., 90 / 10% or 70 / 30%), or the splitting can be performed according to more complex validation techniques, such as K-fold cross-validation, leave-one-out cross-validation, group leave-one-out cross-validation, nested cross-validation, etc., to minimize sampling bias and overfitting. Before or after the splitting, basic natural language preprocessing can be performed on the data asset 645. In some cases, the preprocessing includes tokenizing the utterances of the data asset 645. Tokenization is the division of phrases, sentences, paragraphs, or entire passages into smaller units, such as individual words or terms. Each of these smaller units is called a token. The smaller units are created by locating boundaries, such as word boundaries, where one word ends and the next word begins. For example, the text "How many employees work for company X" can be tokenized into the words "How," "many," "employees," "work," "for," "company," and "X." These tokens help the model understand the context and develop a model for a given task. There are various tokenization techniques that can be used to perform tokenization based on the language and modeling task. For example, tokenization can be performed using natural language toolkits, whitespace tokenization, dictionary-based tokenization, rule-based tokenization, Keras tokenization, Penn Tree-based tokenization, spaCy tokenization, Moses tokenization, subword tokenization, etc.

[0112] In some cases, tokens in data assets 645 may then be embedded into word embeddings (e.g., contextualized word embeddings). Word embeddings are learned representations of text in which words with the same meaning have similar representations. Word embeddings are generated by embedding techniques in which individual words in a predefined vector space are represented as real-valued vectors so that they can be understood by deep learning algorithms. The embedding technique can be a combined or separate embedding technique, such as including an embedding layer within a deep learning algorithm or using a separate model such as a BERT-based pre-trained language model (e.g., BERT, RoBERTa, DeBERTa). A BERT-based model is a pre-trained language model that learns the deep meaning of words and context using self-supervised learning. The embedding layer is a word embedding that is trained together with a neural network model for a specific natural language processing task, such as natural language-logical form transfer (e.g., natural language-logical form (NL-LF) algorithm).

[0113] Model training generally involves selecting hyperparameters for the model 625 and using an optimization algorithm (e.g., a stochastic gradient descent algorithm or its variants, such as batch gradient descent or mini-batch gradient descent) to find the model parameters that correspond to the best fit between predicted and actual outputs. Hyperparameters are settings that can be tuned or optimized to control the behavior of the model 625. Most models explicitly define hyperparameters that control various aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined and optimized to adapt the model to specific scenarios. For example, hyperparameters may include the number of hidden units of the model, the learning rate of the model, the convolution kernel width, the number of kernels in the model, the top-K results, the number of beam levels N, etc.

[0114] During training by the model trainer 640, an error is calculated as the difference between the actual output (e.g., as dictated by the labels 650) and the predicted output (e.g., as determined by the model 625). The function used to calculate this error is known as the objective function (e.g., a loss function or cost function). The error is a function of the model's internal parameters, such as weights and biases. For accurate predictions, the error must be minimized. To minimize the error, the model parameters are incrementally updated by minimizing the objective function over training examples from the preprocessed data assets 645. The objective function can be constructed to measure the difference between the output predicted using the model and the ground truth, which is annotated to the samples using labels. For example, for a supervised learning-based model, the training objective is to learn a function “h( )” (sometimes referred to as the assumption function) that maps the training input space X to the target value space Y such that h(x) can successfully predict the corresponding value of y (h:X → Y). A variety of different techniques can be used to learn this assumption function. In some machine learning algorithms, such as neural networks, this is done using backpropagation. The current error is typically propagated backward to the previous layer, where it is used to modify the weights and bias values ​​so that the error is minimized. The weights are modified using an optimization function. The optimization function usually calculates the error gradient, i.e., the partial derivative of the objective function with respect to the weights, and the weights are modified in the opposite direction of the calculated error gradient. For example, techniques such as backpropagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), Hebbian learning, etc. are used to update the model parameters to minimize or maximize this objective function. This cycle is repeated until a minimum of the objective function is reached.

[0115] In some cases, the natural language-to-logical forms (NL2LF) algorithm is trained by a model trainer 640 using preprocessed data assets 645 (e.g., tokenized data assets). In some cases, the NL2LF algorithm includes an encoder-decoder neural network. The encoder consists of an input layer and one or more encoding layers. The encoding layer or layers are based on contextualized word embedding techniques such as BERT and its family (e.g., RoBERTa, DeBERTa, etc.). The encoder converts relevant text (and optionally a database schema) into a number / vector representation according to an embedding procedure, preserving terms and connections between words and sentences, so that a machine can grasp patterns associated with any text, decipher the context of a sentence, and optionally learn the relationship between words and a given database schema. The result of the encoder is a state vector or context vector. This state vector becomes the input of the decoder. The decoder consists of an input layer, one or more decoding layers, a dense layer, and an output layer (e.g., a layer with a softmax function). One or more decoding layers may include multiple transformer layers. The current recurrent unit accepts hidden states from the previous recurrent unit. The decoder results in a logical form, such as an OMRL query transformed from the utterance in the preprocessed data asset 645.

[0116] Once the set of model parameters has been identified by the model trainer 640, the model 625 is trained and a verifier is configured to validate the model 625 using a validation dataset. The validation process performed by the verifier involves iterative operations of inputting the validation dataset into the trained model 625 using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, group leave-one cross-validation, nested cross-validation, etc. to tune the model parameters and ultimately find an optimal set of model parameters. Once the optimal set of model parameters is obtained, a reserved test set of data from the validation dataset is input into the trained model 625 to obtain outputs that are evaluated against ground truth values ​​using correlation techniques such as Bland-Altman and Spearman rank correlation coefficients to calculate performance metrics such as error, accuracy, precision, recall, and receiver operating characteristic curves (ROC). In some cases, obtaining, training, and validating data processes within model system 600 can be performed iteratively (tuned) by model trainer 640 until predetermined conditions are met, and a final set of model parameters can be provided by model trainer 640.

[0117] As will be appreciated, other training / validation mechanisms are contemplated and may be implemented within model system 600. For example, model 625 may be trained and model parameters may be tuned on a dataset from a subset of the obtained or filtered dataset, and the dataset from the subset of the obtained or filtered dataset may be used solely to test and evaluate the performance of model 625. Moreover, while the training mechanisms described herein focus on training new model 625, these training mechanisms may also be utilized to fine-tune existing models that have been trained from other datasets. For example, in some cases, model 625 may have been pre-trained using datasets from one or more different modalities or tasks. In those cases, model 625 may be used for transfer learning and retrained / validated using training and validation data as described above.

[0118] The training stage 605 outputs a trained model 625 having an optimized set of model parameters for use in the NL2LF conversion stage 610 or the confidence score calibration stage 615. The NL2LF conversion stage 610 includes a predictor for converting input data 655 (e.g., natural language utterances) into a logical form 660 (e.g., an OMRL query). For example, the predictor inputs the input data 655 to a trained NL2LF model 665 and uses the trained NL2LF model 665 to perform a process for generating the logical form 660 based on features in the input data 655. In some cases, the trained NL2LF model 665 performs one or more semantic parsing tasks to generate predictions based on features extracted from the input data 655. The NL2LF conversion stage 610 outputs the logical form 660, which can be used in the confidence score calibration stage 615 and the query execution stage 620.

[0119] The query execution stage 620 includes one or more executors configured to execute the logical form 660 on a system, such as a database 680, to obtain results 685 (e.g., answers to the queries in the natural language utterances). For example, the one or more executors may be configured to translate or convert the semantic expression language of the logical form 660 into a system language query or command, such as SQL, API, REST, GraphQL, PGQL, etc., and execute the system language query or command against a relational database to obtain an answer to the query posed in the natural language utterance.

[0120] Although not explicitly shown, it will be understood that model system 600 may further include a developer device associated with a developer. Communications from the developer device to components of model system 600 can dictate the type of input data, utterances, and / or database schema to be used for the models, the number and type of models to be used, the hyperparameters of each model, e.g., learning rate and number of hidden layers, how data requests are formatted, what training data should be used (e.g., and how to gain access to the training data), and what validation techniques should be used, and / or how the controller process should be configured.

[0121] Techniques for converting natural language utterances into intermediate database query representations FIG. 7 is a flowchart illustrating a process 700 for using a trained machine learning model to convert natural language into an intermediate database query representation, according to various embodiments. The process illustrated in FIG. 7 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 7 and described below is intended to be exemplary and non-limiting. While FIG. 7 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, steps may be performed in some different order, or some steps may also be performed in parallel. In an embodiment, such as the embodiment illustrated in FIGS. 1-6, the process illustrated in FIG. 7 can be performed by a model system (e.g., model system 600) to create, train, update, and deploy one or more machine learning models configured to convert natural language into a logical form.

[0122] In step 702, an input string is generated by concatenating the natural language utterance with a database schema representation of the database. The database schema representation of the database contains information that defines how data is organized in the database. This may include information such as table names, fields, data types, and relationships between table names, fields, and data types. Examples of OMRL database schema representations are discussed further below in the OMRL Schema section. The natural language utterance is a text string such as "List vendors in order of sales for last month." In some cases, the database schema representation is also in the form of a string.

[0123] In some examples, the database schema representation includes table and column names that indicate the database to be queried. As a specific example, consider a simple database schema with one table (Invoice) with four columns, as shown below:

[0124] [Table 1]

[0125] For the natural language utterance "What are the invoices for supplier Mary", an example model input is represented by the following concatenation of the NL utterance and DB schema:

[0126] [Table 2]

[0127] Because some databases may relate to completely different sets of words (e.g., one for employee data, one for department store inventory, one for scientific data), including a database schema specification 436 in the input 434 helps the model contextualize the utterance to fit the schema.

[0128] As explained further below, OMRL schemas include specialization attributes. One such specialization attribute is the link attribute. Link attributes link entries that may be in different tables. This provides a function somewhat similar to foreign keys and primary keys in SQL. However, unlike their SQL equivalents, link attributes can link to entities in different tables without requiring JOINS or making explicit references to the different tables. In OMRL, link attributes have descriptive names that contain metadata within the OMRL specification. For example, in SQL, a question "QUESTION" concatenated with a database schema representation in SQL would look like this:

[0129] <question>criminal (criminal.perpetrator_id:number criminal.people_id: number criminal.date:text criminal.year:number criminal.location:text criminal. country:text criminal.killed:number criminal.injured:number) people (people. people_id:number criminal.name:text criminal.height:number criminal.weight: number criminal.home_town:text) In OMRL, the equivalent query concatenated with the equivalent database schema representation is:

[0130] <question>discloser (culprit_id:number pe ople:link date:text year:number location:text country:text killed:number injured:number) people (people_id: number name:text height:number weight:number home_town:text criminals:link) Using link attributes, the encoder input is shortened by removing the table name for each attribute input. In other words, in some cases, the database schema representation of a database includes link attributes that reference entries in a table without referencing the name of the table. This improves training and inference latency by allowing shorter encoder inputs.

[0131] In step 704, one or more embeddings of a natural language utterance and a database schema representation are generated by a first encoder based on the input string. The input string is provided as input to the first encoder. In some cases, the first encoder is a pre-trained language model (PLM) 530, such as DeBERTa. The first encoder processes the input string to generate one or more embeddings of a natural language utterance and a database schema representation. This can be achieved using a single embedding or multiple embeddings. As described above, embeddings are learned representations of text in which words with the same meaning have similar representations. Embeddings are generated by embedding techniques in which individual words in a predefined vector space are represented as real-valued vectors, so that they can be understood by deep learning algorithms.

[0132] In step 706, a second encoder encodes associations between elements in the database schema representation and words in the natural language utterance based on the generated embedding(s). Elements in the database schema may include table names, column names, row names, fields, relations, view indexes, directories, and / or other database schema elements. The generated embedding(s) are provided from the first encoder to the second encoder. In some cases, the second encoder is a relation-aware transformer (RAT). The second encoder includes multiple self-attention layers configured to apply self-attention to the one or more embeddings to identify associations between entities in the database schema representation and words in the natural language utterance. The use of a RAT is described in Wang et al., RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (2021), the contents of which are incorporated herein by reference in their entirety for all purposes.

[0133] In some cases, schema link associations linking elements in the database schema representation with words in the natural language utterance are also provided to the second encoder, and embeddings are further generated based on the schema link associations. The schema link associations provide information to help the second encoder identify how elements in the database schema representation are related to words in the natural language utterance. The schema links serve to capture potential links between tokens in the utterance and schemas (e.g., entities / attributes in OMRL or tables / columns in SQL). The schema link associations are encoded into the second encoder layer as prior knowledge.

[0134] In some cases, Name-Based Schema Linking (NBSL) is applied. NBSL works to create matches between tokens in a natural language utterance and elements in a database schema representation. NBSL matches entities such as table and column names with words in the input utterance, which can be based on exact or partial matches to elements in the database schema representation of both the primary name and its synonyms.

[0135] In some instances, the OMRL schema includes rich metadata. This metadata includes information that specifies synonyms for multiple different words. For example, car is a synonym for automobile. Using this rich metadata in the OMRL schema, name-based schema links can identify elements in the database schema representation based on identifying synonyms and identifying exact matches. In other words, the schema link association includes metadata that specifies synonyms for words. In some instances, the OMRL schema includes link attributes as described above with respect to step 702.

[0136] In some cases, content-based schema linking (CBSL) is applied. In CBSL for OMRL, data assets to be used in the natural language-logic formal model are preprocessed based on scalable search and content-based schema linking. The preprocessing technique combines named entity recognition and scalable search (e.g., ElasticSearch) to obtain CBSL matches between words or tokens in the utterance and system entities and / or values ​​of attributes in a given database schema. The content-based schema link matches are attached to the utterance using a unique data structure, which is then input into the natural language-logic formal model. The data structure facilitates the encoding and decoding of the input utterance into a logical form by the natural language-logic formal model. Advantageously, the use of scalable search and content-based schema linking can improve the response time and accuracy of the natural language-logic formal model over traditional standardized programming language techniques, such as SQL searches against databases using SQL similarity operators. CBSL technology is described in further detail in U.S. patent application Ser. No. 18 / 065,387, filed December 13, 2022, entitled "Transforming Natural Language To Structured Query Language Based On Scalable Search And Content-Based Schema Linking," the entire contents of which are incorporated herein by reference for all purposes.

[0137] In step 708, the grammar-based decoder generates an intermediate database query representation based on the encoded relevance and one or more embeddings. As described above with respect to Figures 5 and 6, the grammar-based decoder applies a bottom-up generation process using beam search to generate an OMRL tree representing the complete OMRL logical form. The grammar-based decoder obtains one or more raw beam scores generated from one or more beam levels of the grammar-based decoder. The one or more raw beam scores are used to classify the intermediate database query representation as accurate or inaccurate.

[0138] In some cases, a relational algebra (RA) grammar is further provided to the grammar-based decoder. The RA grammar represents the intermediate database query representation as a tree. The RA grammar expresses the intermediate database query representation using configured RA operators. The RA grammar is used to control the syntax of the decoder output to ensure that the generated logical form has the appropriate syntax to affect subsequent processing steps (including execution against an existing database). Thus, the intermediate database query representation is further based on a relational algebra grammar. In some cases, the specialized RA grammar is adapted to OMRL, as further described below with respect to Figures 8-9B.

[0139] In step 710, a database query in a database query language is generated based on the intermediate database query representation and the interface specification. The interface specification establishes a mapping between the intermediate database query representation and a specific target database query language, such as SQL. The interface specification can be incorporated into a transformer specific to the target database query language, such as an OMRL2SQL transformer. In some cases, the transformer is a model that does not require training (e.g., a non-deep learning model).

[0140] In some cases, a database query is executed against a database to retrieve data in response to a natural language utterance. For example, an SQL query may be executed or a PGQL query may be executed, depending on the backend database of interest. As an example, based on the initial utterance input of "What are the top three earning stores?", a database query may be executed to retrieve a list of the top three earning stores from the database.

[0141] Oracle Semantic Representation Language (OMRL) as an intermediate representation In some cases, the intermediate database query language is the Oracle Semantic Representation Language (OMRL), which is a semantic parser logical form target language that encompasses a wide range of back-end systems, including SQL, PGQL (Oracle's property graph-based query language), decision rule and knowledge graph queries.

[0142] design principle Tabular oriented, no fixed ontology OMRL targets querying and manipulating tabular data, such as databases, decision tables, APIs that generate tabular data, etc. OMRL fits into the ontologies of back-end systems.

[0143] Each backend system is described by a schema that specifies: Table tuple names and slot and column names within each tuple (e.g., dept_name) ● Natural language versions of table and column names (for example, "Department Name") ● The type of values ​​in each column (e.g., numbers, currency, strings, etc.) ● Sample values ​​for each slot or column, or NER type for each column Any additional information required to generate the backend representation (e.g., SQL primary keys and foreign key links) tabular orientation An OMRL query has three parts: ● An optional FROM clause that specifies the data source for the query. No joins (though these can be inferred in the OMRL=>SQL transformation) ● A WHERE clause to filter data as in standard SQL ● An OPERATION clause that specifies the action that should be applied to the filtered data (e.g., SELECT, INSERT, UPDATE, etc.) OMRL consists of the following main components: An OMRL schema defines the OMRL expressions that are valid for an OMRL application (an OMRL version of a database schema such as those used in SQL databases). The OMRL Interface Specification specifies how OMRL is instantiated in C2OMRL and back-end systems.

[0144] o The C2OMRL mapping specification provides synonyms that a particular C2OMRL system may require, links to NER systems, etc. The interface specification also specifies how OMRL is transformed into back-end system expressions such as SQL, PGQL, Logical SQL, etc.

[0145] The separation of the OMRL Schema and the OMRL Interface Specification hides the complexity of the interface components from other interfaces, which can vary depending on the interface. ● If a new kind of C2OMRL system requires additional metadata, that metadata will be added to its C2OMRL interface specification without interfering with existing C2OMRL systems. ● Interfaces to different backends may require different interface specifications.

[0146] OMRL Syntax Design OMRL representations are defined under the JavaScript Object Notation (JSON) format, for example: OMRL Expression { "select": [ <col-value>, ...], "from": <from-value>, "where": <value>, "order_by" : [ <dir-col>, ...], "limit": <count>, "group_by": [ , ...], "having": <value> } These JSON properties are optional, e.g. if there is no WHERE constraint, the "where" property is not required. Entity Types, Attributes and Links ● An entity type corresponds to an SQL table. ● An entity corresponds to a row in an SQL table. ● Attributes correspond to SQL columns. ● A link is a type of attribute that corresponds to a foreign key-primary key pair in SQL.

[0147] :col consists of either a non-link attribute of the entity named in the from: property, or a list consisting of the following: ● Zero or more concatenated link attributes starting with a link from the entity identified by the from: property; ● Followed by the attributes of the entity identified by the last link attribute, or by the from: property if there are no link attributes. ● Example: ["Date"] ● Example: ["PurchasedProduct", "Price"] <col-value>:col-value is a list consisting of an optional aggregation operation followed by a . ● Example: ["#MAX", "PurchasedProduct","Price"] Aggregates begin with the special character '#' which distinguishes the aggregate from the entity name.

[0148] <from-value>:from-value is an entity or an OMRL expression (this allows for nested OMRL expressions). <value>The :where constraint is a Boolean expression. Here are some example values: ● Strings, numbers, and Booleans are values. ● <col-value>list ● Functions of one, two (or n) variables are <value>applies to. ● [ <unary-op> , <value> ] ● [ <binary-op> , <value> , <value>] ● Example: ["=", ["Purchased_Product", "Price"], 23.50] ● Unary operators beginning with a letter <unary-op>is preceded by a "#" and therefore cannot be confused with an entity name (which cannot happen with binary operators). ● Boolean expressions such as AND and OR <binary-op>Use to combine Boolean values.

[0149] <dir-col>:This is used in the order_by clause. This is used when the <col-value>(list), or <col-value>There are two directions: "#ASC" (default) and "#DESC". For example: ● [ "#Desc", "Purchased_Product", "Price"] ● [ "#Desc", "#COUNT", "*"] OMRL Example The following example shows the use of select, from, and where in OMRL. The first element of the list tells which action should be applied, for example Purchase and Products are table names, so this list is a table_column search. > is a binary operation applied to the other argument. {"select": [ ["Date"], ["PurchasedProduct", "Product_Name"]], "from": "Purchase", "where": [">", ["Quantity"], 10]}. The equivalent SQL backend is: SELECT Purchase.Date, Products.Product_Name FROM Purchase JOIN Products ON Purchase.Product_ID = Product.ID WHERE Purchase.Quantity > 10 The following example shows the use of order by and limit in OMRL. {"select": [["PurchasedProduct", "Product_Name"], ["Quantity"]], "from": "Purchase", "order_by": [["Quantity"]], "limit": 2} In contrast, the following example shows the equivalent SQL backend query: SELECT Products.Product_Name, Purchase.Quantity FROM Purchase JOIN Products ON Purchase.Product_ID = Purchase.ID ORDERED BY Purchase.Quantity LIMIT 2 The following example shows the use of group by and having in OMRL. {"select": [["PurchasedProduct", "Product_Name"], ["Quantity"]], "from": "Purchase", "group_by": [["PurchasedProduct", "Product_Name"]], "having": [">", [#SUM, "Quantity"], 10]} In contrast, the following example shows the equivalent SQL backend query: SELECT Products.Product_Name, Purchase.Quantity FROM Purchase JOIN Products ON Purchase.Product_ID = Purchase.ID GROUP BY Products.Product_Name HAVING SUM(Purchase.Quantity) > 10 OMRL Schema In some cases, developers provide an OMRL schema file that specifies the interface between the C2OMRL system and the backends that C2OMRL can interface with. This design is similar to the Spider tables.json schema file.

[0150] The example below shows a simple OMRL schema, which has two tables that record purchases and products. { "entities" : [ { "entity" : "Product", "attributes" : [ { "attribute": "ProductID", "type" : "number"}, { "attribute" : "Product_Name", "type" : "string"}, { "attribute" : "ProductPurchases", "type" : "1-to-many_link", "to" : "Purchase"} ] }, { "entity" : "Purchase", "attributes" : [ { "attribute" : "Date", "type" : "string"}, { "attribute" : "Quantity", "type" : "number"}, { "attribute" : "PurchasedProduct", "type" : "many-to-1_link", "to" : "Product"} ] }] } This schema consists of the following schema: It corresponds to an SQL database with Products (ProductID, Product_Name), Purchase (Date, Quantity, PurchasedProductID), and there is a foreign key-primary key relationship between PurchasedProductID and ProductID.

[0151] This schema also corresponds to a PGQL property graph with the following vertices (entities) and edges (links):

[0152] Products(ProductID, Product_Name), Purchase(Date, Quantity) Purchase -(PurchasedProduct)-> Product link A link is a special kind of attribute that points to another entity (for example, an entry in another table). There are two kinds of links: one-to-many_links and many-to-one_links, which correspond to SQL primary key-foreign key and SQL foreign key-primary key relationships, respectively.

[0153] Links typically occur in pairs (because they correspond to a single SQL foreign key-primary key relationship), for example, ProductPurchases is the reverse link of PurchasedProduct. The reverse_link property can be used to encode which link is the reverse of which other link. The reverse_link property is not needed when there is only one way to connect two entities. While SQL foreign keys and primary keys relate entries in different tables using a JOIN operation, OMRL link attributes do not require a JOIN operation. These improved link attributes are used in the RAT functions and RA grammars used in the OMRL pipeline.

[0154] The example below shows an OMR schema with multiple links: Air flights involve a Source and Destination airport. Flights are represented by an entity type and Airports are represented by an entity type. There are two links from the Flights entity to the Airports entity, one link for the departure airport and one link for the destination airport. This distinguishes between these airports. { "entities": [ { "entity" : "Airports", "attributes" : [ { "attribute" : "City", "type" : "string"}, { "attribute" : "AirportCode", "type" : "string"} { "attribute" : "DepartingFlights", "type" : "1-to-many_link", "to" : "Flights", "reverse_link" : "DepartureAirport"}, { "attribute" : "ArrivingFlights", "type" : "1-to-many_link", "to" : "Flights", "reverse_link" : "DestinationAirport"}, ]}, { "entity" : "Flights", "attributes" : [ { "attribute" : "FlightNumber", "type" : "number"}, { "attribute" : "DepartureAirport", "type" : "many-to-1_link", "to" : "Airports", "reverse_link" : "DepartingFlights"}, { "attribute" : "DestinationAirport", "type" : "many-to-1_link", "to" : "Airports", "reverse_link" : "ArrivingFlights"}, ]} ]} The reverse_link JSON property on a link indicates which link is the reverse of which link.

[0155] The example below shows the SQL and OMRL representation of "How many flights arriving in Aberdeen city?" SQL: SELECT COUNT( * ) FROM flights JOIN airports ON flights.destairport = airports.airportcode WHERE airports.city = 'Aberdeen' The OMRL2SQL converter needs to resolve the SourceAirport and DestAirport links to JOIN on the appropriate foreign keys. OMRL: A table column operation is replaced with a list of table, column, or link names. {"select" : [ ["#COUNT", "*"]], "from" :"Flights", "where" : ["=", ["DestinationAirport", "City"], "Aberdeen"]} It uses the first element of the list to tell which kind of action should be applied.

[0156] The example below shows an OMRL representation of "Where can I fly to from Sydney?" { "select" : "City", "from" :"Airports", "where" : ["=", ["ArrivingFlights", "DepartingFrom", "City"], "Sydney"] } OMRL example and schema with three tables This example extends the previous airport and flight example by adding an airline entity. OMRL Schema: { "entities": [ { "entity" : "Airports", "attributes" : [ { "attribute" : "City", "type" : "string"}, { "attribute" : "AirportCode", "type" : "string"} { "attribute" : "DepartingFlights", "type" : "1-to-many_link", "to" : "Flights", "reverse_link" : "DepartureAirport"}, { "attribute" : "ArrivingFlights", "type" : "1-to-many_link", "to" : "Flights", "reverse_link" : "DestinationAirport"}, ]}, { "entity" : "Airline", "attributes" : [ { "attribute" : "Name", "type" : "string"}, { "attribute" : "Code", "type" : "string"}, { "attribute" : "AirlineFlights", "type" : "1-to-many_link", "to" : "Flights", "reverse_link" : "FlightAirline"} ]}, { "entity" : "Flights", "attributes" : [ { "attribute" : "FlightNumber", "type" : "number"}, { "attribute" : "FlightAirline", "type": "many-to-1_link", "to": "Airline", "reverse_link": "AirlineFlights" }, { "attribute" : "DepartureAirport", "type" : "many-to-1_link", "to" : "Airports", "reverse_link" : "DepartingFlights"}, { "attribute" : "DestinationAirport", "type" : "many-to-1_link", "to" : "Airports", "reverse_link" : "ArrivingFlights"}, ]} ]} The example below shows an OMRL representation of "Which airlines fly from Sydney to San Franciso?" { "select" : ["FlightAirline","Name"] "from" : "Flights", "where" : ["AND", ["=", ["DepartureAirport", "City"], "Sydney"], ["=", ["DestinationAirport", "City"], "San Francisco"]]. } The Role of Semantic Parsers The semantic parser is responsible for generating the OMRL, which consists of SELECT, FROM, WHERE, ORDER BY, GROUP BY, HAVING, and LIMIT clauses. The minimum requirements for an OMRL query are SELECT and FROM. In many cases, SELECT will simply be "*". Other clauses may be added if the NL query explicitly includes a request for them. The semantic parser will include in the OMRL those entities and attributes that were explicitly requested in the NL query. If the semantic parser does not find any attributes in the NL query, it will include all attributes (SELECT *). The semantic parser will generate predicates for the attributes of the entities based only on the conditions found in the NL query. These predicates support standard comparison operators, for example (=, !=, <, >, <=, >=, LIKE, NOT LIKE).

[0157] In some cases, the semantic parser includes an ORDER BY clause in OMRL if it is explicitly specified in the NL query via a superlative such as "top", an explicit ordering such as "by xxx", or if the semantic parser finds a superlative such as "top" or "best", this includes ORDER BY * DESC.

[0158] The role of transformation components 4-7, a transformation component transforms an intermediate database query expression (e.g., in OMRL) into a database query in a particular database query language (e.g., SQL). The following examples refer to the OMRL2SQL component, but it should be understood that these examples may apply to a transformation component configured for another target database query language, such as PGQL.

[0159] The OMRL2SQL component will append default clauses to the final SQL statement if the source ORML query does not include that clause, with the exception of a default Where clause, which is applied whether or not it is included in the OMRL query. If the OMRL SELECT includes all attributes of an entity (SELECT *) and the OMRL does not include an ORDER BY clause, the entity's default order-by will be applied. If the OMRL SELECT includes a single attribute and does not include an ORDER BY, the results shall be ordered based on the natural sort order of the requested attributes.

[0160] When an OMRL SELECT explicitly includes multiple attributes and does not include an ORDER BY, the results are ordered based on the natural sort order of each requested attribute in the order in which they appear in the SELECT clause. For example, if the user says "show me employees and the locations where they work", the results will be ordered by employee, location. If the user says "show me locations and the employees that work there", the results will be sorted by location, employee. "DESC" is used to indicate "Top" in the order-by clause. The OMRL specification includes a "superlativeOrder" attribute. The value of this attribute can be used by the OMRL2SQL component for the actual sort order if "DESC" is found.

[0161] Consumption of OMRL specifications As mentioned above, in OMRL there are no foreign keys and primary keys as there are in standard SQL. Instead, OMRL introduces so-called "link attributes". Table 3 shows the terminology used in the OMRL specification. Table 4 shows examples of OMRL link attributes compared to SQL foreign keys and primary keys.

[0162] [Table 3]

[0163] [Table 4] TIFF2025529879000006.tif226162TIFF2025529879000007.tif239162 TIFF2025529879000008.tif234162

[0164] OMRL RAT functionality Specialized RAT functionality is used in OMRL as described above in connection with Figure 7. Table 5 shows an example RA grammar for OMRL.

[0165] [Table 5]

[0166] One specialized RA operation in OMRL is the Cartesian product operator. As mentioned above, OMRL uses link attributes to identify related entries in different tables. Unlike SQL, which uses the JOIN operation, in OMRL the link attribute is instead used to identify entries in different tables without the need to join tables. The Cartesian product operator represents the link attribute in OMRL. When there are multiple links in a link attribute, they are presented in a binary subtree working from left to right. Other specialized RA operations in OMRL include the like start / not like start and like end / not like end operators. These operators are intended to address fuzzy searches, as explained in more detail below.

[0167] Figure 8 shows an example of an RA tree illustrating link capabilities 800. For the utterance "What is the ID and name of the employee with the highest salary?", the OMRL logical form is:

[0168] OMRL: {'select': [['eid'], ['name']], 'from': 'employee', 'order_by': [['#DESC', ['salary']]], 'limit': 1} This is represented by the RA tree 800 shown in FIG.

[0169] Fuzzy search function As mentioned above, the OMRL RA grammar has been adapted to support a fuzzy search capability. A fuzzy search is a search that goes beyond an exact match and finds a term that closely matches another term. A fuzzy search uses a pattern matching algorithm to identify such approximate matches. The fuzzy search capability has been extended to further support prefix / suffix searches using the like operator. As an example, using like (standard), for the NL question "What is all the information about employees with D or S in their first name, ordered by salary descending?" an SQL-like query would look like this:

[0170] SELECT * FROM employees WHERE FIRST_NAME LIKE '%D%' OR FIRST_NAME LIKE '%S%' ORDER BY SALARY DESC The equivalent OMRL query using standard like is:

[0171] OMRL: {'select': [['*']], 'from': 'employees', 'where': ['OR', ['LIKE', ['FIRST_NAME'], '%d%'], ['LIKE', ['FIRST_NAME'], '%s%']], 'order_by': [['#DESC', ['SALARY']]]} The following example shows the use of like with leading support: For the NL question "Which catalog contents have a product stock number that starts from '2'? Show the catalog entry names", the SQL-like query is:

[0172] SELECT catalog_entry_name FROM Catalog_Contents WHERE product_stock_number LIKE '2%' The equivalent OMRL query using like with head support is:

[0173] OMRL: {'select': [['catalog_entry_name']], 'from': 'Catalog_Contents', 'where': ['LIKE', ['product_stock_number'], '2%']} The difference from like (standard) is LIKE '2%' (the value has % at the end). The following example shows the use of like with trailing support: For the NL question "Return the full names and salaries for employees with first names that end with the letter m", the SQL-like query is:

[0174] SELECT FIRST_NAME, LAST_NAME, SALARY FROM employees WHERE FIRST_NAME LIKE '%m' The equivalent OMRL query using like with tail support is:

[0175] OMRL: {'select': [['FIRST_NAME'], ['LAST_NAME'], ['SALARY']], 'from': 'employees', 'where': ['LIKE', ['FIRST_NAME'], '%m']} The Like_start and Like_end operators are specialized RA operators for fuzzy search. Like_start is similar to like, but supports fuzzy search starting with a string. An example RA tree with Like_start 900 is shown in Figure 9A. For the utterance "Which catalog contents have a product stock number that starts from '2'? Show the catalog entry names", the corresponding OMRL is {'select': [['catalog_entry_name']], 'from': 'Catalog_Contents', 'where': ['LIKE', ['product_stock_number'], '2%']} and the transformed RA tree is shown in Figure 9A.

[0176] Like_end is also similar to like, but supports fuzzy searches starting with a string. An example RA tree with Like_end 950 is shown in Figure 9B. For the utterance "Return the full names and salaries for employees with first names that end with the letter m", the corresponding OMRL is {{'select': [['FIRST_NAME'], ['LAST_NAME'], ['SALARY']], 'from': 'employees', 'where': ['LIKE', ['FIRST_NAME'],'%m']} and the transformed RA tree is shown in Figure 9B.

[0177] Additional features of OMRL include improved attribute selection by redefining schema link components and input encoding. Additionally, OMRL implements design picklists for numbers and currencies, which improves the accuracy of numbers and currencies.

[0178] Experimental results The experimental results are shown in Tables 6 and 7. The experiments in Table 6 were conducted using Spider training data (converted to OMRL).

[0179] [Table 6]

[0180] Model accuracy is the exact match accuracy (%) of the C2OMRL model on the evaluation dataset. The evaluation sets include a targeted evaluation set to evaluate the impact of new operators and a standard evaluation set to evaluate whether the new operators have side effects on other functions. Table 7 shows experimental results using a dataset targeted to evaluate the impact of specialized RA operators. As shown in Tables 6 and 7, the techniques described herein provide significant accuracy improvements over conventional systems.

[0181] [Table 7]

[0182] Exemplary System 10 shows a simplified diagram of a distributed system 1000. In the illustrated example, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 can be configured to run one or more applications.

[0183] In various examples, server 1012 can be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 1012 can also provide other services or software applications, which may include non-virtual and virtual environments. In some examples, these services can be provided to users of client computing devices 1002, 1004, 1006, and / or 1008 as web-based or cloud services, such as based on a software-as-a-service (SaaS) model. Users operating client computing devices 1002, 1004, 1006, and / or 1008 can then utilize one or more client applications to interact with server 1012 and utilize the services provided by these components.

[0184] 10 , server 1012 may include one or more components 1018, 1020, and 1022 that perform functions performed by server 1012. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. Note that a variety of different system configurations are possible that may differ from distributed system 1000. Thus, the example shown in FIG. 10 is one example of a distributed system for implementing an exemplary system and is not intended to be limiting.

[0185] A user may use client computing devices 1002, 1004, 1006, and / or 1008 to execute one or more applications, models, or chatbots that may generate one or more events or models that may then be executed or serviced in accordance with the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user through this interface. Although FIG. 10 shows only four client computing devices, any number of client computing devices may be supported.

[0186] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems such as Google Chrome™ OS), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®). Portable handheld devices may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays, and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.), etc. The client devices may be capable of running a variety of different applications, such as various Internet-related apps, communication applications (e.g., email applications, short message service (SMS) applications), etc., and may use a variety of communication protocols.

[0187] Network 1010 may be any type of network familiar to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, network 1010 may be a local area network (LAN), an Ethernet-based network, a Token-Ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating according to any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0188] The servers 1012 may be comprised of one or more general-purpose computers, specialized server computers (including, by way of example, PC (personal computer) servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable configuration and / or combination. The servers 1012 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as an elastic pool of one or more logical storage devices that may be virtualized to maintain virtual storage devices for the servers. In various examples, the servers 1012 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0189] The computing systems within server 1012 may run one or more operating systems, including any of those described above, as well as any commercially available server operating system. Server 1012 may also run any of a variety of additional server and / or middle-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA server, a database server, etc. Exemplary database servers include, without limitation, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc.

[0190] In some embodiments, server 1012 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. By way of example, the data feeds and / or event updates may include real-time updates received from one or more third-party sources and continuous data streams, which may include, without limitation, Twitter® feeds, Facebook® updates, or real-time events related to sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 1012 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0191] The distributed system 1000 may also include one or more data repositories 1014, 1016. These data repositories, in some examples, may be used to store data and other information. For example, one or more of the data repositories 1014, 1016 may be used to store information related to chatbot performance or information such as generated models for use by a chatbot used by the server 1012 in performing various functions according to various embodiments. The data repositories 1014, 1016 may reside in various locations. For example, the data repository used by the server 1012 may be local to the server 1012 or may be remote from the server 1012 and communicate with the server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 may be of different types. In some examples, the data repository used by the server 1012 may be a database, such as a relational database, such as databases provided by Oracle Corporation® and other suppliers. One or more of these databases may be adapted to allow the storage, updating, and retrieval of data to and from the database in response to SQL-formatted commands.

[0192] In some examples, one or more of the data repositories 1014, 1016 may also be used by an application to store application data. The data repositories used by the application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general storage repository backed by a file system.

[0193] In some examples, the functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 11 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services, according to some examples. In the example shown in FIG. 11, a cloud infrastructure system 1102 may provide one or more cloud services that may be requested by users using one or more client computing devices 1104, 1106, and 1108. The cloud infrastructure system 1102 may comprise one or more computers and / or servers, which may include those described above for server 1012. The computers in the cloud infrastructure system 1102 may be organized as general-purpose computers, specialized server computers, server farms, server clusters, or any other suitable configuration and / or combination.

[0194] The network 1110 can facilitate communication and the exchange of data between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network 1110 can include one or more networks. The networks can be of the same or different types. The network 1110 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0195] The example shown in Figure 11 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be noted that in some other examples, cloud infrastructure system 1102 may have more or fewer components than shown in Figure 11, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 11 shows three client computing devices, in alternative examples, any number of client computing devices may be supported.

[0196] The term cloud service is generally used to refer to services made available to users on demand and over a communications network, such as the Internet, by a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from a customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Thus, customers can use cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system may host applications, and users can order and use the applications as needed over the Internet, without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, Java cloud services, etc., are offered by Oracle Corporation of Redwood Shores, California.

[0197] In an example, cloud infrastructure system 1102 may provide one or more cloud services using various models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including hybrid service models. Cloud infrastructure system 1102 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0198] The SaaS model allows applications or software to be delivered to customers as a service over a communications network, such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 1102. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human capital / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, etc.

[0199] The IaaS model is generally used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) as cloud services to customers to provide elastic computing and storage capabilities. Various IaaS services are offered by Oracle Corporation.

[0200] The PaaS model is generally used to provide platform and environment resources as a service, allowing customers to develop, run, and manage applications and services without the customer having to acquire, build, or maintain such resources. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Services (JCS), Oracle Database Cloud Services (DBCS), data management cloud services, various application development solution services, etc.

[0201] Cloud services are generally provided in an on-demand, self-service manner, on a subscription basis, elastically scalable, and highly reliable, available, and secure. For example, a customer may order one or more services provided by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent), as described above, and / or provide the services of a chatbot system as described herein. Cloud infrastructure system 1102 may be configured to provide one or even more cloud services.

[0202] Cloud infrastructure system 1102 can provide cloud services through several different deployment models. In a public cloud model, cloud infrastructure system 1102 may be owned by a third-party cloud service provider, and cloud services are offered to any public customer, where the customer may be an individual or a business. In another example, under a private cloud model, cloud infrastructure system 1102 may operate within an organization (e.g., within a corporate organization) and services offered to customers within the organization. For example, customers may be various departments of a company, such as a human resources department, a payroll department, etc., or even individuals within the company. In another example, under a community cloud model, cloud infrastructure system 1102 and the services offered may be shared by several organizations within an associated community. Various other models, such as hybrids of the above models, may also be used.

[0203] Client computing devices 1104, 1106, and 1108 may be of different types (e.g., client computing devices 1002, 1004, 1006, and 1008 shown in FIG. 10 ) and may be capable of running one or more client applications. Users may use the client devices to interact with cloud infrastructure system 1102, such as to request services provided by cloud infrastructure system 1102. For example, users may use the client devices to request information or actions from a chatbot, as described in this disclosure.

[0204] In some examples, the processing performed by cloud infrastructure system 1102 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing the data in parallel, performing simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 1102 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0205] 11 , cloud infrastructure system 1102 may include infrastructure resources 1130 utilized to facilitate the provision of various cloud services provided by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc. In one example, a storage virtual machine available to service storage requested by an application may be part of cloud infrastructure system 1102. In other examples, the storage virtual machine may be part of a different system.

[0206] In one example, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 1102 to different customers, resources can be bundled into sets of such resources or resource modules (also referred to as “pods”). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In one example, different pods can be pre-provisioned for different types of cloud services. For example, a first set of pods can be provided for database services, a second set of pods can be provided for Java services, and so on, which can include a different combination of resources than the pods in the first set of pods. For some services, resources allocated to provide the service can be shared between services.

[0207] Cloud infrastructure system 1102 may itself use services 1132 internally that are shared by different components of cloud infrastructure system 1102 and that facilitate the provision of services by cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and restore services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0208] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 11 , the subsystems may include a user interface subsystem 1112 that allows users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 where cloud services offered by cloud infrastructure system 1102 are advertised and available for purchase by consumers, and other interfaces 1118. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 1102 using one or more of interfaces 1114, 1116, and 1118 (service request 1134). For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 1102, and place a subscription order for one or more services offered by cloud infrastructure system 1102 to which the customer wishes to subscribe. The service request may include information identifying a customer and one or more services to which the customer wishes to subscribe. For example, a customer may place a subscription order for a service provided by cloud infrastructure system 1102. As part of the order, the customer may provide information identifying a chatbot system to which the service will be provided, and optionally one or more authentication credentials for that chatbot system.

[0209] 11 , cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this processing, OMS 1120 may be configured to create an account for the customer if not already done so, receive billing and / or accounting information from the customer that will be used to bill the customer for providing the customer with the requested services, verify the customer information, and upon verification, register the customer's order, and coordinate various workflows to prepare the order for delivery.

[0210] Upon proper validation, the OMS 1120 may then invoke an order fulfillment subsystem (OPS) 1124, which is configured to provide resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provided for the order and the type of resources provided may depend on the type of cloud service ordered by the customer. For example, according to one workflow, the OPS 1124 may be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for that specific cloud service. The number of pods allocated to the order may depend on the size / amount / level / scope of the service being requested. For example, the number of allocated pods may be determined based on the number of users to be supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized to provide the requested service for the specific requesting customer.

[0211] In one example, the setup phase process described above may be performed as part of the provisioning process by cloud infrastructure system 1102. Cloud infrastructure system 1102 may generate an application ID and select a storage virtual machine for the application from among storage virtual machines provided by cloud infrastructure system 1102 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 1102.

[0212] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service is ready for use at that time. In some cases, information (e.g., a link) may be sent to the customer that enables the customer to begin using and taking advantage of the requested service. In one example, for the customer requesting the service, the response may include a chatbot system ID generated by cloud infrastructure system 1102 and information identifying the chatbot system selected by cloud infrastructure system 1102 for the chatbot system corresponding to the chatbot system ID.

[0213] Cloud infrastructure system 1102 may provide services to multiple customers. For each customer, cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1102 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0214] Cloud infrastructure system 1102 can provide services to multiple customers in parallel. Cloud infrastructure system 1102 can store information for these customers, possibly including sensitive information. In one example, cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage customer information and provide separation of managed information so that information associated with one customer is not accessible by another customer. IMS 1128 can be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0215] 12 illustrates an example of a computer system 1200. In some examples, the computer system 1200 can be used to run a digital assistant or chatbot system in a distributed environment, as well as any of the various servers and computer systems described above. As shown in FIG. 12, the computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 may include a non-transitory computer-readable storage medium, including a storage medium 1222, and a system memory 1210.

[0216] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. While bus subsystem 1202 is shown schematically as a single bus, alternative examples of a bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, which may implement a Mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, etc.

[0217] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may be single-core or multi-core processors. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1204 may include one or more dedicated coprocessors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 1204 may be implemented using custom circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0218] In some examples, processing units within processing subsystem 1204 can execute instructions stored in system memory 1210 or on computer-readable storage medium 1222. In various examples, the processing units can execute various program or code instructions and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code being executed can reside in system memory 1210 and / or on computer-readable storage medium 1222, which in some cases includes one or more storage devices. Through appropriate programming, processing subsystem 1204 can provide the various functions described above. In instances where computer system 1200 is running one or more virtual machines, one or more processing units can be allocated to each virtual machine.

[0219] In an example, a processing acceleration unit 1206 may optionally be provided to perform customized processing or to offload portions of the processing performed by the processing subsystem 1204 so as to accelerate the overall processing performed by the computer system 1200.

[0220] I / O subsystem 1208 may include devices and mechanisms for inputting information into computer system 1200 and / or outputting information from or through computer system 1200. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, acoustic input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion-sensing and / or gesture-recognition devices such as a Microsoft Kinect® motion sensor, a Microsoft Xbox® 360 game controller, or devices that provide an interface for receiving input using gesture and voice commands, allowing a user to control and interact with the input device. The user interface input device may also include an eye gesture recognition device, such as a Google Glass® blink detector, that detects eye activity from the user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gesture as input to the input device (e.g., Google Glass®). Additionally, the user interface input device may include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., Siri® Navigator) through voice commands.

[0221] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, and eye-tracking devices. Additionally, user interface input devices may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0222] In general, use of the term output device(s) is intended to include all possible types of devices and mechanisms for outputting information from computer system 1200 to a user or to another computer. User interface output devices may include non-visual displays such as display subsystems, indicator lights, or acoustic output devices. Display subsystems may be flat-panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. For example, user interface output devices may include various display devices that visually convey text, graphics, and audio / video information, such as, but not limited to, monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0223] The storage subsystem 1218 provides a repository or data store for storing information and data used by the computer system 1200. The storage subsystem 1218 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data structures that provide some example functionality. The storage subsystem 12112 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 can also provide authentication according to the teachings of the present disclosure.

[0224] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 12, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include several memories, including volatile main random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 1200, such as during start-up, may typically be stored in ROM. The RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 1204. In some embodiments, the system memory 1210 may include several different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0225] 12, system memory 1210 may load executing application programs 1212, program data 1214, and operating system 1216, which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc. By way of example, operating system 1216 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0226] The computer-readable storage medium 1222 can store programming and data structures that provide some example functionality. The computer-readable medium 1222 can provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1200. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above can be stored in the storage subsystem 1218. By way of example, the computer-readable storage medium 1222 can include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, and an optical disk drive such as a Blu-ray® disk or other optical media. The computer-readable storage medium 1222 can include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash device, a Secure Digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, etc., DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0227] In some examples, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may be further connected to a computer-readable storage medium 1222. Reader 1220 may receive data from a memory device such as a disk, flash drive, or the like and may be configured to read the data.

[0228] In some examples, computer system 1200 may support virtualization technology, including, but not limited to, virtualization of processing and memory resources. For example, computer system 1200 may provide support for running one or more virtual machines. In some examples, computer system 1200 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may be allocated memory resources, computational resources (e.g., processors, cores), I / O resources, and networking resources. Each virtual machine generally operates independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines executed by computer system 1200. Thus, multiple operating systems may potentially be run simultaneously by computer system 1200.

[0229] The communications subsystem 1224 provides an interface to other computer systems and networks. The communications subsystem 1224 serves as an interface for receiving data from other systems and transmitting data from the computer system 1200 to other systems. For example, the communications subsystem 1224 can enable the computer system 1200 to establish a communications channel to one or more client devices over the Internet to receive and transmit information from the client devices. For example, when the computer system 1200 is used to execute the bot system 120 shown in FIG. 1, the communications subsystem can be used to communicate with a chatbot system selected for the application.

[0230] The communications subsystem 1224 can support both wired and / or wireless communications protocols. In certain examples, the communications subsystem 1224 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some examples, the communications subsystem 1224 can provide wired network connections (e.g., Ethernet) in addition to or instead of a wireless interface.

[0231] The communications subsystem 1224 can receive and transmit data in various formats. In some examples, in addition to other formats, the communications subsystem 1224 can receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. For example, the communications subsystem 1224 can be configured to receive (or transmit) data feeds 1226 in real time from users of social media networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0232] In an example, the communications subsystem 1224 can be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230, which may be continuous or unlimited in nature with no apparent end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0233] The communications subsystem 1224 may also be configured to communicate data from the computer system 1200 to other computer systems or networks. The data may be communicated in a variety of different forms, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1200.

[0234] Computer system 1200 can be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, the description of computer device 1200 shown in FIG. 12 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 12. It will be understood that there are other manners and / or methods for implementing the various examples based on the present disclosure and the teachings provided herein.

[0235] While specific examples have been described, various modifications, variations, alternative configurations, and equivalents are possible. The examples are not limited to operation in one particular data processing environment, but can freely operate in multiple data processing environments. Additionally, while certain examples are described using a particular sequence of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may have additional steps not included in the figures. Various features and aspects of the examples described above may be used individually or jointly.

[0236] Additionally, while certain examples are described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. An example may be performed exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be performed on the same processor or on multiple different processors in any combination.

[0237] Where a device, system, component, or module is described as being configured to perform a certain operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, such as by executing computer instructions or code, or by a processor or core that is programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of technologies, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0238] Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the examples. This description provides illustrative examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the preceding description of the examples provides those skilled in the art with an empowering description for practicing various examples. Various changes can be made in the function and arrangement of elements.

[0239] Accordingly, the specification and drawings should be regarded in an illustrative, and not a restrictive, sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the appended claims. Accordingly, while particular examples have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0240] In the foregoing specification, aspects of the present disclosure have been described with reference to particular examples thereof, but those skilled in the art will recognize that the present disclosure is not limited thereto. Various features and aspects of the above-described disclosure may be used individually or jointly. Moreover, the examples may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the specification and drawings should be considered illustrative and not restrictive.

[0241] In the above description, for purposes of illustration, the methods have been described in a particular order. It should be noted that in alternative examples, the methods may be performed in a different order than that described. It should also be noted that the above-described methods may be performed by hardware components or may be embodied in a series of machine-executable instructions that can be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuitry that has been programmed with the instructions, to perform the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, or any other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0242] Where an element is described as being configured to perform a certain operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0243] Although illustrative examples of the present application have been described in detail herein, it is to be understood that the inventive concepts may be variously embodied and utilized in other manners, and that the appended claims are intended to be construed to include such modifications except insofar as limited by the prior art. < / value> < / value> < / binary-op> < / value> < / unary-op> < / value> < / value> < / value> < / count> < / value> < / question> < / question>

Claims

1. 1. A computer-implemented method for processing natural language, comprising: generating an input string by concatenating the natural language utterance with a database schema representation of the database; generating, by a first encoder, one or more embeddings of the natural language utterance and the database schema representation based on the input string; encoding, by a second encoder, associations between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings; generating, by a grammar-based decoder, an intermediate database query representation based on the encoded relevance and the one or more embeddings; generating a database query in a database query language based on the intermediate database query representation and the interface specification; A computer-implemented method comprising:

2. 10. The method of claim 1, further comprising receiving, by the second encoder, schema link associations linking elements in the database schema representation with words in the natural language utterance, wherein the embeddings are further generated based on the schema link associations.

3. The method of claim 2 , wherein the schema link associations include metadata specifying synonyms of words.

4. 4. The method of claim 1, further comprising receiving, by the grammar-based decoder, a relational algebra grammar representing the intermediate database query representation as a tree, wherein the intermediate database query representation is further based on the relational algebra grammar.

5. the first encoder is a pre-trained language model (PLM); The method of any one of claims 1 to 4, wherein the second encoder is a Relationship Aware Transformer (RAT).

6. The method of any one of claims 1 to 5, further comprising executing the database query against the database to retrieve data in response to the natural language utterance.

7. The method of any one of claims 1 to 5, wherein the database schema representation of the database includes link attributes that refer to entries in the tables without referencing the names of the tables.

8. 1. A system comprising: one or more processors; one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations; Equipped with The operation is generating an input string by concatenating the natural language utterance with a database schema representation of the database; generating, as a first encoder, one or more embeddings of the natural language utterance and the database schema representation based on the input string; as a second encoder, encoding associations between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings; as a grammar-based decoder, generating an intermediate database query representation based on the encoded relevance and the one or more embeddings; generating a database query in a database query language based on the intermediate database query representation and the interface specification; A computer-implemented method comprising:

9. The operation is 10. The system of claim 8, further comprising receiving as the second encoder schema link associations linking elements in the database schema representation with words in the natural language utterance, wherein the embeddings are further generated based on the schema link associations.

10. The system of claim 9 , wherein the schema link associations include metadata specifying synonyms of words.

11. The operation is 11. The system of claim 8, further comprising receiving, as the grammar-based decoder, a relational algebra grammar that represents the intermediate database query representation as a tree, wherein the intermediate database query representation is further based on the relational algebra grammar.

12. the first encoder is a pre-trained language model (PLM); The system of any one of claims 8 to 11, wherein the second encoder is a Relationship Aware Transformer (RAT).

13. The operation is The system of any one of claims 8 to 12, further comprising executing the database query against the database to retrieve data in response to the natural language utterance.

14. The system of any one of claims 8 to 13, wherein the database schema representation of the database includes link attributes that refer to entries in the tables without referencing the names of the tables.

15. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the system to perform operations, the operations including: generating an input string by concatenating the natural language utterance with a database schema representation of the database; generating, as a first encoder, one or more embeddings of the natural language utterance and the database schema representation based on the input string; as a second encoder, encoding associations between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings; as a grammar-based decoder, generating an intermediate database query representation based on the encoded relevance and the one or more embeddings; generating a database query in a database query language based on the intermediate database query representation and the interface specification; 1. One or more non-transitory computer-readable media, including:

16. The operation is 16. The one or more non-transitory computer-readable media of claim 15, further comprising receiving as the second encoder schema link associations linking elements in the database schema representation with words in the natural language utterance, wherein the embeddings are further generated based on the schema link associations.

17. 17. The one or more non-transitory computer-readable media of claim 16, wherein the schema link associations include metadata specifying synonyms of words.

18. The operation is 18. The one or more non-transitory computer-readable media of claim 15, further comprising receiving, as the grammar-based decoder, a relational algebra grammar that represents the intermediate database query representation as a tree, the intermediate database query representation being further based on the relational algebra grammar.

19. the first encoder is a pre-trained language model (PLM); The one or more non-transitory computer-readable media of any one of claims 15 to 18, wherein the second encoder is a Relationship Aware Transformer (RAT).

20. The operation is 20. The one or more non-transitory computer-readable media of any one of claims 15 to 19, further comprising executing the database query against the database to retrieve data in response to the natural language utterance.

21. An apparatus comprising means for performing the operations of the method according to any one of claims 1 to 7.

22. A computer program product comprising computer instructions which, when executed by a processor, perform the operations of the method according to any one of claims 1 to 7.

Images