Techniques for converting natural language utterances into intermediate database query representations

Abstract

Techniques for converting natural language utterances into intermediate database query representations are disclosed herein. An input string is generated by concatenating the natural language utterance with a database schema representation of a database. Based on the input string, the first encoder generates one or more embedding of the natural language utterance and the database schema representation. A second encoder encodes a relationship between an element in the database schema representation and a word in the natural language utterance based on the one or more embedding. The syntax-based decoder generates an intermediate database query representation based on the encoded relationship and the one or more inserts. A database query is generated in a database query language based on the intermediate database query representation and the interface specification.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application of U.S. Application No. 18 / 209,844 filed on June 14, 2023 and U.S. Provisional Application No. 63 / 400,037 filed on August 22, 2022 and claims the benefit and priority thereof by 35 U.S.C. 119(e), the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] The present disclosure relates generally to transforming natural language into a meaning representation language, and more specifically, to machine learning techniques for converting natural language utterances into an intermediate database query representation language, such as Oracle Meaning Representation Language (OMRL). Background Art

[0004] Artificial intelligence has many applications. For example, many users around the world are on instant messaging or chat platforms in order to get instant responses. Organizations often use these instant messaging or chat platforms to have real-time conversations with customers (or end users). However, it can be very costly for organizations to hire service personnel to communicate with customers or end users in real time. Chatbots or robots have begun to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with robots through messaging applications that end users have installed and used. Intelligent robots, which are often powered by artificial intelligence (AI), can communicate context more intelligently in real-time conversations, and therefore can allow more natural conversations between robots and end users to improve the conversation experience. Unlike the end user learning a fixed set of keywords or commands that the robot knows how to respond to, the intelligent robot can understand the end user's intent based on the user's natural language utterances and respond accordingly.

[0005] AI-based solutions (such as chatbots) can have analog (human-machine) interfaces and digital (machine) interfaces for interacting with people and connecting to back-end systems. It is advantageous to be able to extract and analyze the meaning of an utterance (e.g., a request) when a person speaks an utterance (e.g., a request) in natural language, without relying on how the back-end system will handle the utterance. As an example, a request may be for data that needs to be retrieved from a relational database, or the requested data may need to be extracted from a knowledge graph. Meaning Representation Language (MRL) is a universal representation of natural language utterances that a chatbot can translate into any number of target machine-oriented languages. Therefore, chatbots can use MRL to interact and communicate with people and various back-end systems, including systems that communicate using structured query language (SQL), application programming interface (API), representational state transfer (REST), graph query language (GraphQL), feature graph query language (PGQL), etc.

[0006] For example, SQL is a standard database management language for interacting with relational databases. SQL can be used to store, manipulate, retrieve and / or otherwise manage data stored in a relational database management system (RDBMS) and / or for stream processing in a relational data stream management system (RDSMS). SQL includes statements or commands for interacting with relational databases. SQL statements or commands are classified into, in particular, data query language (DQL) statements, data definition language (DDL) statements, data control language (DCL) statements, and data manipulation language (DML) statements. In order to interact with relational databases using SQL, users must know how the database is structured (e.g., knowing the rows and columns in the table and each table), SQL grammar, and how to associate grammar with the database structure. Without this knowledge, users often have difficulty using SQL to interact with these relational databases.

[0007] A natural language interface (e.g., a chatbot) to a database system (NLIDB) (such as an RDBMS) provides a means for users to interact with these relational databases in an intuitive manner without having to know the database management language. For example, using natural language statements and queries (i.e., natural language queries), users can interact with these relational databases in pure language via the NLIDB. Recently, text-to-SQL systems have become popular, and deep learning methods that convert natural language queries into SQL queries have proven to be promising. Using semantic parsing, natural language statements, requests, and questions (i.e., sentences) can be transformed into machine-oriented languages ​​that can be executed by applications (e.g., chatbots, models, programs, machines, etc.). For example, semantic parsing can transform natural language sentences into general programming languages, such as Python, Java, and SQL. Processing for transforming natural language sentences into SQL queries typically includes rule-based, statistical-based, and / or deep learning-based systems. Rule-based systems typically use a series of fixed rules to translate natural language sentences into SQL queries. These rule-based systems are generally domain-specific and are therefore considered to lack flexibility and cannot be well generalized to new use cases (e.g., across different domains). Statistics-based systems (such as slot filling) label tokens (i.e., words or phrases) according to the semantic roles of the tokens in the input natural language sentences, and use the labels to fill the slots in the SQL query. In general, these statistics-based systems have restrictions on the types of sentences that can be parsed (e.g., the sentences must be able to be represented as parse trees). Deep learning-based systems (such as sequence-to-sequence models) involve training deep learning models that directly translate natural language sentences into machine-oriented languages ​​and have been shown to generalize across tasks, domains, and datasets. However, such deep learning systems require a large amount of training data for supervised learning, and obtaining labeled data (e.g., natural language query-SQL statement pairing) is challenging. Therefore, translating natural language sentences into machine-oriented languages ​​based on deep learning cannot avoid the need for a large amount of labeled training data. Summary of the invention

[0008] Machine learning techniques (e.g., methods, systems, non-transitory computer-readable media storing code or instructions executable by one or more processors) are provided for using machine learning models trained to translate natural language into a meaning representation language such as Oracle Meaning Representation Language (OMRL).

[0009] In various embodiments, a computer-implemented method is provided, the method comprising generating an input string by concatenating a natural language utterance with a database schema representation of a database; generating, by a first encoder, one or more embeddings of the natural language utterance and the database schema representation based on the input string; encoding, by a second encoder, relationships between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings; generating, by a grammar-based decoder, an intermediate database query representation based on the encoded relationships and the one or more embeddings; and generating a database query in a database query language based on the intermediate database query representation and an interface specification.

[0010] In some embodiments, the computer-implemented method further comprises providing a schema linking relationship to the second encoder, the schema linking relationship linking an element in the database schema representation with a word in the natural language utterance, wherein the embedding is further generated based on the schema linking relationship. In some embodiments, the schema linking relationship includes metadata specifying synonyms of the word.

[0011] In some embodiments, the method further comprises providing a relational algebra grammar representing the intermediate database query representation as a tree to the grammar-based decoder, wherein the intermediate database query representation is further based on the relational algebra grammar.

[0012] In some embodiments, the first encoder is a pre-trained language model (PLM); and the second encoder is a relation-aware transformer (RAT).

[0013] In some embodiments, the method further comprises executing a database query on a database to retrieve data responsive to the natural language utterance.

[0014] In some embodiments, a database schema representation of a database includes a link attribute that references an entry in a table without referencing the name of the table.

[0015] In various embodiments, a system is provided that includes one or more processors and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform part or all of the operations and / or methods disclosed herein.

[0016] In various embodiments, one or more non-transitory computer-readable media for storing instructions are provided, which, when executed by one or more processors, cause the system to perform part or all of the operations and / or methods disclosed herein.

[0017] In various embodiments, an apparatus is provided that includes components for implementing part or all of the operations and / or methods disclosed herein.

[0018] In various embodiments, a computer program product is provided. The computer program product includes computer instructions. When the instructions are executed by a processor, part or all of the operations and / or methods disclosed herein are implemented.

[0019] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts are only a small part of the numerous implementations and contexts. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a simplified block diagram of a distributed environment incorporating an exemplary embodiment.

[0021] Figure 2 is a simplified block diagram of a computing system that implements a master robot in accordance with certain embodiments.

[0022] Figure 3 is a simplified block diagram of a computing system that implements a skill robot in accordance with certain embodiments.

[0023] Figure 4 is a block diagram illustrating an overview of a C2OMRL architecture and a process of generating queries for a backend interface starting from a natural language utterance, according to various embodiments.

[0024] Figure 5 is a simplified block diagram of a C2OMRL architecture according to various embodiments.

[0025] Figure 6 Illustrated is a model system for training and deploying models according to various embodiments.

[0026] Figure 7 Illustrated is a process flow for converting a natural language utterance into an intermediate database query representation in accordance with various embodiments.

[0027] Figure 8 An example of a RA tree illustrating link capabilities according to various embodiments is illustrated.

[0028] Fig. 9A An example RA tree with Like_start according to various embodiments is illustrated.

[0029] Fig. 9B An example RA tree with Like_end according to various embodiments is illustrated.

[0030] Fig.10 Depicted is a simplified diagram of a distributed system for implementing various embodiments.

[0031] Fig.11is a simplified block diagram of one or more components of a system environment according to various embodiments, by which services provided by one or more components of an embodiment system may be provisioned as cloud services.

[0032] Fig.12 An example computer system is illustrated that may be used to implement various embodiments. DETAILED DESCRIPTION

[0033] In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it is apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not intended to be limiting. The word "exemplary" as used herein means "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or superior to other embodiments or designs.

[0034] introduce

[0035] In recent years, the amount of data powering different industries and their systems has grown exponentially. Most business information is stored in the form of relational databases that store, process, and retrieve data. Databases power information systems across multiple industries, such as consumer technology (e.g., orders, cancellations, refunds), supply chain (e.g., raw materials, inventory, suppliers), healthcare (e.g., medical records), finance (e.g., financial business metrics), customer support, search engines, and more. Modern data-driven companies must track the real-time status of their business in order to quickly understand and diagnose any issues, trends, or anomalies that arise in the data and take immediate corrective actions. This work is typically performed manually by analysts who write complex queries in query languages ​​(e.g., database query languages ​​such as declarative query languages) such as SQL, PGQL, logical database queries, API query languages ​​such as GraphQL, REST, and others). Writing such queries can be used to derive insightful information from data stored in multiple tables. These results are typically processed in the form of charts or graphs to enable users to quickly visualize the results and facilitate data-driven decision making.

[0036] While common database queries (e.g., SQL queries) are often predefined and incorporated into commercial products, any new or subsequent queries still need to be manually encoded and decoded by analysts. Such static interactions between database queries and consumption of corresponding results require time-consuming manual intervention and result in slow feedback cycles. It is much more efficient to let non-technical users (e.g., business leaders, doctors, or other users of data) interact directly with analytical tables via natural language (NL) queries that abstract the underlying query language (e.g., SQL) code. Defining database queries requires a deep understanding of the database schema and query language syntax, and can quickly become overwhelming for beginners and non-technical stakeholders. Efforts to bridge this communication gap have led to the development of a new type of processing called NLIDB. In recent years, this natural search capability has become increasingly popular as companies develop deep learning methods for natural language to logical form (NL2LF), such as natural language to SQL (NL2SQL). Logical form can refer to a meaning representation language and / or a machine-oriented language. NL2SQL seeks to transform natural language questions into SQL, allowing individuals to run unstructured queries against databases. The converted SQL can also enable digital assistants such as chatbots to improve their responses when answers can be found in a different database or table.

[0037] However, the main problem with deep learning methods for NL2SQL is that they require a large amount of training data to build an accurate model. Conventional methods generally ignore this problem and assume that a large, manually curated training dataset is available (e.g., using crowdsourcing to generate NL and SQL data pairs). However, in most cases, collecting and cleaning data is a difficult task that requires a lot of time, effort, and money. Moreover, existing methods for NL2SQL translation attempt to build models that generalize to new and unseen databases, resulting in generally good performance, but not as good as running new queries on the database used for training. That is, the training data used to translate queries for one specific database (such as queries containing words and phrases related to food orders) does not always allow the model to generalize to queries in other domains (such as databases of weather or stock market prices). In addition, given the complexity of the task, models that generate database queries from natural language inputs typically have millions of parameters, making loading and training the model a time-consuming task that uses a lot of computing resources.

[0038] As described above, there are multiple database query languages ​​in use, and requests for data may need to be retrieved from knowledge graphs or different types of relational databases. In addition to NL2SQL for generating SQL queries, other models can be developed to generate queries in other languages. However, when multiple database query languages ​​are taken into account, the above-mentioned large amount of time, effort, and computing resources required to build a model that translates natural language into a given database query language are multiplied. For example, traditionally, building a system that can translate natural language input into both SQL and PGQL requires going through the above-mentioned time-consuming model building and training process to generate a model for generating SQL queries, and then starting from scratch and going through the time-consuming model building and training process again to generate a model for generating PGQL queries. Each additional database query language will multiply the time and resources required to develop a suitable model.

[0039] Thus, different approaches are needed to address these and other challenges. The approach described herein generates a universal representation that can be used to communicate interchangeably with humans and various backend systems (including logic, SQL, API, REST, GraphQL, PGQL, etc.).

[0040] The technology described herein addresses these and other problems using a model configured to generate logical forms in an intermediate database query language based on natural language utterances. The training and development of this model does not need to be repeated to adapt to different back ends. Rather, the logical forms in the intermediate database query language are translated into the appropriate database query language using an interface specification that specifies the translation from the intermediate database query language to the target database query language (e.g., SQL, PGQL, etc.). Unlike the translation model, the interface specification does not require training, so the model can be adapted to a variety of languages, and the computation, time, and manual data preparation are much less than the computation, time, and manual data preparation required to generate separate models for different database query languages.

[0041] The conversion from the natural language utterance to the logical form in the intermediate database query language is performed using a specialized machine learning model. The machine learning model is configured to receive as input a concatenation of the natural language utterance and a database schema representation (e.g., specifying tables, columns, etc. in the database schema). The machine learning model includes two encoders and a grammar-based decoder. The first encoder generates embeddings of the natural language utterance and the database schema representation. The second encoder encodes the relationship between the elements in the database schema representation and the words in the natural language utterance. Based on the encoded relationship and one or more embeddings, the grammar-based decoder generates an intermediate database query representation. The intermediate database query representation can then be converted into a database query in the target database query language using an interface specification specific to the target database query language.

[0042] As used herein, when an action is "based on" something, it means that the action is based, at least in part, on at least a portion of that something. As used herein, the terms "similarly," "substantially," "approximately," and "approximately" are defined as being roughly, but not necessarily exactly as specified (and including exactly as specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "similarly," "substantially," "approximately," or "approximately" may be replaced with "within a specified [percentage]," where percentages include 0.1 percent, 1 percent, 5 percent, and 10 percent.

[0043] Robotics and analytics systems

[0044] A robot (also referred to as a skill, chatbot, chatterbot or conversational robot) is a computer program that can perform a conversation with an end user. The robot can typically respond to a natural language message (e.g., a question or comment) through a messaging application using a natural language message. An enterprise can use one or more robot systems to communicate with end users through a messaging application. A messaging application (which may be referred to as a channel) may be a messaging application preferred by an end user that the end user has installed and is familiar with. Therefore, the end user can chat with the robot system without downloading and installing a new application. The messaging application may include, for example, an over-the-top (OTT) messaging channel (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack or SMS), a virtual private assistant (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), a mobile and web application extension that extends a local or hybrid / responsive mobile application or web application with a chat capability, or a voice-based input (such as a device or application with an interface for interacting using Siri, Cortana, Google Voice or other speech input).

[0045] In some examples, a robot system can be associated with a uniform resource identifier (URI). A URI can use a string of characters to identify a robot system. A URI can be used as a webhook for one or more messaging applications. A URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). A robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. An HTTP post call message can be directed from a messaging application system to a URI. In some embodiments, the message can be different from an HTTP post call message. For example, a robot system can receive a message from a short message service (SMS). Although the discussion herein may refer to communications received as messages by a robot system, it should be understood that the message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0046] End users can interact with the robot system through conversational interactions (sometimes called conversational user interfaces (UIs)), just like human interactions. In some cases, the interaction can include the end user saying "hello" to the robot, and the robot responding with "hi" and asking the end user how he can help. In some cases, the interaction can also be a transactional interaction with, for example, a banking robot, such as transferring money from one account to another; an informational interaction with, for example, an HR robot, such as checking a vacation balance; or an interaction with, for example, a retail robot, such as discussing returning a purchased item or seeking technical support.

[0047] In some embodiments, the robot system can intelligently handle end-user interactions without interacting with the administrator or developer of the robot system. For example, the end user can send one or more messages to the robot system to achieve the desired goal. The message can include certain content, such as text, emoticons, audio, images, videos, or other methods of conveying messages. In some embodiments, the robot system can convert the content into a standardized form (e.g., a representational state transfer (REST) ​​call for enterprise services with appropriate parameters) and generate a natural language response. The robot system can also prompt the end user to add input parameters or request other additional information. In some embodiments, the robot system can also initiate communication with the end user instead of passively responding to the end user's speech. Described herein are various technologies for identifying explicit calls to the robot system and determining the input of the called robot system. In some embodiments, the main robot performs explicit call analysis based on the call name in the detection speech. In response to the detection of the call name, the speech can be refined to be input to the skill robot associated with the call name.

[0048] A conversation with a robot can follow a specific conversation flow including multiple states. The flow can define what will happen next based on the input. In some embodiments, a state machine including user-defined states (e.g., end-user intent) and actions to be taken in a state or from one state to another can be used to implement a robot system. Conversations can take different paths based on end-user input, which can affect the robot's decision on the process. For example, in each state, based on the end-user's input or speech, the robot can determine the end-user's intent to determine the appropriate next action to take. As used herein and in the context of speech, the term "intent" refers to the intent of the user providing the speech. For example, a user may intend to have a conversation with the robot to order pizza, so that the user's intent can be represented by the speech "order pizza". User intent can be for a specific task that the user wants the chatbot to perform on behalf of the user. Therefore, speech can be expressed as questions, commands, requests, etc. that reflect the user's intent. Intent can include goals that the end user wants to achieve.

[0049] In the context of the configuration of a chatbot, the term "intent" is used herein to refer to configuration information that maps a user's utterance to a specific task / action or task / action category that the chatbot can perform. In order to distinguish the intent of an utterance (i.e., user intent) from the intent of a chatbot, the latter is sometimes referred to herein as "robot intent". A robot intent may include a set of one or more utterances associated with an intent. For example, the intent to order pizza may be conveyed by various permutations of utterances expressing the desire to order pizza. These associated utterances may be used to train the chatbot's intent classifier so that the intent classifier can then determine whether the input utterance from the user matches the intent to order pizza. A robot intent may be associated with one or more dialog flows that are used to start a conversation with a user and are in a certain state. For example, the first message of the intent to order pizza may be the question "What kind of pizza do you want?" In addition to the associated utterances, a robot intent may also contain named entities associated with an intent. For example, an intent to order pizza may include variables or parameters (e.g., ingredients 1, ingredients 2, pizza type, pizza size, pizza quantity, etc.) for performing a task to order pizza. The value of an entity is typically obtained by talking to a user.

[0050] Figure 1 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to some embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102 that enables users of the DABP 102 to create and deploy digital assistants or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, Figure 1As shown in , a user 104 representing a particular business can use DABP 102 to create and deploy a digital assistant 106 for users of that particular business. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform can be used by multiple businesses to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0051] For purposes of this disclosure, a "digital assistant" is a tool that helps the user of the digital assistant complete various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital tool implemented using a program, code, or instruction that can be executed by one or more processors), hardware, or a combination of hardware and software. A digital assistant can be implemented or implemented in a variety of physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Thus, for purposes of this disclosure, the terms "digital assistant" and "chatbot system" are interchangeable.

[0052] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks via a natural language-based conversation between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and get responses 112 from the digital assistant 106. The conversation can include one or more of the inputs 110 and the responses 112. Via these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond with an appropriate response to the user.

[0053] User input 110 is generally in the form of natural language and is referred to as utterance. User utterance 110 may be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user utterance 110 may be in the form of audio input or speech, such as when a user speaks or utters something provided as input to the digital assistant 106. The utterance is typically a language spoken by the user 108. For example, the utterance may be English, or some other language. When the utterance is in the form of speech, the speech input is converted into a text form utterance of the particular language, which is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert speech or audio input into text utterances, which are then processed by the digital assistant 106. In some embodiments, the conversion of speech to text may be performed by the digital assistant 106 itself.

[0054] An utterance (which may be a textual utterance or a verbal utterance) may be a fragment, a sentence, a plurality of sentences, one or more words, one or more questions, a combination of the types mentioned above, and the like. The digital assistant 106 is configured to apply natural language understanding (NLU) technology to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that the utterance is a textual utterance provided directly by the user 108 of the digital assistant 106 or is the result of a conversion of an input verbal utterance to a textual form. However, this is not intended to be limiting or restrictive in any way.

[0055] For example, user 108 input can request to order pizza by providing an utterance such as "I want to order pizza". Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. Appropriate action may involve, for example, responding to the user with questions requesting the user to input the type of pizza the user wants to order, the size of the pizza, any ingredients of the pizza, etc. The response 106 provided by the digital assistant may also be in natural language form and is typically the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). For users who order pizza, via a conversation between the user and the digital assistant 106, the digital assistant may guide the user to provide all the information required for pizza ordering, and then order the pizza at the end of the conversation. The digital assistant 106 may end the conversation by outputting information indicating that the pizza has been ordered to the user.

[0056] At a conceptual level, the digital assistant 106 performs various processing in response to utterances received from the user. In some embodiments, this processing involves a sequence or pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to be performed in response to the utterance, causing the action to be performed where appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing can include parsing the received input utterance to understand the structure and meaning of the utterance, refining and reconstructing the utterance to construct a more understandable form (e.g., logical form) or structure for the utterance. Generating a response can include using NLG techniques.

[0057] The NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related processing, such as sentence parsing (e.g., word segmentation, lemmatization, identifying part-of-speech tags of sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing single clauses, resolving references, performing chunking, etc.). In some embodiments, the NLU processing or part thereof is performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 may use other resources to perform part of the NLU processing. For example, the syntax and structure of the sentence can be identified by processing the input utterance sentence using a parser, a part-of-speech tagger, and / or a named entity recognizer. In one embodiment, for English, sentence structure and syntax are analyzed using a parser, a part-of-speech tagger, and a named entity recognizer such as provided by the Stanford Natural Language Processing (NLP) group. These are provided as part of the Stanford CoreNLP toolkit.

[0058] Although the various examples provided in this disclosure show utterances in the English language, this is meant only as an example. In some embodiments, the digital assistant 106 is also capable of handling utterances in languages ​​other than English. The digital assistant 106 can provide subsystems (e.g., components that implement NLU functionality) that are configured to perform processing on different languages. These subsystems can be implemented as pluggable units that can be called using service calls from the NLU core server. This makes NLU processing flexible and scalable for each language, including allowing different processing orders. Language packs can be provided for various languages, where the language pack can register a list of subsystems that can provide services from the NLU core server.

[0059] Digital assistants (such as Figure 1The digital assistant 106 depicted in FIG. 1 may be available or accessible to its user 108 through a variety of different channels, such as, but not limited to, via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. A single digital assistant may have several channels configured for it so that it can run and be accessed on different services simultaneously.

[0060] A digital assistant or chatbot system generally includes or is associated with one or more skills. In some embodiments, these skills are separate chatbots (referred to as skill bots) that are configured to interact with a user and complete specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing gadgets, etc. For example, for Figure 1 In the embodiment depicted in , the digital assistant or chatbot system 106 includes skills 116 - 1, 116 - 2, etc. For the purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skillbot” and “skillbots”, respectively.

[0061] Each skill associated with the digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, where the conversation can include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses can be in the form of text or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for selection by the user.

[0062] Skills or skill robots can be associated or added to digital assistants in various ways. In some cases, skill robots can be developed by an enterprise and then added to a digital assistant using DABP 102. In other cases, skill robots can be developed and created using DABP 102 and then added to digital assistants created using DABP 102. In other cases, DABP 102 provides an online digital store (referred to as a "skill store") that provides a variety of skills for a range of tasks. Skills provided by the skill store can also expose various cloud services. In order to add skills to a digital assistant being generated using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skills, and indicate that the selected skills will be added to the digital assistant created using DABP 102. Skills from the skill store can be added to the digital assistant as is or in a modified form (e.g., a user of DABP 102 can select and clone a specific skill robot provided by the skill store, customize or modify the selected skill robot, and then add the modified skill robot to the digital assistant created using DABP 102).

[0063] A digital assistant or chatbot system may be implemented using a variety of different architectures. For example, in some embodiments, a digital assistant created and deployed using DABP 102 may be implemented using a master robot / child (or slave) robot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more child robots that are skill robots. For example, in Figure 1 In the illustrated embodiment, the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are child robots of the master robot 114. In some embodiments, the digital assistant 106 itself is considered to act as a master robot.

[0064] A digital assistant implemented according to the master-sub robot architecture enables the user of the digital assistant to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user interacts with the digital assistant, the user input is received by the master robot. The master robot then performs processing to determine the meaning of the utterance entered by the user. The master robot then determines whether the task requested by the user in the utterance can be handled by the master robot itself, otherwise the master robot selects an appropriate skill robot to handle the user request and routes the conversation to the selected skill robot. This enables the user to talk to the digital assistant through a shared single interface, and still provides the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot of the digital assistant can be docked with a skill robot with specific functionality (such as a CRM robot for performing functions related to customer relationship management (CRM), an ERP robot for performing functions related to enterprise resource planning (ERP), an HCM robot for performing functions related to human capital management (HCM), etc.). In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a shared master robot interface, and multiple skill robots are provided behind the scenes to handle user requests.

[0065] In some embodiments, in the master robot / sub-robot infrastructure, the master robot is configured to know the available list of skill robots. The master robot has access to metadata that identifies each available skill robot and, for each skill robot, the capabilities of the skill robot include tasks that the skill robot can perform. Upon receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot that can best serve or handle the user request from a plurality of available skill robots. The master robot then routes the utterance (or a portion of the utterance) to the specific skill robot for further handling. Thus, control flows from the master robot to the skill robot. The master robot may support multiple input and output channels. In some embodiments, routing may be performed with the aid of processing performed by one or more available skill robots. For example, as discussed below, a skill robot may be trained to infer the intent of an utterance and determine whether the inferred intent matches the intent that the skill robot is configured with. Thus, routing performed by the master robot may involve the skill robot conveying to the master robot an indication of whether the skill robot has been configured with an intent suitable for handling the utterance.

[0066] Although Figure 1The embodiment in FIG. 1 shows a digital assistant 106 including a master robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide digital assistant functionality. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an implementation using a combination of software and hardware.

[0067] DABP 102 provides an infrastructure and various services and features that enable users of DABP 102 to create a digital assistant including one or more skill robots associated with the digital assistant. In some cases, a skill robot can be created by cloning an existing skill robot, such as cloning a skill robot provided by a skill store. As indicated above, DABP 102 provides a skill store or skill catalog that supplies multiple skill robots for performing various tasks. Users of DABP 102 can clone skill robots from the skill store. The cloned skill robots can be modified or customized as needed. In some other cases, users of DABP 102 create skill robots from scratch using the tools and services provided by DABP 102. As indicated above, the skill store or skill catalog provided by DABP 102 can supply multiple skill robots for performing various tasks.

[0068] In certain embodiments, at a high level, creating or customizing a skill bot involves the following steps:

[0069] (1) Configuring the settings of a new skill robot

[0070] (2) Configure one or more intents for the skill robot

[0071] (3) Configure one or more entities for one or more intents

[0072] (4) Training Skill Robots

[0073] (5) Create a dialogue flow for the skill bot

[0074] (6) Add custom components to skill robots as needed

[0075] (7) Testing and deploying skill robots

[0076] Each of the above steps is briefly described below.

[0077] (1) Configuring settings for a new skill robot - Various settings can be configured for a skill robot. For example, a skill robot designer can specify one or more invocation names for the skill robot being created. These invocation names can then be used by a user of the digital assistant to explicitly invoke the skill robot. For example, a user can enter an invocation name in the user's utterance to explicitly invoke the corresponding skill robot.

[0078] (2) Configure one or more intents and associated example utterances for the skill robot - The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classes of utterances that the skill robot is trained to infer the input. After receiving the utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then responds to the utterance by taking appropriate actions based on the intent inferred for the utterance. In some cases, the intent of the skill robot represents the tasks that the skill robot can perform for the user of the digital assistant. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot may include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0079] For each intent defined for a skill bot, the skill bot designer may also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent utterances that a user might enter into the skill bot for that intent. For example, for the CheckBalance intent, example utterances may include "What is my savings account balance?", "How much money is in my checking account?", "How much money is in my account?", etc. Thus, various permutations of typical user utterances may be specified as example utterances for an intent.

[0080] Intents and their associated example utterances are used as training data to train a skill robot. A variety of different training techniques can be used. As a result of this training, a prediction model is generated that is configured to take an utterance as input and output an intent inferred by the prediction model for the utterance. In some cases, the input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer the intent of the input utterance. The skill robot can then take one or more actions based on the inferred intent.

[0081] (3) Configuring entities for one or more intents of a skill bot - In some cases, additional context may be required to enable a skill bot to respond correctly to user utterances. For example, there may be cases where user-entered utterances resolve to the same intent in a skill bot. For example, in the example above, the utterances "What is my savings account balance?" and "How much money is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests asking different things. To clarify such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "savings" enables the skill bot to parse the user request and respond appropriately. In the example above, although the utterances are resolved to the same intent, the values ​​associated with the AccountType entity are different for the two utterances. This enables the skill bot to perform potentially different actions for the two utterances, even though they resolve to the same intent. One or more entities can be specified for certain intents configured for a skill bot. Thus, entities are used to add context to the intent itself. Entities help to more completely describe the intent and enable the skill bot to complete the user request.

[0082] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by a skill robot designer. Built-in entities are general entities that can be used for a variety of robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by a skill robot designer that enables various banking transactions to be performed by checking user-entered keywords such as checking, savings, and credit card, etc.

[0083] (4) Training a Skill Robot - A skill robot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select intents associated with the received user input. As indicated above, the skill robot must be trained for this. In some embodiments, the skill robot is trained based on the intents configured for the skill robot and example utterances associated with those intents (collectively referred to as training data) so that the skill robot can parse the user input utterance into one of the intents configured for the robot. In some embodiments, the skill robot uses a predictive model that is trained using the training data and allows the skill robot to discern what the user said (or in some cases, attempted to say). DABP 102 provides a variety of different training techniques that can be used by skill robot designers to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion (e.g., 80%) of the training data is used to train the skill robot model, while another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as a trained skill robot) can be used to handle and respond to user utterances. In some cases, the user's utterance may be a question that requires only a single answer and does not require further conversation. To handle such situations, a Q&A (question and answer) intent can be defined for the skill robot. This enables the skill robot to output a reply to the user's request without updating the dialog definition. The Q&A intent is created in a similar manner to the regular intent. The dialog flow for the Q&A intent may be different from the dialog flow for the regular intent.

[0084] (5) Create a dialogue flow for the skill robot - The dialogue flow specified for the skill robot describes how the skill robot reacts when different intents for the skill robot are parsed in response to received user input. The dialogue flow defines the operations or actions that the skill robot will take (for example, how the skill robot responds to user utterances, how the skill robot prompts the user for input, and how the skill robot returns data). The dialogue flow is like a flowchart followed by the skill robot. The skill robot designer uses a language such as markdown to specify the dialogue flow. In some embodiments, a version of YAML called OBotML can be used to specify the dialogue flow of the skill robot. The dialogue flow definition for the skill robot serves as a model for the conversation itself, that is, a model that allows the skill robot designer to choreograph the interaction between the skill robot and the user served by the skill robot.

[0085] In some embodiments, a dialog flow definition for a skill bot contains three parts:

[0086] (a) Context

[0087] (b) Default transition section

[0088] (c) Status section

[0089] Context section - Skill bot designers can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, user variables that enable the skill bot to recognize and retain user preferences, etc.

[0090] Default Transitions Section - You can define transitions for a skill bot in the Dialog Flow States section or in the Default Transitions section. Transitions defined in the Default Transitions section act as fallbacks and are triggered when no applicable transitions are defined within a state or when the conditions required to trigger a state transition cannot be met. The Default Transitions section can be used to define routes that allow a skill bot to gracefully handle unexpected user actions.

[0091] States section - A dialog flow and its associated actions are defined as a series of transition states that govern the logic within the dialog flow. Each state node within a dialog flow definition names a component that provides the functionality required at that point in the dialog. States are therefore built around components. States contain component-specific characteristics and define transitions to other states that are triggered after the component executes.

[0092] You can use the state section to handle special case scenarios. For example, sometimes you may want to give the user the option to temporarily leave the first skill they are interacting with in order to do something in a second skill within the digital assistant. For example, if the user is talking to the shopping skill (e.g., the user has made some purchase choices), the user may want to jump to the banking skill (e.g., the user may want to make sure he / she has enough money for the purchase), and then return to the shopping skill to complete the user's order. To address this, the action in the first skill can be configured to initiate an interaction with a different second skill in the same digital assistant and then return to the original flow.

[0093] (6) Adding custom components to a skill bot - As described above, a state specified in a skill bot's dialog flow names a component that provides the functionality required for that state. Components enable a skill bot to perform a function. In some embodiments, DABP 102 provides a set of preconfigured components for performing a wide range of functions. A skill bot designer can select one or more of these preconfigured components and associate them with a state in a skill bot's dialog flow. A skill bot designer can also use the tools provided by DABP 102 to create custom or new components and associate the custom components with one or more states in a skill bot's dialog flow.

[0094] (7) Testing and deploying skill bots - DABP 102 provides several features that enable skill bot designers to test the skill bots they are developing. The skill bots can then be deployed and included in digital assistants.

[0095] Although the above description describes how to create a skill robot, similar techniques can also be used to create a digital assistant (or master robot). At the master robot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master robot) can handle without invoking the skill robot associated with the digital assistant. Examples of system intents defined for the master robot include: (1) Exit: Applicable when the user signals a desire to exit the current conversation or context in the digital assistant; (2) Help: Applicable when the user requests help or guidance; and (3) Unresolved intent: Applicable to user input that does not match well with the exit and help intents. The digital assistant also stores information about one or more skill robots associated with the digital assistant. This information enables the master robot to select a specific skill robot to handle the utterance.

[0096] At the master robot or digital assistant level, when a user enters a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant determines this using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the utterance entered by the user is to be routed to a specific skill for handling, handled by the digital assistant or master robot itself based on built-in system intents, or handled as a different state in the current conversation flow.

[0097] In some embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill robot using its invocation name. If there is an invocation name in the user input, it is considered an explicit invocation of the skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the skill robot that is explicitly invoked for further processing. If there is no specific or explicit invocation, then in some embodiments, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intent and skill robot associated with the digital assistant. The score calculated for the skill robot or system intent represents the likelihood that the user input represents the task that the skill robot is configured to perform or represents the system intent. Any system intent or skill robot with an associated calculated confidence score that exceeds a threshold (e.g., a confidence threshold routing parameter) will be selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill robot from the identified candidates to further process the user input utterance. In some embodiments, after one or more skill robots are identified as candidates, the intents associated with these candidate skills are evaluated (according to the intent model for each skill) and the confidence score for each intent is determined. In general, any intent with a confidence score exceeding a threshold (e.g., 70%) is considered a candidate intent. If a specific skill robot is selected, the user utterance is routed to that skill robot for further processing. If a system intent is selected, one or more actions are performed by the main robot itself according to the selected system intent.

[0098] Figure 2 2 is a simplified block diagram of a master robot (MB) system 200 according to some embodiments. The MB system 200 can be implemented using only software, only hardware, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multi-intention subsystem (MIS), and a

[0099] 220 , explicit invocation subsystem (EIS) 230 , skill robot invoker 240 , and data repository 250 . Figure 2 The MB system 200 depicted in FIG. 1 is merely an example of an arrangement of components in a master robot. A person skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have more than Figure 2 More or fewer systems or components than shown in the drawings may be combined with two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0100] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided in audio form, the pre-processing subsystem 210 can convert the audio into text using a speech-to-text converter (not shown), which inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text.

[0101] Language detector 212 detects the language of utterance 202 based on the text of utterance 202. The way utterance 202 is processed depends on the language, because each language has its own grammar and semantics. When analyzing the grammar and structure of an utterance, the differences between languages ​​are taken into account.

[0102] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for each language unit (e.g., words) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 can also tokenize the language units of the utterance 202 (e.g., convert each word into a separate token) and lemmatize the words. Lemmatization is the main form of a group of words as represented in a dictionary (e.g., "run" is the lemmatization of run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking of compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". The language parser 214 can also identify relationships between words in the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a specific noun) is a direct object, which part of the utterance is a preposition, and so on. The processing performed by the language parser 214 results in the extracted information 205 , which is provided as input to the MIS 220 along with the utterance 202 itself.

[0103] As indicated above, the utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, the utterance 202 may be treated as a single unit even though it includes multiple sentences. However, in some embodiments, preprocessing may be performed, for example by the preprocessing subsystem 210, to identify a single sentence among multiple sentences for multiple intent analysis and explicit call analysis. In general, the results produced by the MIS 220 and the EIS 230 are substantially the same regardless of whether the utterance 202 is processed at the level of individual sentences or as a single unit containing multiple sentences.

[0104] MIS 220 determines whether utterance 202 indicates multiple intents. Although MIS 220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS 220 does not involve determining whether the intent of utterance 202 matches any intent that has been configured for the robot. Instead, the processing of determining whether the intent of utterance 202 matches the robot's intent can be performed by intent classifier 242 of MB system 200 or an intent classifier of a skill robot (e.g., Figure 3 202). The processing performed by MIS 220 assumes that there is a robot (e.g., a particular skill robot or the master robot itself) that can handle utterance 202. Therefore, the processing performed by MIS 220 does not require knowledge of which robots are in the chatbot system (e.g., the identities of the skill robots registered with the master robot) or knowledge of which intents have been configured for a particular robot.

[0105] To determine that the utterance 202 contains multiple intents, the MIS 220 applies one or more rules from a set of rules 252 in the data repository 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and may include a sentence pattern indicating the presence of multiple intents. For example, the sentence pattern may contain a coordinating conjunction that connects two parts of the sentence (e.g., a conjunction), each of which corresponds to a separate intent. If the utterance 202 matches the sentence pattern, it can be inferred that the utterance 202 represents multiple intents. It should be noted that utterances with multiple intents do not necessarily have different intents (e.g., intents for different robots or intents for different intents within the same robot). Alternatively, the utterance may have separate instances of the same intent (e.g., "order pizza using payment account X, then order pizza using payment account Y").

[0106] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines which parts of utterance 202 are associated with each intent. For each intent represented in an utterance containing multiple intents, MIS 220 constructs a new utterance to replace the original utterance for separate processing, for example, utterance "B" 206 and utterance "C" 208, as shown in FIG. Figure 2. Thus, the original utterance 202 may be divided into two or more separate utterances, one utterance at a time. MIS 220 uses the extracted information 205 and / or by analyzing the utterance 202 itself to determine which of the two or more utterances should be handled first. For example, MIS 220 may determine that utterance 202 contains a marker word indicating that a particular intent should be handled first. The newly formed utterance corresponding to the particular intent (e.g., one of utterance 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance ends (or is temporarily suspended), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) may then be sent to EIS 230 for processing.

[0107] EIS230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) contains a call name for a skill robot. In some embodiments, each skill robot in the chatbot system is assigned a unique call name that distinguishes the skill robot from other skill robots in the chatbot system. A list of call names can be maintained in the data repository 250 as part of the skill robot information 254. When a utterance contains a word that matches the call name, the utterance is considered an explicit call. If there is no explicit call to the robot, the utterance received by EIS230 is considered a non-explicit call utterance 234 and is input to the intent classifier of the main robot (e.g., intent classifier 242) to determine which robot to use to handle the utterance. In some cases, the intent classifier 242 will determine that the main robot should handle the utterance of the non-explicit call. In other cases, the intent classifier 242 will determine to route the utterance to the skill robot for handling.

[0108] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that the master robot must perform. For example, when there is an explicit call, the master robot may not have to perform any intent classification analysis (e.g., using intent classifier 242), or may have to perform a simplified intent classification analysis to select a skill robot. Thus, the explicit call analysis can enable selection of a particular skill robot without resorting to intent classification analysis.

[0109] Furthermore, there may be situations where there is overlap in functionality between multiple skill robots. This may happen, for example, if the intents handled by two skill robots overlap or are very close to each other. In this case, it may be difficult for the master robot to identify which of the multiple skill robots to select based solely on intent classification analysis. In such scenarios, explicit calls can disambiguate the specific skill robot to use.

[0110] In addition to determining that the utterance is an explicit call, the EIS 230 is also responsible for determining whether any part of the utterance should be used as input to the skill robot of the explicit call. In particular, the EIS 230 can determine whether a part of the utterance is irrelevant to the call. The EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. The EIS 230 can send the part of the utterance that is irrelevant to the call to the called skill robot instead of sending the entire utterance received by the EIS 230. In some cases, the input of the called skill robot is formed only by removing any part of the utterance that is relevant to the call. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using PizzaBot" is related to the call of the pizza robot, but is irrelevant to any processing to be performed by the pizza robot. In some cases, the EIS 230 can reformat the part to be sent to the called robot, for example, to form a complete sentence. Therefore, the EIS 230 not only determines that there is an explicit call, but also determines what to send to the skill robot when there is an explicit call. In some cases, there may not be any text to be input to the called robot. For example, if the utterance is "PizzaBot", then EIS 230 can determine that a pizza robot is being called, but there is no text to be processed by the pizza robot. In this scenario, EIS 230 can indicate to the skill robot invoker 240 that there is no content to be sent.

[0111] The skill robot invoker 240 invokes a skill robot in various ways. For example, the skill robot invoker 240 may invoke a robot in response to receiving an indication 235 indicating that a particular skill robot has been selected as a result of an explicit invocation. The indication 235 may be sent by the EIS 230 along with the input for the explicitly invoked skill robot. In this scenario, the skill robot invoker 240 transfers control of the conversation to the explicitly invoked skill robot. The explicitly invoked skill robot will determine the appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0112] Another way that the skill robot invoker 240 can invoke a skill robot is implicitly by using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different categories, one category for each skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. Parameters generated as a result of this training (e.g., a set of values ​​for parameters of a machine learning model) can be stored as part of the skill robot information 254.

[0113] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training of the machine learning model may involve inputting at least a subset of utterances from example utterances associated with various skill robots to generate, as output of the machine learning model, inferences about which robot is the correct robot to handle any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance may be provided as ground truth information. The behavior of the machine learning model may then be adapted (e.g., by backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0114] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot, which indicates the likelihood that the skill robot can handle an utterance (e.g., a non-explicit call utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each system-level intent (e.g., help, exit) that has been configured. If a particular confidence score satisfies one or more conditions, the skill robot caller 240 will call the robot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is the identification of the system intent or the identification of a particular skill robot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must also exceed the next highest confidence score by a certain winning margin. When the confidence scores of multiple skill robots each exceed the threshold confidence score value, applying such a condition will enable routing to a particular skill robot.

[0115] After identifying the robot based on the evaluation of the confidence score, the skill robot invoker 240 hands over processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is a skill robot. In addition, the skill robot invoker 240 will determine what to provide as input 247 to the identified robot. As indicated above, in the case of an explicit invocation, the input 247 can be based on a portion of the utterance that is not relevant to the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invocation, the input 247 can be the entire utterance.

[0116] The data repository 250 includes one or more computing devices that store data used by various subsystems of the master robot system 200. As explained above, the data repository 250 includes rules 252 and skill robot information 254. The rules 252 include, for example, rules for determining when an utterance represents multiple intents by the MIS 220 and how to split utterances that represent multiple intents. The rules 252 also include rules for determining which parts of an utterance that explicitly calls a skill robot are sent to the skill robot by the EIS 230. The skill robot information 254 includes the calling names of the skill robots in the chatbot system, such as a list of the calling names of all skill robots registered with a particular master robot. The skill robot information 254 may also include information used by the intent classifier 242 to determine the confidence score of each skill robot in the chatbot system, such as parameters of a machine learning model.

[0117] Figure 3 is a simplified block diagram of a skill robot system 300 according to some embodiments. The skill robot system 300 is a computing system that can be implemented using only software, only hardware, or a combination of hardware and software. In some embodiments, such as Figure 1 In the embodiment depicted in FIG, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0118] The skill robot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to Figure 2 302 and extracted information 304. The extracted information 304 is similar to the MIS 310. Figure 1The extracted information 205 in the skill robot system 300 may be generated using the language parser 214 or a language parser local to the skill robot system 300.

[0119] The intent classifier 320 can be combined with the above Figure 2 In some embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a specific skill robot using at least a subset of example utterances associated with the specific skill robot as training utterances. The ground truth for each training utterance will be the specific robot intent associated with the training utterance.

[0120] The utterance 302 may be received directly from the user or provided by the master robot. When the utterance 302 is provided by the master robot, for example Figure 2 In the embodiment depicted in , when supplied as a result of processing by MIS 220 and EIS 230, MIS 310 may be bypassed to avoid duplicating processing already performed by MIS 220. However, if utterance 302 is received directly from a user, such as during a conversation that occurs after being routed to a skill robot, then MIS 310 may process utterance 302 to determine whether utterance 302 represents multiple intents. If so, then MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, then MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0121] The intent classifier 320 is configured to match a received utterance (e.g., utterance 306 or 308) with an intent associated with the skill robot system 300. As explained above, a skill robot may be configured with one or more intents, each of which includes at least one example utterance associated with the intent and used to train the classifier. Figure 2In an embodiment, the intent classifier 242 of the master robot system 200 is trained to determine confidence scores for individual skill robots and confidence scores for system intents. Similarly, the intent classifier 320 can be trained to determine confidence scores for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and is therefore more fine-grained. The intent classifier 320 can access intent information 354. The intent information 354 includes a list of utterances for each intent associated with the skill robot system 300, which represent and explain the meaning of the intent and are typically associated with a task that can be performed by the intent. The intent information 354 can also include parameters generated by training the list of utterances.

[0122] The conversation manager 330 receives, as an output of the intent classifier 320, an indication 322 of a particular intent that is identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some cases, the intent classifier 320 is unable to determine any match. For example, if the utterance is for a system intent or an intent for a different skill robot, then the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill robot system 300 may submit the utterance to the master robot for disposal, such as routing to a different skill robot. However, if the intent classifier 320 successfully identifies the intent within the skill robot, then the conversation manager 330 will initiate a conversation with the user.

[0123] The conversation initiated by the conversation manager 330 is a specific conversation specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 can be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine can include a default starting state (e.g., when the intent is invoked without any additional input) and one or more additional states, each of which is associated with an action to be performed by the skill robot (e.g., perform a purchase transaction) and / or a dialog to be presented to the user (e.g., a question, a response). Thus, the conversation manager 330 can determine an action / dialog 335 when receiving an indication 322 of an identified intent, and can determine additional actions or dialogs in response to subsequent utterances received during the conversation.

[0124] The data repository 350 includes one or more computing devices that store data used by the various subsystems of the skill robot system 300. Figure 3 As depicted in , data repository 350 includes rules 352 and intent information 354. In some embodiments, data repository 350 can be integrated into the data repository of a main robot or digital assistant, such as Figure 2In the data repository 250.

[0125] Convert natural language utterances into an intermediate database query language

[0126] The C2OMRL system is powered by a deep learning model that is configured to convert natural language (NL) utterances (or conversations within the Oracle Digital Assistant platform) into a logical form in an intermediate database query language, such as Oracle Meaning Representation Language (OMRL). The logical form can be used to generate queries in a specific database query language, which can then be executed to query an existing database. This deep learning model (called the "C2OMRL semantic parser" or "C2OMRL model") is trained with thousands of example pairs (natural language to logical form).

[0127] Figure 4 4 is a block diagram 400 illustrating an overview of a C2OMRL architecture, with an overview of a process for generating a query to a backend interface 406, the query starting with a natural language (NL) utterance 408 (e.g., received via a human-machine interface 402). For example, the human-machine interface 402 may be a chatbot system as described above that receives spoken speech and translates it into text utterances, or a system in which a user types a request in a natural language, or other suitable interface. The NL utterance 408 may be in the form of a portion of a conversation (e.g., "Hi, can you tell me how many orders we need to ship tomorrow?" or "Search for all employees whose names begin with 'S' and live in California.").

[0128] The NL utterance 408 is provided to a NL2LF model 410, which converts the NL utterance 408 into an intermediate representation 412. The NL2LF model 410 is a machine learning model trained to generate the intermediate representation 412 from the NL utterance 408. The NL2LF model 410 may include multiple layers.

[0129] In some cases, such as Figure 4 , the NL2LF model 410 is a Conversational to Oracle Meaning Representation Language (C2OMRL) model for converting conversational utterances into an intermediate Oracle Meaning Representation Language (OMRL) representation 412. The NL2LF model 410 may be described interchangeably with C2OMRL, but it should be understood that the techniques described herein may be applied to models configured to generate other intermediate representation 412 formats.

[0130] The intermediate representation 412 is a logical representation of an utterance that is configured to be translatable into a specific database query language. In some examples, the intermediate representation 412 is Oracle Meaning Representation Language (OMRL), which is an intermediate database query language with a specialized schema and interface specification as described herein. The intermediate representation 412 may be described herein interchangeably with OMRL, but it should be understood that the techniques described herein may be applied to other intermediate representation 412 formats.

[0131] The intermediate representation 412 is then translated into one or more desired query formats, such as SQL 416 processed using OMRL2SQL 414 translation, or PGQL 420 processed using OMRL2PGQL 418 translation. The translated query (e.g., SQL 416 or PGQL 420) represents the concepts present in the intermediate representation 412 in a manner that conforms to the requirements of the applicable query language. As described above, the process of generating a model for generating a backend interface 406 for query-oriented (e.g., SQL 416 or PGQL 420) is time consuming and involves processing a large amount of data. In contrast, OMRL2SQL 414, OMRL2PGQL 418, and similar frameworks for translating queries from the intermediate representation 412 into languages ​​such as SQL 416, PGQL 420 are relatively fast to build and execute. Since such translations are initiated from an intermediate representation configured to translate into a database query language, the models used to perform the translations 414, 418 do not need to be trained on a large corpus of natural language training samples. Thus, the architecture and techniques described herein can significantly reduce the training time for building a platform for translating natural language utterances into multiple query languages.

[0132] Overview of systems for translating natural language into meaning-representing language

[0133] Figure 5 A C2OMRL system 500 is shown that is powered by a machine learning model to enable natural language (NL) utterances (e.g., such as about Figure 1-Figure 3The C2OMRL model 510 is a machine learning model that converts natural language utterances (or utterances within the digital assistant platform) into logical form (LF) statements (such as OMRL queries or commands), which can then be executed to query existing systems (such as relational databases). This machine learning model (referred to herein as the "C2OMRL semantic parser" or "C2OMRL model") is trained on hundreds to thousands of annotated example pairs (natural language and logical form pairs) to translate natural language utterances into logical form statements. As shown, example 505 (a cascade of natural language utterances and database schemas, such as a sequence of table names and column names) is input into the C2OMRL model 510. Example 505 is first processed by an encoder component 515, which captures representations of natural language utterances and database schemas based on context. The decoder 520 then receives the encoded input and predicts a logical form 525 (e.g., OMRL, which is a SQL-like query) based on the captured representations of the natural language utterances and database schemas.

[0134] In the C2OMRL model 510, the encoder component 515 includes two encoders (1) a first encoder, which is a pre-trained language model (PLM) 530; and (2) a second encoder, which is a relation-aware transformer (RAT) 535. PLM 530 is used to embed natural language utterances and database patterns because it captures representations of natural language utterances and database patterns based on context. In some cases, a transformer-based PLM (called Decoding Enhanced BERT with Disentangled Attention (DeBERTa)) is used as PLM 530. (See He et al., DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (2021)). Transformer-based PLMs use self-supervised learning to learn general language representations from large amounts of text data and transfer this knowledge to downstream tasks. These models provide good background knowledge for downstream tasks, which avoids training downstream models from scratch. RAT535 encodes the relationships between so-called entities (words for tables and columns) in a database schema and words in a natural language utterance (these relationships are called “schema link” relationships).

[0135] The decoder 520 is based on a bottom-up generation process (i.e., the bottom-up generation process generates a tree from the bottom (leaf nodes) to the top (root node)), where the final generation output is an OMRL tree (i.e., a tree-based structure representing the complete OMRL logic form), which can be reconstructed into an OMRL logic form 525 output. The bottom-up generation process is implemented using a beam search, which is a heuristic search algorithm that explores graphs by expanding the most promising nodes in a finite set. The beam search works in steps (e.g., ~10 steps), also known as "beam levels". In each step (e.g., "step i"), the beam search algorithm generates F possible subtrees for the input sequence that can be obtained by expanding the current subtree (from step "i-1"), and then selects the first K subtrees (called the beam width) to retain using the conditional probability associated with each subtree. The conditional probability is referred to as the "raw beam score" in this article, so the first K intermediate results (to be considered in the next generation step) are the K results with the highest raw beam scores. Additional information about the bottom-up generative process is available at "Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressive Bottom-up Semantic Parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 311-324, online. Association for Computational Linguistics", the entire contents of which are incorporated herein by reference for all purposes. The final decoder 520 output is the subtree with the highest unprocessed bundle score at the last step N.

[0136] The predicted OMRL logical form statements 525 (i.e., the OMRL tree with the highest raw bundle score at the last step N) can then be input into a language converter 540 (such as OMRL2SQL) to translate the meaning representation language into a system language query or command, such as SQL, API, REST, GraphQL, PGQL, etc. The system language query or command can then be used to query or perform operations on a system 545 (e.g., a relational database) and obtain output 550 as a result of the query or command.

[0137] Figure 6 A diagram is shown configured for training and deploying a machine learning model (e.g., a NL2LF machine learning model and Figure 4-Figure 5 The model system 600 in this example includes various stages: a training stage 605 for training a machine learning model; an NL2LF translation stage 610 for translating natural language text (e.g., an utterance) into a logical form (such as OMRL); and a query execution stage 620 for executing the logical form on a system (such as a database) to obtain a result (e.g., an answer to a query within the utterance). The training stage 605 builds and trains one or more machine learning models 625a-625n ("n" represents any natural number) for use by other stages (which may be referred to individually or collectively as models 625 herein). For example, the models 625 may include a first model for translating a natural language utterance into a logical form of an intermediate database query language and a second model for translating the logical form into a specific database query language. According to the present disclosure, other types of predictive models (e.g., intent classifiers) may be implemented in other examples.

[0138] In some embodiments, model 625 is configured as a semantic parser or NL2LF model, such as RAT-SQL (as an encoder) combined with a semi-autoregressive bottom-up semantic parsing model (SMBOP) (as a decoder). For additional information about the RAT-SQL model, see "RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers" by Wang et al., published in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, the entire contents of which are hereby incorporated by reference for all purposes. For more additional information about the SMBOP model, see "SmBoP: Semi-autoregressive Bottom-up Semantic Parsing" by Ohad Rubin and Jonathan Berant in 2021, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311-324, online. Association for Computational Linguistics, the entire contents of which are hereby incorporated by reference for all purposes.

[0139] To train the various models 625, the training phase 605 consists of two main subsystems or services: a dataset preparer 630 and a model trainer 640. The dataset preparer 630 facilitates the process of loading data assets 645, splitting the data assets 645 into training sets and validation sets (645-an) so that the system can train and test the models 625, and performing basic natural language preprocessing (e.g., standardization, normalization, tokenizing, augmentation, etc.). The data assets 645 include natural language utterances (e.g., natural language questions / requests) as input and OMRL logical forms as output, obtained from one or more sources (such as from human annotators and / or databases (not shown)). In some cases, the utterances are provided by a client or customer. In other cases, the utterances are automatically generated and / or retrieved from a repository of utterances (e.g., identifying utterances from a repository that are specific to the task to be learned by the model). The data assets 645 may include input text or audio (or input features of text or audio frames) and labels 650 corresponding to the input text or audio (or input features), as a matrix or table of values. For example, for each utterance, the corresponding label may include an indication of the corresponding logical form (e.g., a statement / query, such as an OMRL query) and / or an indication of whether the logical form is correct or incorrect for the utterance, provided as ground truth information for the utterance. The behavior of the model may then be adjusted (e.g., via backpropagation) to minimize the difference between the generated inferences and the ground truth information. For example, data asset 845 may include a natural language utterance "What is the average life expectancy in the United States of America?" and a label including the corresponding logical form "SELECT AVG(life_expectancy)FROM country=United States of America" ​​(and in some cases, whether the predicted logical form is "correct" or "incorrect").

[0140] In some cases, data assets 645 also include database schema information. A database schema defines how data is organized within a database, such as a relational database; this includes logical constraints, such as table names, fields, data types, and relationships between these entities. A relational database may be formed of one or more tables, each of the one or more tables including one or more columns, each of the one or more columns including one or more values. Each table and column of a relational database may be named with a unique identifier, each identifier including one or more words. In some cases, one or more columns of a relational database may be used as a primary key, wherein each value of the one or more columns used as a primary key is unique to each other. In some cases, one or more columns of a relational database may be used as a foreign key for linking a table including one or more columns to another table in a relational database. In some cases, database schema information includes one or more data structures for storing unique identifiers of one or more tables, unique identifiers of one or more columns, and values ​​of each relational database. Data set preparer 630 may store unique identifiers and values ​​in one or more vectors and / or matrices. In some embodiments, the data structure storing schema information of a relational database may store a directed graph representing unique identifiers and values.

[0141] Once the data asset 645 is obtained, the data set can be split into a training data set and a validation data set. The split can be performed randomly (e.g., 90 / 10% or 70 / 30%), or can be performed according to more complex validation techniques (such as K-fold cross validation, leave one cross validation, leave one group cross validation, nested cross validation, etc.) to minimize sampling bias and overfitting. Before or after the split, basic natural language preprocessing can be performed on the data asset 645. In some cases, the preprocessing includes tokenizing the utterances of the data asset 645. Tokenization is the splitting of phrases, sentences, paragraphs, or entire text documents into smaller units, such as individual words or terms. Each of these smaller units is called a token. Smaller units are created by locating boundaries (such as word boundaries), which are the end point of a word and the beginning of the next word. For example, the text "How many employees work for company X" can be tokenized into "How", "many", "employees", "work", "for", "company", "X". These tokens help the model understand the context and develop a model for a given task. There are various tokenization techniques that can be used to perform tokenization based on language and modeling tasks. For example, tokenization can be performed using Natural Language ToolKit, space tokenization, dictionary-based tokenization, rule-based tokenization, Keras tokenization, Penn Tree-based tokenization, spaCy tokenization, Moses tokenization, subword tokenization, etc.

[0142] In some cases, the tokens of the data asset 645 can then be embedded into word embeddings (e.g., contextualized word embeddings). A word embedding is a learned representation of text in which words with the same meaning have similar representations. Word embeddings are generated by embedding techniques, in which individual words are represented as real-valued vectors in a predefined vector space so that they can be understood by a deep learning algorithm. The embedding techniques can be joint or separate embedding techniques, such as including an embedding layer in a deep learning algorithm or using a separate model, such as a BERT-based pre-trained language model (e.g., BERT, RoBERTa, and DeBERTa). A BERT-based model is a pre-trained language model that uses self-supervised learning to learn the deep meaning of words and context. The embedding layer is a word embedding learned jointly with a neural network model on a specific natural language processing task, such as translation from natural language to logical form (e.g., (one or more) natural language-logical form (NL-LF) algorithm).

[0143] Model training generally includes selecting hyperparameters for model 625 and using an optimization algorithm (e.g., a stochastic gradient descent algorithm or a variant thereof, such as batch gradient descent or mini-batch gradient descent) to find the model parameters that correspond to the best fit between the predicted output and the actual output. Hyperparameters are settings that can be tuned or optimized to control the behavior of model 625. Most models explicitly define hyperparameters that control different aspects of the model (such as memory or execution cost). However, additional hyperparameters can be defined and optimized to adapt the model to specific scenarios. For example, hyperparameters can include the number of hidden units of the model, the learning rate of the model, the convolution kernel width, the number of kernels used for the model, the top K results, the N clustering levels, etc.

[0144] During training by the model trainer 640, the error is calculated as the difference between the actual output (e.g., as indicated by the label 650) and the predicted output (e.g., as determined by the model 625). The function used to calculate this error is called the objective function (e.g., loss function or cost function). The error is a function of the internal parameters of the model (e.g., weights and biases). In order to accurately predict, the error needs to be minimized. In order to minimize the error, the model parameters are gradually updated by minimizing the objective function on the training examples from the preprocessed data asset 645. The objective function can be constructed to measure the difference between the output inferred using the model and the basic facts annotated to the sample using the label. For example, for a supervised learning based model, the goal of training is to learn a function "h()" (sometimes also called a hypothesis function) that maps the training input space X to the target value space Y, h: X→Y, so that h(x) is a good predictor of the corresponding value of y. A variety of different techniques can be used to learn this hypothesis function. In some machine learning algorithms (such as neural networks), this is done using backpropagation. The current error is usually propagated back to the previous layer, where it is used to modify the weights and bias values ​​in a way that minimizes the error. The weights are modified using an optimization function. The optimization function usually calculates the error gradient, i.e., the partial derivative of the objective function with respect to the weights, and the weights are modified in the opposite direction of the calculated error gradient. For example, techniques such as backpropagation, stochastic feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), Hebbian learning, etc. are used to update the model parameters in a way that minimizes or maximizes this objective function. This cycle is repeated until the minimum of the objective function is reached.

[0145] In some cases, the model trainer 640 uses the preprocessed data assets 645 (e.g., word segmentation data assets) to train (one or more) natural language logic forms (NL2LF) algorithms. In some cases, (one or more) NL2LF algorithms include encoder-decoder neural networks. The encoder consists of an input layer and one or more encoding layers. One or more encoding layers are based on contextualized word embeddings, such as BERT and its series (e.g., RoBERTa, DeBERTa, etc.). The encoder follows the embedding process to transform the relevant text (and optionally the database pattern) into a digital / vector representation to retain the conditions and connections between words and sentences so that the machine can understand the patterns associated with any text, find the context of the sentence, and optionally learn the relationship between words and given database patterns. The result of the encoder will be a state vector or context vector. This state vector will be the input of the decoder. The decoder consists of an input layer, one or more decoding layers, a dense layer, and an output layer (e.g., a layer with a softmax function). One or more decoding layers may include multiple transformer layers. The current loop unit receives the hidden state from the earlier loop unit. The result of the decoder will be in a logical form, such as an OMRL query translated from the utterances within the preprocessed data asset 645 .

[0146] Once the model trainer 640 identifies a set of model parameters, the model 625 is trained, and the validator is configured to validate the model 625 using a validation data set. The validation process performed by the validator includes an iterative operation of inputting the validation data set into the training model 625 using validation techniques (such as K-fold cross validation, leave one cross validation, leave one group cross validation, nested cross validation, etc.) to tune the model parameters and ultimately find the optimal set of model parameters. Once the optimal set of model parameters is obtained, a reserved test data set from the validation data set is input into the training model 625 to obtain the output, and the output is evaluated with the basic facts using correlation techniques (such as the Bland-Altman method and the Spearman rank correlation coefficient), and performance metrics (such as error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc.) are calculated. In some cases, the acquisition, training, and validation data processes in the model system 600 can be repeatedly performed (adjusted) by the model trainer 640 until a predetermined condition is met, and the final set of model parameters can be provided by the model trainer 640.

[0147] As should be understood, other training / validation mechanisms are contemplated and can be implemented within the model system 600. For example, the model 625 can be trained and the model parameters can be tuned on a dataset from a subset of the obtained or filtered dataset, and the dataset from the subset of the obtained or filtered dataset can only be used to test and evaluate the performance of the model 625. Moreover, although the training mechanisms described herein focus on training new models 625, these training mechanisms can also be used to fine-tune existing models trained from other datasets. For example, in some cases, the model 625 may have been pre-trained using datasets from one or more different modalities or tasks. In those cases, the model 625 can be used for transfer learning and retrained / validated using the training and validation data as described above.

[0148] The training phase 605 outputs a trained model 625 with an optimized set of model parameters for use in the NL2LF translation phase 610 or the confidence score calibration phase 615. The NL2LF translation phase 610 includes a predictor for translating input data 655 (e.g., a natural language utterance) into a logical form 660 (e.g., an OMRL query). For example, the predictor performs processing of inputting the input data 655 into a trained NL2LF model 665 and generating a logical form 660 based on features within the input data 655 using the trained NL2LF model 665. In some cases, the trained NL2LF model 665 performs one or more semantic parsing tasks to generate predictions based on features extracted from the input data 655. The NL2LF translation phase 610 outputs a logical form 660 that can be used in the confidence score calibration phase 615 and the query execution phase 620.

[0149] The query execution stage 620 includes one or more executors configured to execute the logical form 660 on a system, such as a database 680, to obtain a result 685 (e.g., an answer to a query within a (one or more) natural language utterances). For example, the one or more executors may be configured to translate or convert the meaning representation language of the logical form 860 into a system language query or command (such as SQL, API, REST, GraphQL, PGQL, etc.), and execute the system language query or command on a relational database to obtain an answer to the query posed in the (one or more) natural language utterances.

[0150] Although not explicitly shown, it should be appreciated that the model system 600 may also include a developer device associated with a developer. Communications from the developer device to components of the model system 600 may indicate what type of input data, utterances, and / or database schemas to use for the model, the number and type of models to use, hyperparameters for each model (e.g., learning rate and number of hidden layers), how to format data requests, what training data to use (e.g., and how to obtain access to the training data), and which validation techniques to use, and / or how to configure controller processing.

[0151] Techniques for converting natural language statements into intermediate database query representations

[0152] Figure 7 is a flow diagram illustrating a process 700 for using a machine learning model trained to translate natural language into an intermediate database query representation, in accordance with various embodiments. Figure 7 The processing depicted in the can be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 7 The methods presented in and described below are intended to be illustrative rather than limiting. Figure 7 Various processing steps are depicted as occurring in a particular order or sequence, but this is not limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1-Figure 6 In the embodiment depicted in FIG. Figure 7 The processing depicted in can be performed by a model system (e.g., model system 600) to create, train, update, and deploy one or more machine learning models configured to translate natural language into a logical form.

[0153] At step 702, an input string is generated by concatenating a natural language utterance with a database schema representation of a database. The database schema representation of a database includes information that defines how data is organized in the database. This can include information such as table names, fields, data types, and relationships between table names, fields, and data types. Examples of OMRL database schema representations are discussed further in the OMRL Schema section below. A natural language utterance is a text string, such as "List suppliers in order of sales last month." In some cases, the database schema representation is also in the form of a string.

[0154] In some examples, the database schema representation includes table and column names indicating the database to be queried. As a specific example, consider a simple database schema with one table (invoice) having 4 columns as shown below.

[0155]

[0156] Table 1

[0157] For the natural language utterance “What are the invoices for supplier Mary”, the example model input is represented by the following concatenation of the NL utterance and DB schema:

[0158]

[0159] Table 2

[0160] Since some databases may be associated with completely different sets of words (e.g., one for employee data, one for department store inventory, one for scientific data, etc.), including database schema specifications 436 in input 434 helps the model contextualize utterances to schemas.

[0161] As described further below, the OMRL schema includes specialized attributes. One such specialized attribute is a link attribute. A link attribute links entries that may be located in different tables. This provides functionality somewhat similar to foreign keys and primary keys in SQL. But unlike the SQL equivalents, link attributes can link entities in different tables without the need for JOINS or explicit references to different tables. In OMRL, link attributes have descriptive names that include metadata in the OMRL specification. For example, in SQL, a question "QUESTION" cascaded with the SQL's database schema representation would be:

[0162] <question>perpetrator

[0163] (perpetrator.perpetrator_id:number perpetrator.people_id:

[0164] number perpetrator.date:text perpetrator.year:number

[0165] perpetrator.location:text perpetrator.country:text

[0166] perpetrator.killed:number perpetrator.injured:number)

[0167] people(people.people_id:number perpetrator.name:text

[0168] perpetrator.height:number perpetrator.weight:number

[0169] perpetrator.home_town:text).

[0170] In OMRL, an equivalent problem cascading with an equivalent database schema representation would be:

[0171] <question>perpetrator(perpetrator_id:number pe

[0172] ople:link date:text year:number location:text country:text

[0173] killed:number injured:number)people(people_id:

[0174] number name:text height:number weight:number

[0175] home_town:text perpetrators:link)

[0176] Using linked attributes, the encoder input is shortened by removing the table name from each attribute input. In other words, in some cases, the database schema representation of the database includes linked attributes that reference entries in a table without referencing the name of the table. This results in improved training and inference latency due to shorter encoder inputs.

[0177] At step 704, based on the input string, one or more embeddings of natural language utterances and database schema representations are generated by a first encoder. The input string is provided as an input to the first encoder. In some cases, the first encoder is a pre-trained language model (PLM) 530, such as DeBERTa. The first encoder processes the input string to generate one or more embeddings of natural language utterances and database schema representations. This can be achieved using a single embedding or multiple embeddings. As described above, an embedding is a learned representation of a text in which words with the same meaning have similar representations, and the embeddings are generated by an embedding technique in which individual words are represented as real-valued vectors in a predefined vector space so that a deep learning algorithm can understand them.

[0178] At step 706, the second encoder encodes the relationship between elements in the database schema representation and words in the natural language discourse based on the generated one or more embeddings. The elements in the database schema may include table names, column names, row names, fields, relations, view indexes, directories, and / or other database schema elements. The generated one or more embeddings are provided from the first encoder to the second encoder. In some cases, the second encoder is a relation-aware transformer (RAT). The second encoder includes multiple self-attention layers configured to apply self-attention to one or more embeddings to identify the relationship between entities in the database schema representation and words in the natural language discourse. The use of RAT is described in RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (2021) by Wang et al., the entire contents of which are incorporated herein by reference for all purposes.

[0179] In some cases, pattern link relations linking elements in the database schema representation and words in the natural language utterance are also provided to the second encoder, and embeddings are further generated based on the pattern link relations. The pattern link relations provide information to help the second encoder identify how elements in the database schema representation are related to words in the natural language utterance. Pattern links are used to capture potential links between tokens in the utterance and the schema (e.g., entities / attributes in OMRL or tables / columns in SQL). The pattern link relations are encoded into the layers of the second encoder as prior knowledge.

[0180] In some cases, name-based schema linking (NBSL) is applied. NBSL is used to generate matches between tokens in natural language utterances and elements in the database schema representation. NBSL matches entities such as table names and column names with words in the input utterance, which can be based on exact or partial matches of the primary name and its synonyms with elements in the database schema representation.

[0181] In some cases, the OMRL schema includes rich metadata. This metadata includes information that specifies synonyms for different words. For example, car is a synonym for automobile. Using this rich metadata in the OMRL schema, name-based schema linking can identify elements in the database schema representation based on identifying synonyms as well as identifying exact matches. In other words, the schema link relationship includes metadata that specifies synonyms for a word. In some cases, the OMRL schema includes link attributes, as described above with respect to step 702.

[0182] In some cases, content-based pattern linking (CBSL) is applied. In CBSL for OMRL, data assets to be used in the natural language to logical form model are preprocessed based on scalable search and content-based pattern linking. The preprocessing technique combines named entity recognition and scalable search (e.g., elastic search) to obtain CBSL matches between words or tokens of an utterance and values ​​of system entities and / or attributes within a given database schema. Content-based pattern linking matches are attached to the utterance using a unique data structure, and the data structure is then input into the natural language to logical form model. The data structure facilitates encoding and decoding of input utterances into logical forms through the natural language to logical form model. Advantageously, by using scalable search and content-based pattern linking, the response time and accuracy of the natural language to logical form model can be improved over conventional standardized programming language techniques (such as SQL searches of databases using SQL similarity operators). CBSL technology is further described in detail in U.S. patent application 18 / 065,387, filed on December 13, 2022, entitled "Transforming NaturalLanguage To Structured Query Language Based On Scalable Search And Content-Based Schema Linking," the entire contents of which are hereby incorporated by reference for all purposes.

[0183] At step 708, the grammar-based decoder generates an intermediate database query representation based on the encoded relations and one or more embeddings. Figure 5 and Figure 6 As described, the grammar-based decoder applies a bottom-up generation process using a beam search to generate an OMRL tree representing a complete OMRL logical form. The grammar-based decoder obtains one or more raw bundle scores generated from one or more bundle levels of the grammar-based decoder. The one or more raw bundle scores are used to classify an intermediate database query representation as correct or incorrect.

[0184] In some cases, a relational algebra (RA) grammar is also provided to the grammar-based decoder. The RA grammar represents the intermediate database query representation as a tree. The RA grammar uses configured RA operators to represent the intermediate database query representation. The RA grammar is used to control the syntax of the decoder output to ensure that the generated logical form has the correct syntax, which will affect subsequent processing steps (including execution on an existing database). Therefore, the intermediate database query representation is further based on a relational algebra grammar. In some cases, a specialized RA grammar is suitable for OMRL, as described below with respect to Figure 8-9B Further described.

[0185] At step 710, a database query in a database query language is generated based on the intermediate database query representation and the interface specification. The interface specification establishes a mapping between the intermediate database query representation and a specific target database query language (such as SQL). The interface specification can be built into a converter (such as the OMRL2SQL converter) specific to the target database query language. In some cases, the converter is a model that does not require training (e.g., a non-deep learning model).

[0186] In some cases, a database query is performed on the database to retrieve data in response to the natural language utterance. For example, depending on the backend database of interest, a SQL query is performed, or a PGQL query is performed, etc. As an example, based on the initial speech input "What are the top three earning stores?", a database query is performed to retrieve a list of the top three earning stores from the database.

[0187] Oracle Meaning Representation Language (OMRL) as an intermediate representation

[0188] In some cases, the intermediate database query language is Oracle Meaning Representation Language (OMRL), a semantic parser logical formal target language that covers a wide range of backend systems, including SQL, PGQL (Oracle's property graph-based query language), decision rules, and knowledge graph queries.

[0189] Design principles

[0190] Table orientation, no fixed body

[0191] OMRL is designed to query and manipulate tabular data, such as databases, decision tables, APIs that generate tabular data, etc. OMRL adapts to the ontology of the backend system.

[0192] Each backend system is described by a schema, which specifies:

[0193] The table tuple name and the slot or column name within each tuple (e.g., dept_name )

[0194] Natural language versions of table and column names (e.g., "Department Name")

[0195] The type of values ​​in each column (e.g., number, currency, string, etc.)

[0196] Sample values ​​for each slot or column, or NER type for each column

[0197] Generate any additional information needed for backend representation (e.g. primary and foreign key links for SQL)

[0198] Table Orientation

[0199] An OMRL query has 3 parts:

[0200] An optional FROM clause that specifies the source of the query.

[0201] No JOIN (but can be inferred in OMRL=>SQL translation)

[0202] A WHERE clause, which filters data as in standard SQL, and

[0203] • The OPERATION clause (eg, SELECT, INSERT, UPDATE, etc.) specifies the operation that should be applied to the filtered data.

[0204] OMRL consists of the following main components:

[0205] OMRL schemas define valid OMRL expressions for OMRL applications. (OMRL schemas are the OMRL version of database schemas used in SQL databases).

[0206] The OMRL interface specification specifies how to instantiate OMRL in C2OMRL and in backend systems.

[0207] o The C2OMRL mapping specification provides synonyms that a particular C2OMRL system may require, links to NER systems, etc. The interface specification also specifies how to translate OMRL into a backend system representation (such as SQL, PGQL, logical SQL, etc.).

[0208] The separation of OMRL patterns from OMRL interface specifications hides the complexity of interface components behind other interfaces. Interface specifications can vary depending on the interface.

[0209] If a new C2OMRL system requires additional metadata, then that metadata is added to its C2OMRL interface specification without interfering with existing C2OMRL systems.

[0210] Interfacing with different backends may require different interface specifications.

[0211] OMRL grammar design

[0212] OMRL expressions can be defined in JavaScript Object Notation (JSON) format.

[0213] For example:

[0214] OMRL Expression

[0215] {"select":[ <col-value>,...],"from": <from-value>,

[0216] "where": <value>,

[0217] "order_by":[ <dir-col>,...],"limit": <count>,

[0218] "group_by":[,...],"having": <value>

[0219] }

[0220] These JSON features are optional; for example, if there are no WHERE constraints, then the "where" feature is not required.

[0221] Entity types, properties, and links

[0222] Entity types correspond to SQL tables.

[0223] Entities correspond to rows in SQL tables.

[0224] Attributes correspond to SQL columns.

[0225] A link is a property that corresponds to a foreign key-primary key pair in SQL.

[0226] :col consists of either non-linked attributes of the entity named in the from: attribute or a list consisting of:

[0227] Zero or more connected link attributes, beginning with a link to the entity identified by the from: attribute,

[0228] Followed by the attributes of the entity identified by the last link attribute, or if there are no link attributes, by the from: attribute.

[0229] Example: ["Date"]

[0230] Example: ["PurchasedProduct","Price"]

[0231] <col-value>:col-value is a list consisting of an optional aggregation operation followed by a .

[0232] Example: ["#MAX","PurchasedProduct","Price"]

[0233] Aggregate operations begin with a special character "#" to distinguish the aggregate operation from the entity name.

[0234] <from-value>:from-value is an entity or an OMRL expression. (This allows nested OMRL expressions)

[0235] <value>The :where constraint is a Boolean expression. Here are some examples of values:

[0236] Strings, numbers, and Booleans are all values.

[0237] · <col-value>List

[0238] ·One-dimensional, two-dimensional (or n-dimensional) functions are applicable to <value> 。

[0239] ·[ <unary-op> , <value> ]

[0240] ·[ <binary-op> , <value> , <value>]

[0241] Example: ["=", ["Purchased_Product", "Price"], 23.50]

[0242] Unary operators starting with a letter <unary-op>Prefixed with "#" so they cannot be confused with entity names. (This does not happen with binary operators.)

[0243] We use Boolean <binary-op>(such as AND, OR) to combine Boolean values.

[0244] <dir-col>: This is used in the order_by clause. It can be either <col-value>(list), or with a direction prefix <col-value>There are two directions: "#ASC" (default) and "#DESC". Example:

[0245] ·["#Desc","Purchased_Product","Price"]

[0246] ["#Desc", "#COUNT", "*"]

[0247] OMRL Examples

[0248] The following example illustrates the use of select, from, and where in OMRL. The first element of the list tells us what operation should be applied; for example, Purchase and Products are table names, so the list is a table_column lookup. > is a binary operation that applies to the other arguments.

[0249] {"select":[["Date"],["PurchasedProduct",

[0250] "Product_Name"]],"from":"Purchase",

[0251] "where":[">",["Quantity"],10]}.

[0252] The equivalent SQL backend is:

[0253] SELECT Purchase.Date, Products.Product_Name

[0254] FROM Purchase JOIN Products ON Purchase.Product_ID=

[0255] Product.ID WHERE Purchase.Quantity>10.

[0256] The following example illustrates the use of order by and limit in OMRL:

[0257] {"select":[["PurchasedProduct","Product_Name"],

[0258] ["Quantity"]],"from":"Purchase",

[0259] "order_by":[["Quantity"]],"limit":2}.

[0260] In contrast, the following example illustrates the equivalent SQL backend query:

[0261] SELECT Products.Product_Name, Purchase.Quantity

[0262] FROM Purchase JOIN Products ON Purchase.Product_ID=

[0263] Purchase.ID ORDERED BY Purchase.Quantity LIMIT 2.

[0264] The following example illustrates the use of group by and having in OMRL:

[0265] {"select":[["PurchasedProduct","Product_Name"],

[0266] ["Quantity"]],"from":"Purchase",

[0267] "group_by":[["PurchasedProduct","Product_Name"]],"having":[">",[#SUM,"Quantity"],10]}.

[0268] In contrast, the following example illustrates the equivalent SQL backend query:

[0269] SELECT Products.Product_Name, Purchase.Quantity

[0270] FROM Purchase JOIN Products ON Purchase.Product_ID=Purchase.ID GROUPBY Products.Product_Name HAVING SUM(Purchase.Quantity)>10.

[0271] OMRL Mode

[0272] In some cases, developers provide an OMRL Schema file that specifies the interface between the C2OMRL system and the backends that C2OMRL can interface with. This design is similar to the Spider tables.json schema file.

[0273] The following example illustrates a simple OMRL schema. It has 2 tables, recording purchases and products.

[0274] {

[0275] "entities":[

[0276] {

[0277] "entity":"Product","attributes":[

[0278] {"attribute":"ProductID","type":"number"},

[0279] {"attribute":"Product_Name","type":"string"},

[0280] {"attribute":"ProductPurchases","type":"1-to-many_link","to":"Purchase"} ]

[0282] },

[0283] {

[0284] "entity":"Purchase","attributes":[

[0285] {"attribute":"Date","type":"string"},

[0286] {"attribute":"Quantity","type":"number"},

[0287] {"attribute":"PurchasedProduct","type":"many-to-1_link","to":"Product"} ]

[0289] }]

[0290] }.

[0291] This mode corresponds to a SQL database with the following schema:

[0292] Products(ProductID,Product_Name),Purchase(Date,Quantity,PurchasedProductID), and there is a foreign key~primary key relationship between PurchasedProductID and ProductID.

[0293] This pattern also corresponds to the PGQL property graph with the following vertices (entities) and edges (links):

[0294] Products(ProductID,Product_Name),Purchase(Date,Quantity)Purchase-(PurchasedProduct)->Product.

[0295] Link

[0296] A link is a special attribute that points to another entity (e.g., an entry in another table). There are two kinds of links: one-to-many links and many-to-one links, corresponding to SQL primary key-foreign key and SQL foreign key-primary key relationships, respectively.

[0297] Links usually come in pairs (because they correspond to a single SQL foreign key-primary key relationship), for example, ProductPurchases is the reverse link of PurchasedProduct. The reverse_link attribute can be used to encode which link is the reverse of another link. The reverse_link attribute is not required when there is only one way to connect two entities. While SQL foreign keys and primary keys use a JOIN operation to relate entries in different tables, the OMRL link attributes do not require a JOIN operation. These improved link attributes are used in the RAT features and in the RA syntax used in the OMRL pipeline.

[0298] The following example illustrates an OMR schema with multiple links. A flight involves a source airport and a destination airport. Flight is represented by an entity type, and Airport is represented by an entity type. There are two links from the Flight entity to the Airport entity, one for the source airport and one for the destination airport. This differentiates these airports.

[0299] {"entities":[

[0300] {"entity":"Airports","attributes":[

[0301] {"attribute":"City","type":"string"},

[0302] {"attribute":"AirportCode","type":"string"}

[0303] {"attribute":"DepartingFlights","type":"1-to-many_link","to":"Flights","reverse_link":"DepartureAirport"},

[0304] {"attribute":"ArrivingFlights", "type":"1-to-many_link", "to":"Flights", "reverse_link":"DestinationAirport"},

[0305] ]},

[0306] {"entity":"Flights","attributes":[

[0307] {"attribute":"FlightNumber","type":"number"},

[0308] {"attribute":"DepartureAirport", "type":"many-to-1_link", "to":"Airports", "reverse_link":"DepartingFlights"},

[0309] {"attribute":"DestinationAirport", "type":"many-to-1_link", "to":"Airports", "reverse_link":"ArrivingFlights"},

[0310] ]}

[0311] ]}

[0312] The reverse_link JSON feature on links indicates which links are the reverse of which links.

[0313] The following example illustrates the SQL and OMRL representation of "How many flights arriving in Aberdeen city?".

[0314] SQL:

[0315] SELECT COUNT(*)FROM flights JOIN airports ON flights.destairport=airports.airportcode WHERE airports.city

[0316] ='Aberdeen'

[0317] The OMRL2SQL translator must resolve the SourceAirport and DestAirport links into JOINs on the appropriate foreign keys.

[0318] OMRL: Table_column operation is replaced by a list of table, column, or link names.

[0319] {"select":[["#COUNT","*"]],

[0320] "from":"Flights",

[0321] "where":["=",["DestinationAirport","City"],

[0322] "Aberdeen"]}

[0323] This uses the first element of the list to tell us what kind of operation should be applied.

[0324] The following example illustrates OMRL representation of "(Where can I fly to from Sydney? Where can I fly to from Sydney?)"

[0325] {"select":"City",

[0326] "from":"Airports",

[0327] "where":["=",["ArrivingFlights","DepartingFrom","City"],"Sydney"]

[0328] }

[0329] OMRL example and schema with three tables

[0330] This example extends the previous Airports and Flights example by adding an airline entity.

[0331] OMRL Mode:

[0332]

[0333]

[0334] The following example illustrates OMRL for "Which airlines fly from Sydney to SanFranciso?"

[0335] {"select":["FlightAirline","Name"]"from":"Flights",

[0336] "where":["AND",["=",["DepartureAirport",

[0337] "City"],"Sydney"],

[0338] ["=",["DestinationAirport","City"],"San Francisco"]].

[0339] }

[0340] The role of a semantic parser

[0341] The semantic parser is responsible for generating an OMRL consisting of SELECT, FROM, WHERE, ORDER BY, GROUP BY, HAVING, and LIMIT clauses. The minimum requirements for an OMRL query are SELECT and FROM. In many cases, SELECT will be just "*". If the NL query explicitly includes a request for other clauses, then other clauses can be added. The semantic parser includes in the OMRL those entities and attributes that are explicitly requested in the NL query. If the semantic parser does not detect any attributes in the NL query, then it includes all attributes (SELECT*). The semantic parser generates predicates for the attributes of an entity based only on the conditions detected in the NL query. These predicates support standard comparison operators such as (=, !=, <, >, <=, >=, LIKE, NOT LIKE).

[0342] In some cases, the semantic parser includes ORDER BY in the OMRL if explicitly specified in the NL query via a superlative term (such as "top"), an explicit ordering (such as "byxxx"), or if the semantic parser detects a superlative term (such as "top" or "best"), it includes ORDER BY *DESC.

[0343] The role of the conversion component

[0344] As mentioned above about Figure 4-Figure 7 As described, the conversion component converts an intermediate database query representation (e.g., in OMRL) into a database query in a specific database query language (e.g., SQL). The following examples relate to the OMRL2SQL component, but it should be understood that these examples can be applied to a conversion component configured for another target database query language (such as PGQL, etc.).

[0345] If the source ORML query does not include a default clause, the OMRL2SQL component appends that clause to the final SQL statement, with the exception of the default Where clause, which is applied regardless of whether it is included in the OMRL query. If the OMRL SELECT includes all attributes of an entity (SELECT *) and the OMRL does not include an ORDER BY clause, the default ordering of the entity is applied (order-by). If the OMRL SELECT includes a single attribute but does not include an ORDER BY, the results should be sorted based on the natural sort order of the requested attribute.

[0346] If the OMRL SELECT explicitly includes multiple attributes, and does not include an ORDER BY, then the results are sorted based on the natural sort order of each requested attribute in the order they appear in the SELECT clause. For example, if the user says "show me employees and the locations where they work", then the results are sorted by employees, locations. If the user says "show me locations and the employees that work there", then the results are sorted by locations, employees. "DESC" is used in the order-by clause to indicate "Top". The OMRL specification includes a "superlativeOrder" attribute. If "DESC" is detected, then the value of this attribute can be used by the OMRL2SQL component for the actual sort order.

[0347] OMRL Specifications Consumption

[0348] As mentioned above, in OMRL, there are no foreign keys and primary keys as in standard SQL. Instead, OMRL introduces so-called "link attributes". Table 3 illustrates the terminology used in the OMRL specification. Table 4 shows examples of link attributes in OMRL and comparisons with SQL foreign keys and primary keys.

[0349]

[0350] Table 3 - SQL and OMRL

[0351]

[0352]

[0353]

[0354]

[0355]

[0356]

[0357] Table 4 - OMRL and SQL specifications

[0358] OMRL RAT Characteristics

[0359] As mentioned above about Figure 7 As described, specialized RAT features are used for OMRL. Table 5 illustrates an example RA syntax for OMRL.

[0360]

[0361]

[0362] Table 5: RA syntax

[0363] One specialized RA operation of OMRL is the Cartesian product operator. As mentioned above, OMRL uses link attributes to identify related entries in different tables. Unlike SQL, which uses JOIN operations, in OMRL, link attributes are used instead to identify entries in different tables without the need for table joins. The Cartesian product operator represents the link attribute in OMRL. If there are multiple links in the link attribute, they will be presented in a binary subtree in a left-to-right manner. Other specialized RA operations of OMRL include like start / not like start and like end / not like end operators. These operators are used to handle fuzzy searches, as described in further detail below.

[0364] Figure 8 An example of an RA tree is illustrated, which illustrates the linking capability 800. For the utterance "What is the ID and name of the employee with the highest salary?", the OMRL logical form is:

[0365] OMRL:{'select':[['eid'],['name']],'from':'employee',

[0366] 'order_by':[['#DESC',['salary']]],'limit':1}

[0367] This is due to Figure 8 The RA tree 800 shown in FIG.

[0368] Fuzzy search capability

[0369] As mentioned above, the OMRL RA grammar is suitable for supporting fuzzy search capabilities. Fuzzy search is a search that goes beyond exact matches to find terms that approximately match another term. Fuzzy search uses a pattern matching algorithm to identify such approximate matches. The fuzzy search capability is extended by the like operator and supports additional start / end searches. As an example, using like(criteria), for the NL question: "What is all the information about employees with D orS in their first name, ordered by salary descending?", the SQL-like query would be:

[0370] SELECT*FROM employees WHERE FIRST_NAME

[0371] LIKE'%D%'OR FIRST_NAME LIKE'%S%'ORDER

[0372] BY SALARY DESC.

[0373] The equivalent OMRL query using the standard like is:

[0374] OMRL:{'select':[['*']], 'from':'employees', 'where':['OR',

[0375] ['LIKE',['FIRST_NAME'],'%d%'],['LIKE',

[0376] ['FIRST_NAME'],'%s%']],'order_by':[['#DESC',

[0377] ['SALARY']]]}

[0378] The following example illustrates the use of like with Starting support. For the NL question: "Which catalog contents have a product stock number that starts from '2?' Show the catalog entry names", the SQL-like query is:

[0379] SELECT catalog_entry_name FROM Catalog_Contents

[0380] WHERE product_stock_number LIKE'2%'.

[0381] The equivalent OMRL query using LIKE with leading support is:

[0382] OMRL:{'select':[['catalog_entry_name']],'from':

[0383] 'Catalog_Contents','where':['LIKE',

[0384] ['product_stock_number'],'2%']}

[0385] The difference from like (standard) is LIKE '2%' (the % is at the end of the value).

[0386] The following example illustrates the use of like with ending support. For the NL problem: "Return the fullnames and salaries for employees with first names that end with the letter m", the SQL-like query is:

[0387] SELECT FIRST_NAME,LAST_NAME,SALARY

[0388] FROM employees WHERE FIRST_NAME LIKE'%m'.

[0389] The equivalent OMRL query using LIKE with trailing support is:

[0390] OMRL:{'select':[['FIRST_NAME'],['LAST_NAME'],

[0391] ['SALARY']], 'from':'employees', 'where':['LIKE',

[0392] ['FIRST_NAME'], '%m']}

[0393] Like_start and Like_end operators are specialized RA operators for fuzzy search. Like_start is similar to like, but supports fuzzy search starting with a string. Fig. 9A An example RA tree 900 with Like_start is shown in FIG. For the utterance "Which catalog contents have a product stock number that startsfrom'2'? Show the catalog entry names", the corresponding OMRL is {'select': [['catalog_entry_name']], 'from': 'Catalog_Contents', 'where': ['LIKE', ['product_stock_number'], '2%']}, and the transformed RA tree is shown in FIG. Fig. 9A Shown in.

[0394] Like_end is also similar to like, but supports fuzzy searches starting with a string. Fig. 9B An example RA tree with Like_end 950 is shown in . For the utterance "Return the full names and salaries for employees with first names that end with the letter m", the corresponding OMRL is {{'select':[['FIRST_NAME'],['LAST_NAME'],['SALARY']],'from':'employees','where':['LIKE',['FIRST_NAME'],'%m']}, and the converted RA tree is in Fig. 9B Shown in.

[0395] Additional features of OMRL include improved attribute selection by redefining pattern link components and input encoding. In addition, OMRL implements a design pick list for numbers and currencies, which provides improved accuracy for numbers and currencies.

[0396] Experimental Results

[0397] The experimental results are shown in Tables 6 and 7. The experiments in Table 6 are conducted using the Spider training data (converted to OMRL).

[0398]

[0399] Table 6: Experimental results

[0400] Model accuracy is the exact match accuracy (%) of the C2OMRL model on the evaluation dataset. The evaluation set includes a targeted evaluation set for evaluating the impact of new operators and a standard evaluation set for evaluating whether the new operator has side effects on other capabilities. Table 7 shows the experimental results using a dataset for evaluating the impact of specialized RA operators. As shown in Tables 6 and 7, the techniques described in this article provide significant accuracy improvements over previous systems.

[0401]

[0402]

[0403] Table 7: Experimental results

[0404] Descriptive System

[0405] Fig.10 Depicted is a simplified diagram of a distributed system 1000. In the illustrated example, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 may be configured to execute one or more applications.

[0406] In various examples, the server 1012 may be adapted to run one or more services or software applications that enable one or more embodiments described in the present disclosure. In some examples, the server 1012 may also provide other services or software applications that may include non-virtualized and virtualized environments. In some examples, these services may be provided to users of client computing devices 1002, 1004, 1006, and / or 1008 as web-based services or cloud services, such as under a software as a service (SaaS) model. Users operating client computing devices 1002, 1004, 1006, and / or 1008 may then interact with the server 1012 using one or more client applications to utilize the services provided by these components.

[0407] exist Fig.10 In the configuration depicted in , the server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by the server 1012. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. It should be appreciated that a variety of different system configurations are possible, which may differ from the distributed system 1000. Therefore, Fig.10 The example shown in is one example of a distributed system for implementing the example system and is not intended to be limiting.

[0408] A user may use client computing devices 1002, 1004, 1006, and / or 1008 to execute one or more applications, models, or chatbots, which may generate one or more events or models, which may then be implemented or serviced in accordance with the teachings of the present disclosure. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although Fig.10 Only four client computing devices are depicted, but any number of client computing devices may be supported.

[0409] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors and other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, Linux or Linux-like operating systems such as Google Chrome TM OS), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM , Palm Portable handheld devices may include cellular phones, smart phones (e.g., ), tablets (e.g. ), personal digital assistants (PDAs), etc. Wearable devices can include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Gesture Input Devices Microsoft Game consoles, Sony System, by Various game systems provided by and others), etc. The client device may be able to execute a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use various communication protocols.

[0410] The network(s) 1010 may be any type of network familiar to those skilled in the art, and may support data communications using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internetwork Packet Exchange), The network(s) 1010 may be, by way of example only, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., in any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suites, and / or any other wireless protocol operating under the network) and / or any combination of these and / or other networks.

[0411] The server 1012 may be composed of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, The server 1012 may be a server, a mid-range server, a mainframe computer, a rack-mounted server, etc.), a server farm, a server cluster, or any other suitable arrangement and / or combination. The server 1012 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain a virtual storage device of the server. In various examples, the server 1012 may be suitable for running one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0412] The computing system in the server 1012 can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. The server 1012 can also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines) and other commercially available database servers.

[0413] In some embodiments, server 1012 may include one or more applications to analyze and integrate data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. As examples, data feeds and / or event updates may include, but are not limited to, data received from one or more third-party information sources and continuous data streams. feed, Updates or real-time updates, which may include real-time events related to sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 1012 may also include one or more applications that display data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0414] The distributed system 1000 may also include one or more data repositories 1014, 1016. In some examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 1014, 1016 may be used to store information, such as information related to chatbot performance or generated models, for use by the chatbot used by the server 1012 when performing various functions according to various embodiments. The data repositories 1014, 1016 may reside in various locations. For example, the data repository used by the server 1012 may be local to the server 1012 or may be remote from the server 1012 and communicate with the server 1012 via a network-based connection or a dedicated connection. The data repositories 1014, 1016 may be of different types. In some examples, the data repository used by the server 1012 may be a database, such as a relational database, such as an Oracle database. Databases provided by companies and other suppliers. One or more of these databases may be adapted to respond to SQL formatted commands to store, update, and retrieve data in the database.

[0415] In some examples, the application may also use one or more of the data repositories 1014, 1016 to store application data. The data repositories used by the application may be of different types, such as, for example, a key-value storage repository, an object storage repository, or a general storage repository backed by a file system.

[0416] In some examples, the functionality described in this disclosure can be provided as a service via a cloud environment. Fig.11 is a simplified block diagram of a cloud-based system environment in which various services can be provided as cloud services according to certain examples. Fig.11 , cloud infrastructure system 1102 may provide one or more cloud services that may be requested by users using one or more client computing devices 1104, 1106, and 1108. Cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above for server 1012. The computers in cloud infrastructure system 1102 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0417] The network(s) 1110 can facilitate communication and data exchange between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network(s) 1110 can include one or more networks. The networks can be of the same or different types. The network(s) 1110 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0418] Fig.11 The example depicted in FIG. 1 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in some other examples, the cloud infrastructure system 1102 may have more Fig.11 The components shown may be more or fewer components, may be combined with two or more components, or may have a different configuration or arrangement of components. Fig.11 Three client computing devices are depicted, but any number of client computing devices may be supported in alternative examples.

[0419] The term "cloud service" is generally used to refer to services that are made available to users on demand by a service provider's system (e.g., cloud infrastructure system 1102) and via a communications network such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own internal servers and systems. The cloud service provider's system is managed by the cloud service provider. Customers can therefore take advantage of cloud services provided by a cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the service. For example, a cloud service provider's system can host applications, and users can order and use the applications on demand via the Internet without having to purchase infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle Corp. of Redwood Shores, California, offers cloud services. The company provides several cloud services such as middleware services, database services, Java cloud services, etc.

[0420] In some examples, cloud infrastructure system 1102 can provide one or more cloud services using different models such as software as a service (SaaS) model, platform as a service (PaaS) model, infrastructure as a service (IaaS) model, and other models including hybrid service models. Cloud infrastructure system 1102 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0421] The SaaS model enables applications or software to be delivered to customers as a service over a communication network such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by the cloud infrastructure system 1102. Examples of SaaS services provided by the Company include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and others.

[0422] The IaaS model is typically used to provide infrastructure resources (e.g., servers, storage devices, hardware, and networking resources) to customers as cloud services to provide elastic computing and storage capabilities. The company provides a variety of IaaS services.

[0423] The PaaS model is typically used to provide platform and environment resources as a service, which enables customers to develop, run, and manage applications and services without having to purchase, build, or maintain such resources. Examples of PaaS services provided by the Company include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

[0424] Cloud services are typically provided on an on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure basis. For example, a customer may order one or more services provided by the cloud infrastructure system 1102 via a subscription order. The cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use speech to request that the cloud infrastructure system take an action (e.g., an intent), as described above, and / or provide services for the chatbot system described herein. The cloud infrastructure system 1102 may be configured to provide one or even multiple cloud services.

[0425] The cloud infrastructure system 1102 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 1102 can be owned by a third-party cloud service provider, and the cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In certain other examples, under a private cloud model, the cloud infrastructure system 1102 can be operated within an organization (e.g., within an enterprise organization) and provide services to customers within the organization. For example, customers can be various departments of an enterprise, such as a human resources department, a payroll department, etc., or even individuals within an enterprise. In certain other examples, under a community cloud model, the cloud infrastructure system 1102 and the services provided can be shared by several organizations in a related community. Various other models can also be used, such as a hybrid of the models mentioned above.

[0426] Client computing devices 1104, 1106, and 1108 may be of different types (such as Fig.10 1004, 1006, and 1008) and may be capable of operating one or more client applications. A user may use a client device to interact with the cloud infrastructure system 1102, such as to request services provided by the cloud infrastructure system 1102. For example, a user may use a client device to request information or actions from a chatbot, as described in the present disclosure.

[0427] In some examples, the processing performed by the cloud infrastructure system 1102 to provide services may involve model training and deployment. Such analysis may involve using, analyzing, and manipulating data sets to train and deploy one or more models. Such analysis may be performed by one or more processors, possibly processing data in parallel, using data to perform simulations, and the like. For example, big data analysis may be performed by the cloud infrastructure system 1102 to generate and train one or more models for a chatbot system. The data used for such analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0428] As in Fig.11 As depicted in the example of , cloud infrastructure system 1102 may include infrastructure resources 1130 for facilitating the provision of various cloud services provided by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that may be used to service storage requested from an application may be part of cloud infrastructure system 1102. In other examples, the storage virtual machine may be part of a different system.

[0429] In some examples, to facilitate efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 1102 for different customers, the resources can be bundled into groups of resources or resource modules (also referred to as "pods"). Each resource module or cluster can include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods can be pre-provisioned for different types of cloud services. For example, a first group of pods can be provisioned for database services, and a second group of pods can be provisioned for Java services, where the second group of pods can include a different combination of resources than the pods in the first group of pods. For some services, resources allocated to provision the service can be shared between services.

[0430] The cloud infrastructure system 1102 itself may internally use services 1132 that are shared by different components of the cloud infrastructure system 1102 and facilitate the service provisioning of the cloud infrastructure system 1102. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, and the like.

[0431] The cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. Fig.11 As shown, the subsystem may include a user interface subsystem 1112 that enables users or customers of the cloud infrastructure system 1102 to interact with the cloud infrastructure system 1102. The user interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 (where cloud services provided by the cloud infrastructure system 1102 are advertised and available for purchase by consumers), and other interfaces 1118. For example, a customer may use a client device to request (service request 1134) one or more services provided by the cloud infrastructure system 1102 using one or more of the interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 1102, and place an order for one or more services provided by the cloud infrastructure system 1102 that the customer wishes to subscribe to. The service request may include information identifying the customer and the one or more services that the customer wishes to subscribe to. For example, a customer may place an order for services provided by the cloud infrastructure system 1102. As part of an Order, Customer may provide information identifying the chatbot system for which services are to be provided and, optionally, one or more credentials for the chatbot system.

[0432] In some examples, such as Fig.11 In the example depicted in , cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this process, OMS 1120 may be configured to: create an account for a customer (if not already created); receive billing and / or accounting information from the customer, which will be used to bill the customer for providing the requested service to the customer; verify the customer information; after verification, book an order for the customer; and orchestrate various workflows to prepare the order for supply.

[0433] Once properly authenticated, OMS 1120 may call an order provisioning subsystem (OPS) 1124, which is configured to provision resources for the order, including processing resources, memory resources, and networking resources. Provisioning may include allocating resources to an order, and configuring resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for an order and the type of resources provisioned may depend on the type of cloud service that the customer has ordered. For example, according to a workflow, OPS 1124 may be configured to determine a specific cloud service requested, and identify multiple clusters that may have been pre-configured for that specific cloud service. The number of clusters allocated for an order may depend on the size / quantity / level / scope of the requested service. For example, the number of clusters to be allocated may be determined based on the number of users supported by the service, the duration of the service being requested, and the like. The allocated clusters may then be customized for a specific requesting customer to provide the requested service.

[0434] In some examples, the setup phase processing described above can be performed as part of the provisioning process by cloud infrastructure system 1102. Cloud infrastructure system 1102 can generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by cloud infrastructure system 1102 itself or from storage virtual machines provided by systems other than cloud infrastructure system 1102.

[0435] The cloud infrastructure system 1102 may send a response or notification 1144 to the requesting client to indicate when the requested service is now ready for use. In some cases, information (e.g., a link) may be sent to the client enabling the client to begin using and taking advantage of the benefits of the requested service. In some examples, for the client requesting the service, the response may include a chatbot system ID generated by the cloud infrastructure system 1102 and information identifying the chatbot system selected by the cloud infrastructure system 1102 for the chatbot system corresponding to the chatbot system ID.

[0436] The cloud infrastructure system 1102 can provide services to multiple customers. For each customer, the cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the order, and providing the requested services to the customer. The cloud infrastructure system 1102 can also collect usage statistics about the customer's use of the subscribed services. For example, statistics can be collected for the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information can be used to bill the customer. Billing can be performed, for example, on a monthly cycle.

[0437] The cloud infrastructure system 1102 may provide services to multiple customers in parallel. The cloud infrastructure system 1102 may store information of these customers, including potentially proprietary information. In some examples, the cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 that is configured to manage customer information and provide separation of managed information so that information associated with one customer is not accessible to another customer. The IMS 1128 may be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles and related capabilities, and the like.

[0438] Fig.12 An example of a computer system 1200 is shown. In some examples, the computer system 1200 can be used to implement any digital assistant or chatbot system in a distributed environment and within the various servers and computer systems described above. Fig.12 As shown, computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. Storage subsystem 1218 may include non-transitory computer-readable storage media, including storage media 1222 and system memory 1210.

[0439] The bus subsystem 1202 provides a mechanism for enabling the various components and subsystems of the computer system 1200 to communicate with each other as desired. Although the bus subsystem 1202 is schematically shown as a single bus, alternative examples of the bus subsystem can utilize multiple buses. The bus subsystem 1202 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard, etc.

[0440] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core or multi-core processor. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1204 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing subsystem 1204 may be implemented using custom circuits, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0441] In some examples, the processing units in the processing subsystem 1204 can execute instructions stored in the system memory 1210 or on the computer-readable storage medium 1222. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in the system memory 1210 and / or on the computer-readable storage medium 1222, including possibly on one or more storage devices. With appropriate programming, the processing subsystem 1204 can provide the various functions described above. In the case where the computer system 1200 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0442] In some examples, a processing acceleration unit 1206 may optionally be provided for performing customized processing or for offloading some of the processing performed by processing subsystem 1204 , thereby speeding up the overall processing performed by computer system 1200 .

[0443] I / O subsystem 1208 may include devices and mechanisms for inputting information to computer system 1200 and / or for outputting information from or via computer system 1200. In general, use of the term "input device" is intended to include all possible types of devices and mechanisms for inputting information to computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as a mouse or trackball, touch pads or touch screens incorporated into a display, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include software such as Microsoft Windows that enables a user to control and interact with input devices. Motion sensing and / or gesture recognition devices, Microsoft 360 game controllers, devices that provide an interface for receiving input using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as detecting eye activity from a user (e.g., a "wink" when taking a picture and / or making a menu selection) and translating eye gestures to input devices (e.g., Google ) In addition, the user interface input device may include a user interface that enables the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0444] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or indicator rods, game pads and graphic tablets, and audio / video devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, and eye tracking devices. In addition, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0445] In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1200 to a user or other computers. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as one utilizing a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0446] The storage subsystem 1218 provides a repository or data repository for storing information and data used by the computer system 1200. The storage subsystem 1218 provides a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide some of the example functionality. The storage subsystem 1218 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 can also provide authentication according to the teachings of the present disclosure.

[0447] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. Fig.12 As shown, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include multiple memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines that help to transfer information between elements within the computer system 1200, such as during startup, may typically be stored in ROM. The RAM typically contains data and / or program modules currently operated and executed by the processing subsystem 1204. In some embodiments, the system memory 1210 may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0448] By way of example and not limitation, Fig.12 As shown, system memory 1210 may load application programs 1212 being executed, which may include various applications (such as a web browser, a middle-tier application, a relational database management system (RDBMS)), program data 1214, and an operating system 1216. As an example, operating system 1216 may include various versions of Microsoft Apple and / or Linux operating system, various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems such as iOS, Phone, OS, OS, OS operating system, and other operating systems.

[0449] Computer-readable storage media 1222 may store programming and data structures that provide some example functionality. Computer-readable media 1222 may provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 1200. When executed by processing subsystem 1204, software (programs, code modules, instructions) that provide the above functionality may be stored in storage subsystem 1218. As an example, computer-readable storage media 1222 may include non-volatile memory, such as a hard drive, a disk drive, a hard drive such as a CD ROM, a DVD, (Blu-ray) discs or other optical media. Computer readable storage media 1222 may include, but are not limited to, Drive, flash memory card, universal serial bus (USB) flash drive, secure digital (SD) card, DVD disk, digital video tape, etc. The computer-readable storage medium 1222 may also include a solid-state drive (SSD) based on non-volatile memory (such as a flash memory-based SSD, an enterprise flash drive, a solid-state ROM, etc.), a volatile memory-based SSD (such as a solid-state RAM, dynamic RAM, static RAM, DRAM-based SSD, magnetoresistive RAM (MRAM) SSD), and a hybrid SSD using a combination of DRAM-based and flash memory-based SSDs.

[0450] In some examples, storage subsystem 1218 may also include a computer-readable storage medium reader 1220, which may also be connected to computer-readable storage medium 1222. Reader 1220 may receive and be configured to read data from a memory device such as a disk, a flash drive, or the like.

[0451] In some examples, computer system 1200 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, computer system 1200 can provide support for executing one or more virtual machines. In some examples, computer system 1200 can execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Memory, computing (e.g., processors, cores), I / O, and networking resources can be allocated to each virtual machine. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same or different from the operating system executed by other virtual machines executed by computer system 1200. Thus, computer system 1200 can potentially run multiple operating systems simultaneously.

[0452] The communication subsystem 1224 provides an interface to other computer systems and networks. The communication subsystem 1224 is used as an interface for receiving data from the computer system 1200 and transmitting data to other systems. For example, the communication subsystem 1224 can enable the computer system 1200 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and sending information to the client devices. For example, when the computer system 1200 is used to implement Figure 1 When the robot system 120 is depicted, the communication subsystem can be used to communicate with the chat robot system selected for the application.

[0453] The communication subsystem 1224 can support both wired and / or wireless communication protocols. In some examples, the communication subsystem 1224 may include a radio frequency (RF) transceiver component, a global positioning system (GPS) receiver component, and / or other components for accessing a wireless voice and / or data network (e.g., using cellular telephone technology, advanced data network technology (such as 3G, 4G or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards), or other mobile communication technologies, or any combination thereof). In some examples, the communication subsystem 1224 may provide a wired network connection (e.g., Ethernet) in addition to or in lieu of a wireless interface.

[0454] The communication subsystem 1224 can receive and transmit data in various forms. In some examples, the communication subsystem 1224 can receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., among other forms. For example, the communication subsystem 1224 can be configured to receive real-time data from users of social media networks and / or other platforms such as feed, The data feed 1226 is received (or sent) by other communication services such as web feeds, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0455] In some examples, the communication subsystem 1224 can be configured to receive data that may be continuous or unbounded in nature without a clear end in the form of a continuous data stream, where the continuous data stream can include an event stream 1228 of real-time events and / or event updates 1230. Examples of applications that generate continuous data can include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0456] The communication subsystem 1224 may also be configured to transmit data from the computer system 1200 to other computer systems or networks. The data may be transmitted in a variety of different forms, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. to one or more databases that may communicate with one or more streaming data sources coupled to the computer system 1200.

[0457] Computer system 1200 may be one of various types, including a handheld portable device (e.g., Cellular phone, computing tablets, PDAs), wearable devices (e.g., Google head mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, Fig.12 The description of the computer system 1200 depicted in FIG. 1 is intended to be a specific example only. Fig.12 Many other configurations of more or fewer components of the systems depicted are possible. Based on the disclosure and teachings provided herein, it should be appreciated that there are other ways and / or methods to implement the various examples.

[0458] Although specific examples have been described, various modifications, changes, alternative constructions, and equivalents are possible. Examples are not limited to operations within certain specific data processing environments, but can be freely operated within multiple data processing environments. In addition, although certain examples have been described using a series of specific transactions and steps, it is clear to those skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as sequential processing, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. Processing may have other steps not included in the figure. Various features and aspects of the above examples can be used alone or in combination.

[0459] In addition, although certain examples have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are also possible. Certain examples may be implemented using only hardware or only software or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0460] Where a device, system, component or module is described as being configured to perform certain operations or functions, such configuration may be achieved, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations, such as by executing computer instructions or code, or a processor or core is programmed to execute code or instructions stored on a non-transitory storage medium, or any combination thereof to perform the operations. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0461] Specific details are given in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description only provides illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. On the contrary, the previous description of the examples will provide those skilled in the art with an enabling description for implementing various examples. Various changes can be made to the functions and arrangements of the elements.

[0462] The description and drawings are therefore to be regarded as illustrative rather than restrictive. However, it will be apparent that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, although specific examples have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0463] In the foregoing description, aspects of the present disclosure are described with reference to their specific examples, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects disclosed above can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, examples can be used in any number of environments and applications beyond the environments and applications described herein. Thus, this specification and the accompanying drawings are considered to be illustrative rather than restrictive.

[0464] In the foregoing description, for the purpose of illustration, methods are described in a particular order. It should be appreciated that, in alternative examples, these methods may be performed in an order different from the described order. It should also be appreciated that the above methods may be performed by hardware components or may be implemented in a sequence of machine executable instructions, which may be used to enable a machine, such as a general or special processor or a logic circuit programmed with instructions to perform the method. These machine executable instructions may be stored on one or more machine-readable media, such as CD-ROM or other types of optical disks, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, these methods may be performed by a combination of hardware and software.

[0465] Where a component is described as being configured to perform certain operations, such configuration may be accomplished by, for example, designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0466] While illustrative examples of the present application have been described herein in detail, it should be understood that the inventive concept may be otherwise variously embodied and employed, and that the appended claims are intended to be interpreted to include such variations, except as limited by the prior art. < / value> < / value> < / binary-op> < / value> < / unary-op> < / value> < / value> < / value> < / count> < / value> < / question> < / question>

Claims

1. A computer-implemented method for processing natural language, comprising: generating an input string by concatenating a natural language utterance with a database schema representation of a database; Based on the input string, generating, by a first encoder, one or more embeddings of natural language utterances and database schema representations; encoding, by a second encoder, relationships between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings; generating, by a grammar-based decoder, an intermediate database query representation based on the encoded relation and the one or more embeddings; as well as A database query is generated in a database query language based on the intermediate database query representation and the interface specification.

2. The method of claim 1, further comprising: Schema linking relations linking elements in the database schema representation with words in the natural language utterance are received by a second encoder, wherein embeddings are further generated based on the schema linking relations.

3. The method of claim 2, wherein the schema link relationship includes metadata specifying synonyms of the word.

4. The method according to any one of claims 1 to 3, further comprising: A relational algebra grammar representing an intermediate database query representation as a tree is received by a grammar-based decoder, wherein the intermediate database query representation is further based on the relational algebra grammar.

5. The method according to any one of claims 1 to 4, wherein: The first encoder is a pre-trained language model (PLM); and The second encoder is a Relation-Aware Transformer (RAT).

6. The method according to any one of claims 1 to 5, further comprising: A database query is performed on the database to retrieve data responsive to the natural language utterance.

7. A method as claimed in any one of claims 1 to 5, wherein the database schema representation of the database comprises a link attribute that references an entry in a table without referring to the name of the table.

8. A system comprising: one or more processors; as well as One or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations including: generating an input string by concatenating a natural language utterance with a database schema representation of a database; Based on the input string, generating, as a first encoder, one or more embeddings of natural language utterances and database schema representations; encoding, as a second encoder, relationships between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings; as a grammar-based decoder, generating an intermediate database query representation based on the encoded relations and the one or more embeddings; and A database query is generated in a database query language based on the intermediate database query representation and the interface specification.

9. The system of claim 8, the operations further comprising: As a second encoder, pattern linking relations linking elements in the database pattern representation with words in the natural language utterance are received, wherein embeddings are further generated based on the pattern linking relations.

10. The system of claim 9, wherein the schema link relationship includes metadata specifying synonyms of the word.

11. The system of any one of claims 8 to 10, the operations further comprising: As a grammar-based decoder, a relational algebra grammar representing an intermediate database query representation as a tree is received, wherein the intermediate database query representation is further based on the relational algebra grammar.

12. The system of any one of claims 8 to 11, wherein: The first encoder is a pre-trained language model (PLM); and The second encoder is a Relation-Aware Transformer (RAT).

13. The system of any one of claims 8 to 12, the operations further comprising: A database query is performed on the database to retrieve data responsive to the natural language utterance.

14. The system of any of claims 8-13, wherein the database schema representation of the database includes a link attribute that references an entry in a table without referring to the name of the table.

15. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the system to perform operations comprising: generating an input string by concatenating a natural language utterance with a database schema representation of a database; Based on the input string, generating, as a first encoder, one or more embeddings of natural language utterances and database schema representations; encoding, as a second encoder, relationships between elements in the database schema representation and words in the natural language utterance based on the one or more embeddings; generating, as a grammar-based decoder, an intermediate database query representation based on the encoded relations and the one or more embeddings; as well as A database query is generated in a database query language based on the intermediate database query representation and the interface specification.

16. The one or more non-transitory computer readable media of claim 15, the operations further comprising: As a second encoder, pattern linking relations linking elements in the database pattern representation with words in the natural language utterance are received, wherein embeddings are further generated based on the pattern linking relations.

17. The one or more non-transitory computer-readable media of claim 16, wherein the schema link relationship includes metadata specifying synonyms of the word.

18. The one or more non-transitory computer readable media of any of claims 15-17, the operations further comprising: As a grammar-based decoder, a relational algebra grammar representing an intermediate database query representation as a tree is received, wherein the intermediate database query representation is further based on the relational algebra grammar.

19. One or more non-transitory computer readable media as recited in any one of claims 15-18, wherein: The first encoder is a pre-trained language model (PLM); and The second encoder is a Relation-Aware Transformer (RAT).

20. The one or more non-transitory computer readable media of any of claims 15-19, the operations further comprising: A database query is performed on the database to retrieve data responsive to the natural language utterance.

21. An apparatus comprising: Means for carrying out the operations of the method as claimed in any one of claims 1-7.

22. A computer program product comprising computer instructions which, when executed by a processor, implement the operations of the method of any one of claims 1-7.

Images