Techniques for transforming natural language conversations into visual representations

Abstract

Techniques for transforming natural language conversations into visual outputs are disclosed herein. In one aspect, a computer-implemented method includes generating an input string by joining a natural language utterance with a schema representation including a set of entities for visualizing an action; generating, by a first encoder of the machine learning model, one or more embedding of the input string; encoding, by a second encoder of the machine learning model, a relationship between an element in the schema representation and a word in the natural language utterance based on the one or more embedding; generating, by a syntax-based decoder of the machine learning model, an intermediate logical form representing at least the query, the one or more visualization actions, or a combination thereof based on the encoded relationship and the one or more inserts; and generating a command for the computing system based on the intermediate logical form.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Patent Application No. 18 / 616,801, filed March 26, 2024; U.S. Provisional Application No. 63 / 582,931, filed September 15, 2023; and U.S. Provisional Application No. 63 / 520,877, filed August 21, 2023, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure generally relates to transforming natural language into logical forms, and more specifically, to techniques for transforming natural language into visual representations. Background Technology

[0004] Artificial intelligence has numerous applications. For example, many users around the world use instant messaging or chat platforms for immediate responses. Organizations frequently use these platforms to engage in real-time conversations with customers (or end users). However, the cost for organizations to hire service personnel to communicate with customers or end users in real time can be very high. Chatbots or bots have begun to be developed to simulate conversations with end users, especially over the internet. End users can communicate with the bots through messaging applications they already have installed and use. Intelligent bots, often powered by artificial intelligence (AI), can communicate more intelligently and context-awarely in real-time conversations, thus allowing for more natural conversations between the bot and the end user, improving the conversation experience. Instead of end users learning a fixed set of keywords or commands to which the bot knows how to respond, intelligent bots can be able to understand the end user's intent and respond accordingly based on the user's utterances in natural language.

[0005] Artificial intelligence-based solutions, such as chatbots, can have both analog (human) and digital (machine) interfaces for interacting with humans and connecting to backend systems. It is advantageous to be able to extract and analyze the meaning of utterances (e.g., requests) when a person speaks in natural language, without relying on how the backend system will handle those utterances. As an example, a request might be for data that needs to be retrieved from a relational database, or it might require extracting the requested data from a knowledge graph. MRL is a general representation of natural language utterances, which chatbots can translate into any number of target machine-oriented languages. Therefore, chatbots can leverage MRL to communicate interchangeably with humans and various backend systems, including systems that communicate using Structured Query Language (SQL), Application Programming Interface (API), Representational State Transition (REST), Graph Query Language (GraphQL), Attribute Graph Query Language (PGQL), and more.

[0006] For example, SQL is the standard database management language used to interact with relational databases. SQL can be used to store, manipulate, retrieve, and / or otherwise manage data held in a relational database management system (RDBMS) and / or for stream processing in a relational data flow management system (RDSMS). SQL comprises statements or commands used to interact with relational databases. SQL statements or commands are categorized into Data Query Language (DQL) statements, Data Definition Language (DDL) statements, Data Control Language (DCL) statements, and Data Manipulation Language (DML) statements, among others. To interact with a relational database using SQL, a user must understand how the database is structured (e.g., knowledge of tables and the rows and columns within each table), SQL syntax, and how to relate the syntax to the database structure. Without this knowledge, users often find it difficult to interact with these relational databases using SQL.

[0007] Natural language interfaces (NLEs) to National LIDB database systems (such as RDBMS) (e.g., chatbots) provide users with a means to interact with these relational databases in an intuitive way without requiring knowledge of database management languages. For example, using natural language statements and queries (i.e., natural language queries), users can interact with these relational databases in pure language via NLIDBs. Recently, text-to-SQL systems have become popular, and deep learning methods for translating natural language queries into SQL queries have proven promising. Using semantic parsing, natural language statements, requests, and questions (i.e., sentences) can be transformed into machine-oriented languages ​​that can be executed by applications (e.g., chatbots, models, programs, machines, etc.). For example, semantic parsing can transform natural language sentences into general-purpose programming languages ​​such as Python, Java, and SQL. The processing for transforming natural language sentences into SQL queries typically includes rule-based, statistical, and / or deep learning-based systems. Rule-based systems typically use a fixed set of rules to translate natural language sentences into SQL queries. These rule-based systems are generally domain-specific and are therefore considered inflexible and do not generalize well to new use cases (e.g., across different domains). Statistical systems (such as slot filling) tag tokens (i.e., words or phrases) in an input natural language sentence based on their semantic roles within the sentence and use these tags to fill slots in an SQL query. Generally, these statistical systems have limitations on the types of sentences that can be parsed (e.g., the sentence must be able to be represented as a parse tree). Deep learning-based systems (such as sequence-to-sequence models) involve training deep learning models that directly translate natural language sentences into machine-oriented language and have been shown to generalize across tasks, domains, and datasets. However, such deep learning systems require large amounts of supervised training data, and obtaining labeled data (e.g., natural language query-SQL statement pairings) is challenging. Therefore, deep learning-based translation of natural language sentences into machine-oriented language inevitably requires large amounts of labeled training data. Summary of the Invention

[0008] Machine learning techniques are provided for transforming natural language into logical forms (e.g., methods, systems, storage of code or instructions executable by one or more processors, and more specifically, techniques are provided for transforming natural language into visual representations.

[0009] In various embodiments, this document provides a computer-implemented method comprising: generating an input string by concatenating (a) a natural language utterance including a query, one or more visualizations, or a combination thereof, with (b) a schema representation including a set of entities for the visualizations; generating one or more embeddings of the input string by a first encoder of a semparizer machine learning model based on the input string; encoding relationships between elements in the schema representation and words in the natural language utterance by a second encoder of the semparizer machine learning model based on the one or more embeddings; generating an intermediate logical form by a syntax-based decoder of the semparizer machine learning model based on the encoded relationships and the one or more embeddings, the intermediate logical form representing at least the query, the one or more visualizations, or a combination thereof; and generating a command for a computing system based on the intermediate logical form, wherein the command includes the query, the one or more visualizations, or a combination thereof in the form of a system programming language.

[0010] In some embodiments, the computer-implemented method further includes: executing a command on a computing system, wherein the execution causes the computing system to obtain results for a query; generating, updating, or modifying one or more visualizations, or a combination thereof, based on the one or more visualization actions; and transmitting the results, the one or more visualizations, or the combination thereof, to a user providing natural language discourse, wherein the natural language discourse includes the one or more visualization actions, and the one or more visualization actions are represented as one or more entities in the set of entities within a pattern representation.

[0011] In some embodiments, the computer-implemented method further includes: providing an input string to a filter machine learning model before generating one or more embeddings of natural language utterances and pattern representations; generating a score by the filter machine learning model based on the input string, the score indicating whether the natural language utterances are translatable into commands for the computing system and should be routed to a semparizer machine learning model; comparing the score to a threshold; and providing the input string to the semparizer machine learning model in response to determining that the score exceeds the threshold.

[0012] In some embodiments, the computer-implemented method further includes: converting an intermediate logic form into an enhanced intermediate logic form, the enhanced intermediate logic form including a query in the form of a Meaning Representation Language (MRL) that can be translated into a system programming language, the one or more visual actions, or a combination thereof; and generating a command for the computing system based on the enhanced intermediate logic form, wherein the command includes a query in the form of a system programming language, the one or more visual actions, or a combination thereof.

[0013] In some embodiments, the computer-implemented method further includes providing a schema-linking relation to a second encoder of the semparizer machine learning model, the schema-linking relation linking entities in the schema representation to visual action words in natural language discourse, wherein the embeddings are further generated based on the schema-linking relation, and wherein the schema-linking relation includes metadata specifying synonyms for the visual action words.

[0014] In some embodiments, the schema representation also includes a schema specification for a given schema, and a link attribute that connects one or more entities from the set of entities used to visualize actions to one or more fact entities associated with the given schema.

[0015] In some embodiments, the computer-implemented method further includes providing a relational algebra grammar to a grammar-based decoder of the semparizer machine learning model, the relational algebra grammar representing intermediate logical forms as trees, wherein the intermediate logical forms are generated by the grammar-based decoder based on encoded relations, the one or more embeddings, and the relational algebra grammar.

[0016] In various embodiments, a system is provided that includes one or more processors and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0017] In various embodiments, one or more non-transitory computer-readable media are provided for storing instructions that, when executed by one or more processors, cause a system to perform some or all of the operations and / or methods disclosed herein.

[0018] In various embodiments, an apparatus is provided that includes some or all of the components for implementing the operations and / or methods disclosed herein.

[0019] In various embodiments, a computer program product is provided that includes computer instructions that, when executed by a processor, implement some or all of the operations and / or methods disclosed herein.

[0020] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts represent only a small fraction of the many implementations and contexts. Attached Figure Description

[0021] Figure 1This is a simplified block diagram of a distributed environment in conjunction with exemplary embodiments.

[0022] Figure 2 This is a simplified block diagram of a computing system for implementing a master robot according to certain embodiments.

[0023] Figure 3 This is a simplified block diagram of a computing system for implementing a skill-based robot according to certain embodiments.

[0024] Figure 4 This is a simplified block diagram illustrating an NL2LF or C2MRL architecture according to various embodiments, and an overview of the processing for generating queries for a backend interface starting from natural language discourse.

[0025] Figure 5 This is a simplified block diagram of the C2OMRL architecture according to various embodiments.

[0026] Figure 6A It is a diagram of a computational workflow, according to various embodiments, for converting representations of natural language (NL) discourse into structured query language (SQL) for database execution.

[0027] Figure 6B It is a diagram of a modifiable, visual computational workflow for converting NL discourse into a dataset, according to various embodiments.

[0028] Figure 6C It is a diagram of a single computing architecture environment for providing different types of output based on categories of NL discourse input, according to various embodiments.

[0029] Figure 7 These are exemplary designs, according to various embodiments, for representing display entities that represent different actions associated with Viz-creation, Viz-Incremental, and Viz-Manipulation queries.

[0030] Figure 8 It is a visual representation of the schema specification enhanced with additional Display, Display_Element hub entities and associated attributes for the tabular OAC Superstore dataset, according to various embodiments.

[0031] Figure 9 It is a visual representation of a complete model pipeline flow for transforming natural language queries into a visual representation, according to various embodiments.

[0032] Figure 10 This illustrates how computing architectures, according to various embodiments, support analytical queries in a visual representation.

[0033] Figure 11This is a visual representation illustrating how computing architectures according to various embodiments support Viz-Creation queries.

[0034] Figure 12 This is a visual representation illustrating how computing architectures according to various embodiments support Viz-Incremental and Viz-Manipulation queries.

[0035] Figure 13 A flowchart illustrating techniques for transforming natural language conversation into a visual representation, according to various embodiments, is depicted.

[0036] Figure 14 A simplified diagram of a distributed system for implementing various embodiments is depicted.

[0037] Figure 15 It is a simplified block diagram of one or more components of a system environment according to various embodiments, through which the services provided by one or more components of the embodiment system can be provided as cloud services.

[0038] Figure 16 The illustration shows an example computer system that can be used to implement various embodiments. Detailed Implementation

[0039] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent, however, that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting. The word “exemplary” as used herein means “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or superior to other embodiments or designs.

[0040] introduce

[0041] In recent years, the amount of data supporting various industries and their systems has grown exponentially. Most business information is stored in relational databases for storing, processing, and retrieving data. Databases support information systems across multiple industries, such as consumer technology (e.g., orders, cancellations, refunds), supply chain (e.g., raw materials, inventory, suppliers), healthcare (e.g., medical records), finance (e.g., financial business metrics), customer support, search engines, and more. Modern data-driven companies need to track the real-time status of their operations to quickly understand and diagnose any problems, trends, or anomalies in the data and take immediate corrective action. This work is typically performed manually by analysts who write complex queries using query languages ​​such as SQL, PGQL, logical database queries, and API query languages ​​such as GraphQL and REST. Writing such queries can be used to derive insightful information from data stored across multiple tables. These results are often processed in the form of charts or graphs to enable users to quickly visualize the results and facilitate data-driven decision-making.

[0042] While common database queries (e.g., SQL queries) are often predefined and integrated into commercial products, any new or subsequent queries still require manual coding by analysts. This static interaction between database queries and the consumption of corresponding results demands time-consuming manual intervention and results in slow feedback cycles. Allowing non-technical users (e.g., business leaders, doctors, or other users of the data) to interact directly with analytical tables via natural language (NL) queries—which abstract away the underlying query language (e.g., SQL) code—is far more efficient. Defining database queries requires a deep understanding of database schemas and query language syntax, and can quickly become daunting for beginners and non-technical stakeholders. Efforts to bridge this communication gap have led to the development of a new type of processing known as NLIDB. This natural search capability has become more popular in recent years as companies develop deep learning methods for Natural Language to Logical Forms (NL2LF), such as Natural Language to SQL (NL2SQL). Logical Forms can refer to semantic representation languages ​​and / or machine-oriented languages. NL2SQL seeks to transform natural language problems into SQL, allowing individuals to run unstructured queries on databases. The transformed SQL can also improve the response of digital assistants, such as chatbots, when answers can be found in different databases or tables.

[0043] In the specific context of translating NL into Meaning Representation Language (MRL), NL utterances (e.g., user questions) are typically fed to an NL2LF model, which transforms the NL utterances into a logical form, such as MRL with intermediate representations. The NL2LF model is a machine learning model trained to generate intermediate representations from NL utterances. Translation processing can then be used to translate the intermediate representations into one or more desired query formats, such as SQL or PGQL. Subsequently, the utterances in the desired query format can be executed on a backend system (such as a database) that supports the desired query format to obtain query-related data and formulate a response to the NL utterances (e.g., an answer to a user question) for user review.

[0044] In some instances, the intermediate representation uses a language called Oracle Meaning Representation Language (OMRL) [also known as or described as Oracle Meaning Representation Query Language (OMRQL)], and the Conversation-to-Oracle Meaning Representation Language (C2OMRL) system performs the conversion of natural language utterances into logical forms. The C2OMRL system is powered by a deep learning model configured to translate NL utterances (or conversations within the Oracle Digital Assistant (ODA) platform) into logical forms in an intermediate database query language such as OMRL. These logical forms can then be used to generate queries in a specific database query language (e.g., SQL), which can then be executed to query an existing database.

[0045] However, conventional NL2LF models generally only support text-based use cases (e.g., finding the number of universities with more than 2000 enrollments for each membership type). A very attractive next step for NL2LF models is to support visualization-based use cases (e.g., creating data visualizations, such as graphs and charts showing the number of universities with more than 2000 enrollments for each membership type, as well as animations). Existing backend queries, such as ODA's SQL Dialogs, are powered by the C2OMRL backend, translating the discourse into tables that can be executed on the database and returning the results. Beyond this, a primary goal of the NL2LF platform will also be the ability to handle discourse requests for visualization of a given dataset or any modifications to the current visualization. This will be advantageous in the sense that users can request visualizations of a given dataset or schema supported by a given platform (e.g., Oracle Analytics Cloud (OAC)) to gain better data analysis and insights.

[0046] To address these and other challenges, the developed method described in this paper leverages an architecture capable of supporting standard text use cases (e.g., NL2SQL), visualization use cases (e.g., NL2OAC), and combinations of both text and visualization use cases. More specifically, this architecture utilizes NL2LF technology via entities representing different actions of creation-type visualization discourses (described herein as Viz-Creation), modification-type visualization discourses (described herein as Viz-Manipulation), and incremental-type visualization discourses (described herein as Viz-Incremental), while retaining the existing advantages of text-based backend services (such as SQL Dialogs) supporting use cases. Creation-type visualization discourses are natural language discourses and their corresponding logical forms requesting the system (e.g., OAC) to generate or create visualizations (e.g., charts or graphs) for a given dataset. Manipulation-type visualization discourses are natural language discourses and their corresponding logical forms requesting the system (e.g., OAC) to modify attributes of previously visualized data (e.g., modifying chart_type, legend, etc.). Incremental type visualization discourse is a natural language discourse and its corresponding logical form that requests a system (e.g., OAC) to refine the previous meaning.

[0047] Table 1 below summarizes the visualization types of statements that are typically asked on the OAC platform.

[0048] Table 1:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

[0061] While these examples are specific to the OAC platform, and in many instances the AI-based techniques described herein are described within the context of the C2OMRL system, it should be understood that similar examples will be seen in many platforms that support visualization, and the AI-based techniques described herein are applicable to other NL2LF systems and / or models configured to generate other intermediate representation formats.

[0062] Advantageously, using a single pre-built NL2LF model to handle both use cases can reduce maintenance and processing resources and workload, but this comes with the challenge of ensuring no or very few regressions to existing text-based use cases. The method described herein also addresses this and other challenges. For example, in an exemplary embodiment, a computer-implemented method is provided, comprising: generating an input string by concatenating (a) a natural language utterance including a query, one or more visualization actions, or a combination thereof, with (b) a pattern representation including a set of entities for visualization actions; generating one or more embeddings of the input string based on the input string by a first encoder of a semparizer machine learning model; encoding relationships between elements in the pattern representation and words in the natural language utterance based on the one or more embeddings by a second encoder of the semparizer machine learning model; generating an intermediate logical form based on the encoded relationships and the one or more embeddings by a syntax-based decoder of the semparizer machine learning model, the intermediate logical form representing at least the query, the one or more visualization actions, or a combination thereof; and generating a command for a computing system based on the intermediate logical form, wherein the command includes the query, the one or more visualization actions, or a combination thereof in the form of a system programming language. The pattern representation of a set of entities used for visualizing actions helps overcome the challenge of ensuring no or very few regressions during the training of a semparizer machine learning model, enabling the model to support standard text use cases (e.g., NL2SQL), visualization use cases (e.g., NL2OAC), and combinations of both text and visualization use cases during the inference phase.

[0063] As used herein, when an action is “based on” something, it means that the action is at least partially based on at least a portion of that thing. As used herein, the terms “similarly,” “substantially,” “approximately,” and “about” are defined as generally, but not necessarily entirely, as understood by one of ordinary skill in the art (and include being entirely specified). In any disclosed embodiment, the terms “similarly,” “substantially,” “approximately,” or “about” may be replaced with “within a specified [percentage],” where percentages include 0.1%, 1, 5, and 10 percent.

[0064] Robots and Analysis Systems

[0065] A bot (also known as a skill, chatbot, chatterbox, or talking bot) is a computer program that can engage in conversation with an end user. Bots typically respond to natural language messages (such as questions or comments) via messaging applications that use natural language messaging. Businesses can use one or more bot systems to communicate with end users through messaging applications. The messaging application (which may be referred to as a channel) can be a preferred messaging application that the end user has already installed and is familiar with. Therefore, the end user does not need to download and install a new application to chat with the bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), mobile and web application extensions that extend native or hybrid / responsive mobile or web applications with chat capabilities, or voice-based input (such as devices or applications with interfaces that allow interaction using Siri, Cortana, Google Voice, or other speech input).

[0066] In some examples, a robot system may be associated with a Uniform Resource Identifier (URI). A URI is a string of characters that identifies the robot system. A URI can be used as a webhook for one or more messaging applications. A URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The robot system may be designed to receive messages from a messaging application (e.g., a Hypertext Transfer Protocol (HTTP) POST message). An HTTP POST message can be directed from the messaging application to the URI. In some embodiments, the message may differ from an HTTP POST message. For example, the robot system may receive messages from a Short Message Service (SMS). While the discussion herein may refer to communication where the robot system is receiving messages, it should be understood that the message can be an HTTP POST message, an SMS message, or any other type of communication between the two systems.

[0067] End users can interact with robot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), much like human-to-human interactions. In some cases, the interaction may involve the end user saying "hello" to the robot, which responds with "hi" and asks how it can help. In other cases, the interaction may be transactional with, for example, a banking robot, such as transferring money from one account to another; informational with, for example, an HR robot, such as checking vacation balances; or with, for example, a retail robot, such as discussing returning purchased goods or seeking technical support.

[0068] In some embodiments, a robot system can intelligently handle end-user interactions without requiring interaction with the robot system's administrator or developers. For example, an end-user can send one or more messages to the robot system to achieve a desired goal. Messages may include content such as text, emojis, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., a Representation State Transition (REST) ​​call for an enterprise service with appropriate parameters) and generate a natural language response. The robot system may also prompt the end-user for additional input parameters or request additional information. In some embodiments, the robot system may also initiate communication with the end-user rather than passively responding to end-user utterances. This document describes various techniques for identifying explicit calls to a robot system and determining the input of the called robot system. In some embodiments, explicit call analysis is performed by the master robot based on the call name detected in the utterance. In response to the detection of the call name, the utterance can be refined to be input to the skill robot associated with the call name.

[0069] Conversations with a chatbot can follow a specific conversation flow that includes multiple states. This flow can be defined based on input to determine what will happen next. In some embodiments, the chatbot system can be implemented using a state machine that includes user-defined states (e.g., end-user intentions) and actions to be taken in or from one state to another. Conversations can take different paths based on end-user input, which affects the chatbot's decisions regarding the flow. For example, in each state, based on the end-user's input or utterances, the chatbot can determine the end-user's intention to determine the appropriate next action to take. As used herein and in the context of utterances, the term "intention" refers to the intention of the user providing the utterance. For example, a user might intend to have a conversation with the chatbot to order pizza, so the user's intention can be expressed through the utterance "order pizza". User intentions can be directed toward a specific task that the user wants the chatbot to perform on their behalf. Therefore, utterances can be expressed as questions, commands, requests, etc., that reflect the user's intention. Intentions can include goals that the end-user wants to achieve.

[0070] In the context of chatbot configuration, the term "intent" is used herein to refer to configuration information that maps a user's utterances to specific tasks / actions or categories of tasks / actions that the chatbot can perform. To distinguish the intent of a utterance (i.e., the user's intent) from the intent of the chatbot, the latter is sometimes referred to herein as "bot intent." A bot intent can include one or more utterances associated with the intent. For example, the intent to order pizza can be conveyed through various permutations of utterances expressing the desire to order pizza. These associated utterances can be used to train the chatbot's intent classifier so that the classifier can subsequently determine whether an input utterance from the user matches the intent to order pizza. A bot intent can be associated with one or more conversational flows used to initiate a conversation with the user and to be in a certain state. For example, the first message of an order pizza intent might be the question "What kind of pizza do you want?". In addition to associated utterances, a bot intent can also contain named entities related to the intent. For example, an order pizza intent can include variables or parameters used to perform the task of ordering pizza (e.g., toppings 1, toppings 2, pizza type, pizza size, number of pizzas, etc.). The values ​​of the entities are typically obtained through conversation with the user.

[0071] Figure 1 This is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. Environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistant or chatbot systems. DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as... Figure 1 As shown, user 104 of a specific enterprise can use DABP 102 to create and deploy digital assistants 106 for that specific enterprise's users. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform can be used by multiple enterprises to create digital assistants. As another example, restaurant owners (e.g., pizzerias) can use DABP 102 to create and deploy digital assistants that enable their customers to order food (e.g., order pizza).

[0072] For the purposes of this disclosure, a "digital assistant" is a tool that assists its user in performing various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital tool implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. Digital assistants can be implemented or realized in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Therefore, for the purposes of this disclosure, the terms "digital assistant" and "chatbot system" are interchangeable.

[0073] Digital assistants (such as digital assistant 106 built using DABP 102) can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and receive a response 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and the response 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the task requested by the user and respond appropriately to the user.

[0074] User input 110 is generally in the form of natural language and is referred to as a speech. User speech 110 can be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user speech 110 can be in the form of audio input or speech, such as when a user speaks or utters something provided as input to the digital assistant 106. Speech is typically in the language spoken by user 108. For example, speech can be English or some other language. When speech is in speech form, the speech input is converted into a text form of speech in that particular language, and then the text speech is processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert speech or audio input into text speech, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion can be performed by the digital assistant 106 itself.

[0075] Discourse (which can be textual or verbal discourse) can be fragments, sentences, multiple sentences, one or more words, one or more questions, combinations of the types mentioned above, etc. Digital Assistant 106 is configured to apply Natural Language Understanding (NLU) technology to discourse to understand the meaning of user input. As part of NLU processing for discourse, Digital Assistant 106 is configured to perform processing to understand the meaning of discourse, which involves identifying one or more intentions and one or more entities corresponding to the discourse. After understanding the meaning of the discourse, Digital Assistant 106 can perform one or more actions or operations in response to the understood meaning or intention. For the purposes of this disclosure, it is assumed that the discourse is either textual discourse directly provided by user 108 of Digital Assistant 106 or the result of a conversion of input verbal discourse into textual form. However, this is not intended to be limiting or restrictive in any way.

[0076] For example, user 108 can request to order pizza by providing a statement such as "I want to order pizza." Upon receiving such a statement, digital assistant 106 is configured to understand the meaning of the statement and take an appropriate action. The appropriate action may involve, for example, responding to the user with questions requesting input about the type of pizza the user wants to order, the size of the pizza, any toppings, etc. The response 106 provided by the digital assistant can also be in natural language and is typically the same language as the input statement. As part of generating these responses, digital assistant 106 can perform natural language generation (NLG). For the user ordering pizza, through the conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the information needed for ordering pizza, and then order the pizza at the end of the conversation. Digital assistant 106 can end the conversation by outputting a message to the user indicating that the pizza has been ordered.

[0077] At the conceptual level, the digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a sequence or pipeline of processing steps, including, for example, understanding the meaning of the input utterances, (sometimes referred to as Natural Language Understanding (NLU)) determining the action to be performed in response to the utterances, performing the action where appropriate, generating a response to be output to the user in response to the user's utterances, and outputting the response to the user, etc. NLU processing may include parsing the received input utterances to understand the structure and meaning of the utterances, refining and reconstructing the utterances to build a more understandable form (e.g., logical form) or structure for the utterances. Generating a response may include using NLG techniques.

[0078] NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related processes, such as sentence parsing (e.g., word segmentation, lemmatization, identifying part-of-speech tags for sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing individual clauses, parsing references, performing chunking, etc.). In some embodiments, NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of NLU processing. For example, the syntax and structure of the input utterance sentence may be identified by processing it using a parser, part-of-speech tagger, and / or named entity recognizer. In one implementation, for English, sentence structure and syntax are analyzed using parsers, part-of-speech taggers, and named entity recognizers, such as those provided by the Stanford Natural Language Processing (NLP) Group. These are provided as part of the Stanford CoreNLP toolkit.

[0079] While the various examples provided in this disclosure illustrate utterances in the English language, this is merely illustrative. In some embodiments, the digital assistant 106 is also capable of handling utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing on different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This allows for flexibility and scalability of NLU processing for each language, including allowing for different processing sequences. Language packs may be provided for various languages, whereby the language packs may register a list of subsystems capable of providing services from the NLU core server.

[0080] Digital assistants (such as Figure 1 The digital assistant 106 depicted can be available or accessible to its user 108 through various channels, such as, but not limited to, certain applications, social media platforms, various messaging services and applications, and other applications or channels. A single digital assistant may have several channels configured for it, enabling it to operate and be accessed simultaneously on different services.

[0081] Digital assistant or chatbot systems typically contain or are associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill bots) configured to interact with users and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing gadgets, etc. For example, for Figure 1In the embodiments depicted, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the terms "skill" and "skills" are used synonymously with the terms "skillbot" and "skillbots," respectively.

[0082] Each skill associated with the digital assistant assists the user in completing tasks through conversation with the user, where the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may be in the form of text or audio messages given to the user and / or using simple user interface elements (e.g., a selection list) presented to the user for selection.

[0083] Skills or skill bots can be associated with or added to digital assistants in various ways. In some cases, skill bots can be developed by businesses and then added to digital assistants using DABP 102. In other cases, skill bots can be developed and created using DABP 102 and then added to digital assistants created using DABP 102. In still other cases, DABP 102 provides an online digital store (called the “Skill Store”) that offers a variety of skills for a range of tasks. Skills offered through the Skill Store can also expose various cloud services. To add a skill to a digital assistant created using DABP 102, a DABP 102 user can access the Skill Store via DABP 102, select the desired skill, and instruct the selected skill to be added to the digital assistant created using DABP 102. Skills from the Skill Store can be added to the digital assistant as is or in a modified form (for example, a DABP 102 user can select and clone a specific skill bot offered by the Skill Store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0084] Various architectures can be used to implement digital assistant or chatbot systems. For example, in some embodiments, a master robot / sub (or slave) robot paradigm or architecture can be used to implement a digital assistant created and deployed using DABP 102. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more sub-robots acting as skill-based robots. For example, in... Figure 1 In the illustrated embodiment, the digital assistant 106 includes a main robot 114 and skill robots 116-1, 116-2, etc., which are sub-robots of the main robot 114. In some embodiments, the digital assistant 106 itself is considered to act as the main robot.

[0085] Digital assistants implemented using a master-slave robot architecture enable users to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user interacts with the digital assistant, the master robot receives the user input. The master robot then processes the input to determine its meaning. It then determines whether the task requested by the user can be handled by the master robot itself; otherwise, it selects the appropriate skill robot to handle the request and routes the conversation to that selected skill robot. This allows users to interact with the digital assistant through a shared, single interface while still providing the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for a business, the master robot can interface with skill robots with specific functionalities, such as a CRM robot for performing functions related to Customer Relationship Management (CRM), an ERP robot for performing functions related to Enterprise Resource Planning (ERP), an HCM robot for performing functions related to Human Capital Management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access it through a shared master robot interface, while multiple skill robots are available behind the scenes to handle user requests.

[0086] In some embodiments, in a master robot / sub-robot infrastructure, the master robot is configured to know a list of available skill robots. The master robot has access to metadata identifying each available skill robot, and for each skill robot, the capabilities of that skill robot include the tasks it can perform. Upon receiving a user request in the form of a utterance, the master robot is configured to identify or predict from among the multiple available skill robots that a particular skill robot can best serve or handle the user request. The master robot then routes the utterance (or a portion thereof) to that particular skill robot for further handling. Thus, control flows from the master robot to that skill robot. The master robot may support multiple input and output channels. In some embodiments, routing may be performed using processes performed by one or more available skill robots. For example, as discussed below, a skill robot may be trained to infer the intent of a utterance and determine whether the inferred intent matches an intent configured on the skill robot. Therefore, routing performed by the master robot may involve the skill robot conveying an indication to the master robot regarding whether it is configured with an intent suitable for handling the utterance.

[0087] Although Figure 1The embodiments illustrated in the text depict a digital assistant 106 including a master robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide digital assistant functionality. These systems and subsystems may be implemented solely in software (e.g., code or instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in a combination of software and hardware.

[0088] The DABP 102 provides the infrastructure, services, and features that enable DABP 102 users to create digital assistants that include one or more skill bots associated with the digital assistant. In some cases, skill bots can be created by cloning existing skill bots, such as those offered by a skill store. As previously indicated, the DABP 102 provides a skill store or skill catalog that supplies multiple skill bots for performing various tasks. DABP 102 users can clone skill bots from the skill store. The cloned skill bot can be modified or customized as needed. In other cases, DABP 102 users create skill bots from scratch using the tools and services provided by the DABP 102. As previously indicated, the DABP 102 skill store or skill catalog can supply multiple skill bots for performing various tasks.

[0089] In some embodiments, at a high level, creating or customizing a skill-based robot involves the following steps:

[0090] (1) Configure settings for new skill robots

[0091] (2) Configure one or more intentions for the skill robot

[0092] (3) Configure one or more entities for one or more intents.

[0093] (4) Training skills robots

[0094] (5) Create dialogue flow for skill robots

[0095] (6) Add custom components to the skill robot as needed.

[0096] (7) Testing and deploying skills robots

[0097] The following is a brief description of each of the above steps.

[0098] (1) Configuring settings for new skill robots – Various settings can be configured for skill robots. For example, skill robot designers can specify one or more invocation names for the skill robot being created. These invocation names can then be used by the digital assistant user to explicitly invoke the skill robot. For example, the user can enter the invocation name in their speech to explicitly invoke the corresponding skill robot.

[0099] (2) Configuring one or more intents and associated example utterances for the skill robot—The skill robot designer assigns one or more intents (also called robot intents) to the skill robot being created. The skill robot is then trained based on these assigned intents. These intents represent the category or class of utterances the skill robot is trained to infer from inputs. Upon receiving an utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes an appropriate action to respond to the utterance based on the intent inferred for that utterance. In some cases, the intents of the skill robot represent tasks that the skill robot can perform for a digital assistant user. Each intent is given an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents assigned to the skill robot could include “CheckBalance”, “TransferMoney”, “DepositCheck”, etc.

[0100] For each intent defined for a skill bot, the skill bot designer can also provide one or more example statements that represent and illustrate that intent. These example statements are intended to represent the statements a user might input into the skill bot in response to that intent. For example, for the CheckBalance intent, example statements could include “What is my savings account balance?”, “How much money do I have in my checking account?”, “How much money do I have in my account?”, etc. Thus, various permutations of typical user statements can be specified as example statements for the intent.

[0101] Intents and their associated example utterances are used as training data to train the skill robot. Various training techniques can be used. As a result of this training, a predictive model is generated, which is configured to take utterances as input and output the intent inferred by the predictive model for those utterances. In some cases, the input utterances are provided to an intent analysis engine, which is configured to use the trained model to predict or infer the intent of the input utterances. The skill robot can then take one or more actions based on the inferred intent.

[0102] (3) Configuring Entities for One or More Intents of a Skill Robot – In some cases, additional context may be required for the skill robot to respond correctly to user requests. For example, there may be situations where user-inputted utterances are resolved to the same intent in the skill robot. For instance, in the example above, the utterances “What is my savings account balance?” and “How much money do I have in my checking account?” both resolve to the same CheckBalance intent, but these utterances are different requests asking different things. To clarify such requests, one or more entities are added to the intent. Using the banking skill robot example, an entity called AccountType, which defines values ​​called “checks” and “savings,” enables the skill robot to resolve user requests and respond appropriately. In the example above, although the utterances are resolved to the same intent, the values ​​associated with the AccountType entity are different for the two utterances. This allows the skill robot to perform potentially different actions for the two utterances, even though they resolve to the same intent. One or more entities can be specified for certain intents configured for the skill robot. Thus, entities are used to add context to the intent itself. Entities help to describe the intent more completely and enable the skill robot to fulfill the user request.

[0103] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by the skill robot designer. Built-in entities are general entities that can be used in a variety of robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, repetition period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for banking skills, the AccountType entity can be defined by the skill robot designer, enabling various banking transactions by examining keywords entered by the user (such as checks, savings, and credit cards).

[0104] (4) Training the Skill Robot—The skill robot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select an intent associated with the received user input. As indicated above, the skill robot must be trained for this purpose. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example utterances associated with those intents (collectively, training data), such that the skill robot can parse user input utterances as one of the intents configured for the robot. In some embodiments, the skill robot uses a predictive model that is trained on the training data and allows the skill robot to discern what the user said (or, in some cases, what they are trying to say). DABP 102 provides a variety of different training techniques that can be used by skill robot designers to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion of the training data (e.g., 80%) is used to train the skill robot model, while another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as the trained skill robot) can be used to process and respond to user utterances. In some situations, a user's utterance may be a question that only requests a single answer without requiring further conversation. To handle such situations, a Q&A (question-answering) intent can be defined for the skill bot. This allows the skill bot to output a response to the user's request without updating the dialogue definition. Q&A intents are created in a similar manner to regular intents. The dialogue flow used for Q&A intents can differ from the dialogue flow used for regular intents.

[0105] (5) Creating a Dialogue Flow for the Skilled Robot—The dialogue flow specified for the skilled robot describes how the skill robot reacts when different intentions for the skill robot are parsed in response to received user input. The dialogue flow defines the actions or behaviors the skill robot will take (e.g., how the skill robot responds to user utterances, how the skill robot prompts the user for input, how the skill robot returns data). The dialogue flow is like a flowchart followed by the skill robot. Skill robot designers use languages ​​such as markdown languages ​​to specify the dialogue flow. In some embodiments, a version of YAML called OBotML can be used to specify the dialogue flow of the skill robot. The dialogue flow definition for the skill robot acts as a model of the conversation itself, that is, a model that allows skill robot designers to orchestrate the interactions between the skill robot and the users served by the skill robot.

[0106] In some embodiments, the dialogue flow definition for a skill-based robot comprises three parts:

[0107] (a) Contextual part

[0108] (b) Default transition section

[0109] (c) State section

[0110] The context section—skill bot designers can define variables used in the conversation flow within the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, user variables that enable the skill bot to recognize and retain user preferences, etc.

[0111] Default Transition Section – Transitions for the skill bot can be defined either in the dialog flow state section or in the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when no suitable transition is defined within the state or when the conditions required to trigger a state transition are not met. The default transition section can be used to define routes that allow the skill bot to gracefully handle unexpected user actions.

[0112] The state component—the dialogue flow and its associated operations—is defined as a series of transitional states that manage the logic within the dialogue flow. Each state node within the dialogue flow definition is named after a component that provides the required functionality at that point in the dialogue. Therefore, states are built around components. A state contains component-specific characteristics and defines the transitions to other states triggered after the execution of that component.

[0113] State sections can be used to handle special scenarios. For example, sometimes you might want to offer users the option to temporarily leave their primary skill to do something in a secondary skill within the digital assistant. For instance, if a user is talking to a shopping skill (e.g., the user has made some purchase choices), they might want to jump to a banking skill (e.g., the user might want to ensure they have enough money for the purchase) and then return to the shopping skill to complete their order. To address this, the action in the primary skill can be configured to initiate an interaction with a different secondary skill within the same digital assistant and then return to the original flow.

[0114] (6) Adding Custom Components to the Skill Robot—As described above, the state names specified in the skill robot's dialogue flow provide components corresponding to the functionality required for that state. Components enable the skill robot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured components for performing a wide range of functions. The skill robot designer can select one or more of these pre-configured components and associate them with states in the skill robot's dialogue flow. The skill robot designer can also use the tools provided by DABP 102 to create custom or new components and associate custom components with one or more states in the skill robot's dialogue flow.

[0115] (7) Testing and Deploying Skilled Robots – DABP 102 provides several features that enable skilled robot designers to test the skilled robots they are developing. The skilled robots can then be deployed and incorporated into digital assistants.

[0116] While the above description describes how to create skill robots, similar techniques can be used to create digital assistants (or master robots). At the master robot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master robot) can handle without invoking the skill robots associated with that digital assistant. Examples of system intents defined for the master robot include: (1) Exit: applicable when a user signals that they wish to exit the current conversation or context within the digital assistant; (2) Help: applicable when a user requests help or guidance; and (3) Unresolved intent: applicable to user input that does not well match the exit and help intents. The digital assistant also stores information about one or more skill robots associated with it. This information enables the master robot to select a specific skill robot to handle the utterance.

[0117] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The routing model determines whether the conversation corresponding to the user's input utterance should be routed to a specific skill for handling, handled by the digital assistant or master robot itself based on built-in system intent, or handled as a different state within the current conversation flow.

[0118] In some embodiments, as part of this process, the digital assistant determines whether the utterance of user input explicitly uses its invocation name to identify a skill bot. If an invocation name is present in the user input, it is considered an explicit invocation of the skill bot corresponding to that invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. If no specific or explicit invocation is found, in some embodiments, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intent and skill bot associated with the digital assistant. The score calculated for the skill bot or system intent represents the likelihood that the user input represents a task configured to be performed by that skill bot or represents the system intent. Any system intent or skill bot with an associated calculated confidence score exceeding a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill bots are identified as candidates, the intents associated with these candidate skills are evaluated (according to an intent model for each skill) and a confidence score is determined for each intent. Generally, any intent with a confidence score exceeding a threshold (e.g., 70%) is considered a candidate intent. If a specific skill bot is selected, the user's utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the main bot itself based on the selected system intent.

[0119] Figure 2 This is a simplified block diagram of a master robot (MB) system 200 according to certain embodiments. The MB system 200 can be implemented using only software, only hardware, or a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, a multi-intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data repository 250. Figure 2 The MB system 200 depicted is merely an example of component arrangement in the main robot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more than Figure 2 The more or fewer systems or components shown may be combined into two or more subsystems, or may have different subsystem configurations or arrangements.

[0120] Preprocessing subsystem 210 receives the utterance “A” 202 from the user and processes it via language detector 212 and language parser 214. As indicated above, the utterance can be provided in various forms, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation marks. For example, if the utterance 202 is provided in audio form, then preprocessing subsystem 210 can use a speech-to-text converter (not shown) to convert the audio into text, which inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text.

[0121] The language detector 212 detects the language of discourse 202 based on the text of discourse 202. The way discourse 202 is processed depends on the language, as each language has its own grammar and semantics. Differences between languages ​​must be considered when analyzing the syntax and structure of discourse.

[0122] Language parser 214 parses utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. Language parser 214 can also segment the linguistic units of utterance 202 (e.g., convert each word into a separate lexical) and perform lemmatization on the words. Lemmatization is the lemmatization of a group of words as represented in a dictionary (e.g., "run" is the lemmatization of run, runs, ran, running, etc.). Other types of preprocessing that language parser 214 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". Language parser 214 can also identify relationships between words in utterance 202. For example, in some embodiments, language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a specific noun) is a direct object, which part of the utterance is a preposition, etc. The processing result performed by the language parser 214 forms the extracted information 205, which, together with the utterance 202 itself, is provided as input to the MIS 220.

[0123] As indicated above, utterance 202 may include more than one sentence. To detect multiple intents and explicit calls, utterance 202 can be treated as a single unit even if it comprises multiple sentences. However, in some embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210 to identify a single sentence among multiple sentences for multi-intent analysis and explicit call analysis. Generally, the results produced by MIS 220 and EIS 230 are substantially the same regardless of whether utterance 202 is processed at the level of a single sentence or as a single unit containing multiple sentences.

[0124] MIS 220 determines whether utterance 202 represents multiple intentions. While MIS 220 can detect the presence of multiple intentions in utterance 202, the processing performed by MIS 220 does not involve determining whether the intention of utterance 202 matches any intentions already configured for the robot. Instead, the processing to determine whether the intention of utterance 202 matches a robot intention can be performed by the intention classifier 242 of MB system 200 or the intention classifier of the skill robot (e.g., as...). Figure 3 (As shown in the embodiment). The processing performed by MIS 220 assumes the existence of a bot that can handle utterance 202 (e.g., a skill-specific bot or the main bot itself). Therefore, the processing performed by MIS 220 does not need to know which bots are in the chatbot system (e.g., the identities of skill-specific bots registered with the main bot) or what intents have been configured for a particular bot.

[0125] To determine that utterance 202 contains multiple intentions, MIS 220 applies one or more rules from a set of rules 252 in data repository 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include sentence patterns indicating the presence of multiple intentions. For example, a sentence pattern might contain a coordinating conjunction that connects two parts of a sentence (e.g., a conjunction), where each part corresponds to a separate intention. If utterance 202 matches a sentence pattern, then it can be inferred that utterance 202 represents multiple intentions. It should be noted that utterances with multiple intentions do not necessarily have different intentions (e.g., intentions for different robots or intentions for different intentions within the same robot). Instead, utterances may have separate instances of the same intention (e.g., “order pizza using payment account X, then order pizza using payment account Y”).

[0126] As part of determining that utterance 202 represents multiple intentions, MIS 220 also determines which parts of utterance 202 are associated with each intention. For each intention expressed in a utterance containing multiple intentions, MIS 220 constructs a new utterance to process separately in place of the original utterance, such as utterance “B” 206 and utterance “C” 208, as... Figure 2As depicted in [the document]. Therefore, the original utterance 202 can be divided into two or more separate utterances, one utterance at a time. MIS 220 uses the extracted information 205 and / or by analyzing the utterance 202 itself to determine which of the two or more utterances should be processed first. For example, MIS 220 may determine that utterance 202 contains marker words indicating that a particular intention should be processed first. The newly formed utterance corresponding to that particular intention (e.g., one of utterances 206 or 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance ends (or is temporarily suspended), the next highest priority utterance (e.g., another of utterances 206 or 208) can then be sent to EIS 230 for processing.

[0127] EIS 230 determines whether a received utterance (e.g., utterance 206 or utterance 208) contains a call name for a skill bot. In some embodiments, each skill bot in the chatbot system is assigned a unique call name that distinguishes it from other skill bots in the chatbot system. The list of call names may be maintained as part of skill bot information 254 in data repository 250. When a utterance contains words that match a call name, it is considered an explicit call. If no bot is explicitly called, the utterance received by EIS 230 is considered a non-explicit call utterance 234 and is fed into the master bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to handle the utterance. In some cases, intent classifier 242 will determine that the master bot should handle non-explicitly called utterances. In other cases, intent classifier 242 will determine to route the utterance to the skill bot for handling.

[0128] The explicit invocation functionality provided by EIS 230 offers several advantages. It reduces the amount of processing the master robot must perform. For example, when explicit invocation is present, the master robot may not need to perform any intent classification analysis (e.g., using intent classifier 242), or it may need to perform a simplified intent classification analysis to select the skill robot. Therefore, explicit invocation analysis enables the selection of a specific skill robot without resorting to intent classification analysis.

[0129] Furthermore, there may be situations where multiple skill robots have overlapping functions. This can happen, for example, if two skill robots handle overlapping or very similar intentions. In such cases, the master robot may struggle to identify which of the multiple skill robots to select based solely on intention classification analysis. In these scenarios, explicit invocation can eliminate ambiguity regarding the specific skill robot to use.

[0130] In addition to determining whether a utterance is an explicit call, EIS 230 is also responsible for determining whether any part of the utterance should be used as input to the skill robot that made the explicit call. Specifically, EIS 230 can determine whether a portion of the utterance is irrelevant to the call. EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. EIS 230 may send the irrelevant portion of the utterance to the called skill robot instead of sending the entire utterance received by EIS 230. In some cases, the input to the called skill robot is formed only by removing any call-related parts of the utterance. For example, “I want to order pizza using Pizza Bot” can be shortened to “I want to order pizza” because “using Pizza Bot” is relevant to the call to the Pizza Bot but irrelevant to any processing the Pizza Bot is to perform. In some cases, EIS 230 can reformat the portion to be sent to the called robot, for example, to form a complete sentence. Therefore, EIS 230 not only determines the existence of an explicit call but also determines what to send to the skill robot when an explicit call exists. In some cases, there may be no text to be input into the invoked robot. For example, if the utterance is "Pizza Bot," then EIS 230 can determine that a pizza robot is being invoked, but there is no text to be processed by the pizza robot. In this scenario, EIS 230 can indicate to the skill robot invoker 240 that there is no content to send.

[0131] Skill robot invoker 240 invokes skill robots in various ways. For example, skill robot invoker 240 may invoke a robot in response to receiving instruction 235, which indicates that a particular skill robot has been selected as a result of explicit invocation. Instruction 235 may be sent by EIS 230 along with input for the explicitly invoked skill robot. In this scenario, skill robot invoker 240 transfers control of the conversation to the explicitly invoked skill robot. The explicitly invoked skill robot will determine the appropriate response to the input from EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, where the initial state of the new conversation depends on the input sent from EIS 230.

[0132] Another way the skill robot invoker 240 can invoke a skill robot is through implicit invocation using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that a particular utterance represents a task configured to be performed by the skill robot. The intent classifier 242 is trained on different categories, one category per skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task the new skill robot can perform. The parameters generated as a result of this training (e.g., a set of values ​​for the parameters of a machine learning model) can be stored as part of the skill robot information 254.

[0133] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described further in detail herein. Training the machine learning model may involve taking input from at least a subset of example utterances associated with various skill-based robots to generate an inference as the output of the machine learning model about which robot is the correct robot to dispose of any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as basic fact information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the discrepancy between the generated inference and the basic fact information.

[0134] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot. This confidence score indicates the likelihood that the skill robot can handle a utterance (e.g., a non-explicit invocation utterance 234 received from EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skill robot invoker 240 will invoke the robot associated with that particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is an identification of a system intent or a particular skill robot. In some embodiments, in addition to satisfying the threshold confidence score value, the confidence score must also exceed the next highest confidence score certain win margin. Imposing such a condition when the confidence scores of multiple skill robots each exceed the threshold confidence score value will enable routing to a particular skill robot.

[0135] After identifying the robot based on an evaluation of the confidence score, the skill robot invoker 240 will hand over processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is the skill robot. Furthermore, the skill robot invoker 240 will determine what to provide as input 247 to the identified robot. As indicated above, in the case of explicit invoking, input 247 may be based on a part of the utterance unrelated to the invoking, or input 247 may be nothing (e.g., an empty string). In the case of implicit invoking, input 247 may be the entire utterance.

[0136] Data repository 250 includes one or more computing devices that store data used by various subsystems of the master bot system 200. As explained above, data repository 250 includes rules 252 and skill bot information 254. Rule 252 includes, for example, rules for determining, by MIS 220, when a utterance represents multiple intents and how to split the utterance representing multiple intents. Rule 252 also includes rules for determining, by EIS 230, which parts of the utterance that explicitly invokes a skill bot will be sent to the skill bot. Skill bot information 254 includes the invocation names of skill bots in the chatbot system, for example, a list of invocation names of all skill bots registered with a particular master bot. Skill bot information 254 may also include information used by intent classifier 242 to determine the confidence score of each skill bot in the chatbot system, for example, parameters of a machine learning model.

[0137] Figure 3 This is a simplified block diagram of a skill robot system 300 according to certain embodiments. The skill robot system 300 is a computing system that can be implemented using only software, only hardware, or a combination of hardware and software. In some embodiments, such as Figure 1 In the embodiments illustrated, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0138] The skill robot system 300 includes a MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to... Figure 2 The MIS 220 in the data repository 350 provides similar functionality, including operability to determine, using rule 352 in the data repository 350: (1) whether the utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by the MIS 310 for detecting multiple intents and for splitting utterances are the same as those applied by the MIS 220. The MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to Figure 1The extracted information 205 can be generated using the language parser 214 or the local language parser of the skill robot system 300.

[0139] The intent classifier 320 can be combined with the above. Figure 2 The intent classifier 242 discussed in the embodiments is trained in a manner similar to and further described in detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a robot with a specific skill, using at least a subset of example utterances associated with that robot with that specific skill as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with the training utterance.

[0140] Message 302 can be received directly from the user or supplied through the main robot. When message 302 is supplied through the main robot, for example, in... Figure 2 In the embodiments illustrated, when the result of processing by MIS 220 and EIS 230 is supplied, MIS 310 can be bypassed to avoid repeating the processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a conversation that occurs after routing to a skill robot, then MIS 310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If so, then MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, for example, utterance “D” 306 and utterance “E” 308. If utterance 302 does not represent multiple intents, then MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0141] Intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skill robot system 300. As explained above, the skill robot can be configured with one or more intents, each intent including at least one example utterance associated with the intent and used to train the classifier. Figure 2In this embodiment, the intent classifier 242 of the main robot system 200 is trained to determine the confidence scores of individual skill robots and the system intent. Similarly, the intent classifier 320 can be trained to determine the confidence score of each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level, thus providing finer granularity. The intent classifier 320 can access intent information 354. The intent information 354 includes a list of utterances for each intent associated with the skill robot system 300, which represent and explain the meaning of the intent and are generally associated with a task that the intent can perform. The intent information 354 may also include parameters generated during training on the list of utterances.

[0142] The conversation manager 330 receives an indication 322 of a specific intent as the output of the intent classifier 320, which is identified by the intent classifier 320 as the best match for the utterance input to the intent classifier 320. In some cases, the intent classifier 320 may not be able to determine any match. For example, if the utterance is directed to a system intent or an intent of a different skill robot, the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill robot system 300 may submit the utterance to the master robot for processing, for example, routing it to a different skill robot. However, if the intent classifier 320 successfully identifies an intent within a skill robot, the conversation manager 330 will initiate a conversation with the user.

[0143] The conversation initiated by the conversation manager 330 is a specific conversation specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 can be implemented using a state machine configured to execute a dialogue flow for the identified intent. The state machine may include a default starting state (e.g., when the intent is invoked without any additional input) and one or more additional states, each associated with an action (e.g., performing a purchase transaction) and / or a dialogue (e.g., a question, a response) to be presented to the user. Thus, the conversation manager 330 can determine an action / dialogue 335 upon receiving an instruction 322 for the identified intent, and can determine additional actions or dialogues in response to subsequent utterances received during the conversation.

[0144] Data repository 350 includes one or more computing devices that store data used by various subsystems of skill robot system 300. For example... Figure 3 As illustrated, the data repository 350 includes rules 352 and intent information 354. In some embodiments, the data repository 350 may be integrated into the data repository of the main robot or digital assistant, for example, Figure 2The data is stored in repository 250.

[0145] Convert natural language statements into logical queries

[0146] In some aspects, the model is trained to generate a logical form of an intermediate query representation based on natural language utterances. This intermediate query representation can then be translated into a suitable backend system language, such as SQL, PGQL, OAC API, etc. In some instances, the intermediate representation uses a language called Oracle Meaning Representation Language (OMRL), and the conversation-to-Oracle Meaning Representation Language (C2OMRL) system performs the conversion from natural language utterances to logical forms. The C2OMRL system is powered by a deep learning model configured to convert natural language (NL) utterances (or conversations within the Oracle Digital Assistant platform) into logical forms of an intermediate query language, such as Oracle Meaning Representation Language (OMRL). The logical forms can be used to generate queries or commands in a specific backend system language, which can then be executed to query or control the backend system, such as an existing database or OAC. This deep learning model (called a "C2OMRL semantic parser" or "C2OMRL model") is trained on thousands of examples (natural language to logical forms).

[0147] Figure 4 This is a block diagram 400 illustrating the C2OMRL architecture and an overview of the processing used to generate queries or commands to backend interface 406, which begins with NL utterance 408 (e.g., received via human interface 402). For example, human interface 402 could be a chatbot system that receives spoken speech and translates it into text utterance as described above, or a system where the user types a request in natural language, or other suitable interfaces. NL utterance 408 could be in the form of part of a conversation (e.g., “Hello, could you tell me how many orders we need to place tomorrow?” or “Search all employees whose names begin with 'S' and who reside in California.”).

[0148] NL utterances 408 are fed to an NL2LF model 410, which transforms the NL utterances 408 into intermediate representations 412 (e.g., MRL or OMRL). The NL2LF model 410 is a machine learning model trained to generate intermediate representations 412 from NL utterances 408. The NL2LF model 410 includes multiple layers and algorithms for generating intermediate representations 412 from NL utterances 408, as described in more detail herein. In some instances, such as... Figure 4The NL2LF model 410 described herein is a C2OMRL model used to convert conversational utterances into OMRL 412. The NL2LF model 410 can be described interchangeably with C2OMRL, but it should be understood that the techniques described herein can be applied to models configured to generate other intermediate representation 412 formats. An intermediate representation 412 is a logical representation of utterances configured to be translatable into a specific system query language. In some examples, the intermediate representation 412 is OMRL, an intermediate database query language with specialized schemas and interface specifications. The intermediate representation 412 can be described interchangeably with OMRL herein, but it should be understood that the techniques described herein can be applied to other intermediate representation 412 formats.

[0149] The intermediate representation 412 can then be translated into one or more desired backend system languages ​​(such as SQL 416, PGQL 420, or OAC API 422) using one or more system language translation processes (such as OMRL2SQL 414, OMRL2PGQL 418, or OMRL2OAC 424). The translated queries or commands (e.g., SQL 416, PGQL 420, or OAC API 422) represent the concepts present in the intermediate representation 412 in a manner consistent with the requirements of the applicable system language.

[0150] An overview of systems for translating natural language into meaning-representing languages.

[0151] Figure 5 A C2OMRL system 500 powered by a machine learning model is shown to enable the translation of NL discourse (e.g., about...) Figures 1-3 The utterances within the digital assistant platform are converted into LF statements (such as OMRL queries or commands), which can then be executed to query existing systems (such as relational databases) or analytics platforms (such as OAC). This machine learning model (referred to herein as the "C2OMRL Semantic Parser" or "C2OMRL Model" or simply the "Parser") is trained on hundreds to thousands of annotated example pairs (natural language and logical form pairs) to translate NL utterances into LF statements. As shown in the figure, example 505 (a connection between a natural language utterance and one or more patterns (e.g., a database schema including a sequence of table names and column names)) is input into C2OMRL model 510. Example 505 is first processed by encoder component 515, which captures representations of the natural language utterances and patterns based on context. Decoder 520 then receives the encoded input and predicts logical form 525 (e.g., OMRL, which is a SQL-like query) based on the captured representations of the natural language utterances and patterns.

[0152] In the C2OMRL model 510, the encoder component 515 comprises two encoders: (1) a first encoder, which is a pre-trained language model (PLM) 530; and (2) a second encoder, which is a relation-aware transformer (RAT) 535. The PLM 530 is used to embed natural language utterances and patterns because it captures representations of natural language utterances and patterns based on context. In some cases, a transformer-based PLM (called Decoding Enhanced BERT with Untangled Attention (DeBERTa)) is used as the PLM 530. (See He et al., DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (2021), the entire contents of which are incorporated herein by reference for all purposes). Transformer-based PLMs use self-supervised learning to learn general language representations from large amounts of text data and transfer this knowledge to downstream tasks. These models provide good background knowledge for downstream tasks, which avoids training downstream models from scratch. RAT 535 encodes the relationships between entities in a pattern and words in a natural language discourse (these relationships are called "pattern links").

[0153] Decoder 520 is based on a bottom-up generative process (i.e., a bottom-up generative process that generates trees from left to right), where the final generated output is an OMRL tree (i.e., a tree-based structure representing the complete OMRL logical form), which can be mapped to the final OMRL logical form 525. The bottom-up generative process is implemented using beam search, a graph-partitioning heuristic search algorithm that explores the most promising nodes by expanding a finite set. The beam search works in steps (e.g., ~10 steps), also known as “beam levels”. In each step (e.g., “step i”), the beam search algorithm generates F possible subtrees for the input sequence that can be obtained by expanding the current subtree (from step “i-1”), and then selects the top K subtrees (called the beam width) to retain using the conditional probability associated with each subtree. The conditional probability is referred to in this paper as the “unprocessed beam score”, so the top K intermediate results (which will be considered in the next generation step) are the K results with the highest unprocessed beam scores. Additional information on bottom-up generative processing can be found in “Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressive Bottom-up Semantic Parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 311–324, online. Association for Computational Linguistics,” the entire contents of which are incorporated herein by reference for all purposes. The final decoder 520 output is the subtree with the highest unprocessed bundle score at point N in the final step.

[0154] For example, in the first step (bundle level 1), the encoded input utterance and pattern are fed into decoder 520, which applies a softmax function to all lexics in the vocabulary or grammar (e.g., relational algebra grammar) to find the best alternative to the first subtree (e.g., the first lexicographical symbol or node of the tree). To generate the number F of possible subtrees (called the frontier), decoder 520 makes a prediction of the conditional probability of each lexicographical symbol in the vocabulary or grammar in the sequence (possible values ​​of yi+1, conditioned on previous lexics y1,…,yi and the context variable c generated by the encoder to represent the input sequence). The vocabulary or grammar is obtained from a corpus of words or terms that include the target logical form (e.g., OMRL). In some instances, the corpus also includes rules for words or terms targeting the target logical form. The rules define how to use words or terms to create appropriate phrases or operations in the target logical form (e.g., combinations of terms that work together to form an appropriate OMRL query). The bundle search algorithm then selects the top K subtrees with the highest conditional probabilities or unprocessed bundle scores as the most likely possible choices for that time step. In this example, it is assumed that the width of the first K subtrees or bundles is 2, and the subtrees with the highest conditional probability P(y1 | c) in the first step are sub-tree_1 and sub-tree_12. The first K results can be optional and / or optimizable hyperparameters. Sub-tree_1 and sub-tree_12, along with their corresponding conditional probabilities or unprocessed bundle scores, are stored in memory.

[0155] In the second step (bundle level 2), the two selected trees (sub-tree_1 and sub-tree_12) from the first step are input to decoder 520, and decoder 520 applies a softmax function to all lexicons in the vocabulary or grammar to find two optimal alternatives to the second subtree (e.g., the first and second lexicons or nodes of the tree). In doing so, the bundle search algorithm uses conditional probabilities to determine the most likely combination of first and second lexicons or nodes to form a pair or second subtree. In other words, for all... The bundle search algorithm calculates P(sub-tree_1,y2|c) = P(sub-tree_1|c)P(y2|sub-tree_1,c) and P(sub-tree_12,y2|c) = P(sub-tree_12|c)P(y2|sub-tree_12,c), and selects the two largest values ​​among these, such as P(sub-tree_22|c) and P(sub-tree_37|c). Sub-tree_22 and sub-tree_37, along with their corresponding conditional probabilities or unprocessed bundle scores, are stored in memory.

[0156] In the third step (bundle level 3), the two selected trees (sub-tree_22 and sub-tree_37) from the second step are input to decoder 520, which applies a softmax function to all tokens in the vocabulary or grammar to find two optimal alternatives to the third subtree (e.g., the first, second, and third tokens or nodes of the tree). In doing so, the bundle search algorithm uses conditional probabilities to determine the combinations of the first, second, and third tokens or nodes most likely to form the string or the third subtree. In other words, for all... The bundle search algorithm calculates P(sub-tree_22,y3|c) = P(sub-tree_22|c)P(y3|sub-tree_22,c) and P(sub-tree_37,y3|c) = P(sub-tree_37|c)P(y3|sub-tree_37,c), and selects the top K subtrees. The top K subtrees and their corresponding conditional probabilities or raw bundle scores are stored in memory. This process continues until N bundle levels are completed (this can be optimized or selected hyperparameters). The final model output is the subtree with the highest conditional probability or raw bundle score at the final step N (bundle levels N). The tokens or nodes of this final subtree can then be mapped to the final logical form, such as OMRL logical form statement 525.

[0157] The MRL logical form statement 525 (e.g., an OMRL tree with the highest unprocessed bundle score at the final step N) can then be input into a language converter 540 (such as OMRL2SQL or OMRL2OAC) to translate the semantic representation language into a system language query or command, such as SQL, API, REST, GraphQL, PGQL, OAC API, etc. The system language query or command can then be used to query or execute operations on system 545 (e.g., a relational database or analytics platform) and obtain output 550 as the result of the query or command.

[0158] Technology for transforming natural language speech into visual representations

[0159] As discussed, conventional NL2LF models generally only support text-based use cases. To overcome this challenge, along with other challenges, this paper describes an architecture (regarding...). Figures 6A-6C (Detailed description provided) to transform natural language conversation into a visual representation. For example... Figure 6AAs shown, in text-based use cases such as the SQL Dialogs skill on the Oracle Digital Assistant (ODA) platform, OMRL is translated into SQL and executed against the database. The first subsystem 605 of the computational architecture (e.g., C2OMRL) employs a machine learning stack of multiple deep learning-based models 610 to transform NL utterances 615 into logical form 620. An example of the first subsystem 605 is described above regarding... Figure 4 and Figure 5 The description is provided and is further detailed in co-pending U.S. Application No. 18 / 209,2444, filed June 14, 2023, which is incorporated herein by reference in its entirety for all purposes. Logical form 620 (e.g., Oracle Meaning Representation Language (OMRL)) is an intermediate representation. Using a second subsystem 625 (e.g., an SQL backend and an Oracle DB), logical form 620 is mapped to one or more backend programming languages ​​630, such as SQL (e.g., an OMRL2SQL backend). This one or more backend programming languages ​​630 are then executed by execution engine 635 against one or more systems (or components of the systems, such as a database 640 from an Oracle DB) to retrieve text output 645. Text output 645 can be used by the ODA platform to formulate a response to dialogue 615.

[0160] like Figure 6B As shown, in visualization-based use cases (such as charts / visualizations supported by the OAC platform), OMRL will be translated into the OAC API and executed against OAC. Similar to... Figure 6A The computational architecture shown employs a machine learning stack of multiple deep learning-based models 610 to transform NL utterances 615 into logical forms 620. However, in this instance, utterance 615 can be recognized as a request for visualization output (e.g., charts, graphs, tables, line graphs, etc.). This request may be referred to as a visualization query. Logical form 620 (e.g., Oracle Meaning Representation Language (OMRL)) is an intermediate representation. Using a second subsystem 625 (e.g., OAC API and OAC), logical form 620 is mapped to one or more backend programming languages ​​630, such as OAC (e.g., OMRL2OAC backend). This one or more backend programming languages ​​630 are then executed against one or more systems (or components of the systems, such as analytics programs from OAC) via application programming interface 650 (OAC API) to retrieve visualization output 655. Visualization output 655 can be used by the ODA platform to formulate a response to utterance 615. In some instances, the user can make additional statements that cause the first subsystem 605 and the second subsystem 625 to modify the visualization output.

[0161] like Figure 6C As shown, to integrate the platform's text-based capabilities with analytics platforms such as OAC to support visualization use cases, a neuro-semantic parsing machine learning model (semparizer) is augmented / trained to understand patterns related to both text-based and visualization specifications in a way that a single pre-built model can support both text-based and visualization use cases. Furthermore, the routing model (e.g., out-of-domain, out-of-scope, and confusion span detection (OOCS) model) is modified so that visualization-related discourse can be smoothly routed to the semparizer. Similar to... Figure 6A and Figure 6B The architecture shown, a single computational architecture for handling both text and visualization use cases, includes a first subsystem 605 that employs a machine learning stack of multiple deep learning-based models 610 to transform NL utterances 615 into logical forms 620. The machine learning stack of multiple deep learning-based models 610 includes two models: an OOCS model 660 and a semparizer (semantic parser) model 665. The OOCS model 660 filters out utterances that are irrelevant to the semparizer model 665 or that the semparizer model 665 may have difficulty parsing properly. Advantageously, OOCS can minimize output that will disappoint the user.

[0162] Discourse 615 may include requests for resolving computational architecture. Resolving these requests may involve presenting output to the user. Each output can be categorized as text output or visual output. Dialogue input and discourse input associated with text output can be referred to as text discourse input. Dialogue input and discourse input associated with visual output can be referred to as visual discourse input or viz input. OOCS model 660 can recognize each of these two types of input and seamlessly route text discourse input or viz input to semparizer model 665.

[0163] The OOCS model 660 first embeds the input using an initial / transformer-based encoder. This encoded input is then consumed by a translatability head that predicts a translatability score. More specifically, the translatability head is trained for a binary classification task, aiming to predict probabilities rather than class labels. The model objective is to minimize the binary cross-entropy between the predicted probabilities and class labels; for example, maximizing the likelihood of observing translatable and untranslatable class labels given the predicted probabilities. The translatability head takes the result from the initial encoder as input and outputs a translatability score, which is a probability score indicating the likelihood that an NL utterance can be successfully translated into a query for the backend system using the Semparizer model 665. The translatability head can also output a label. This label indicates whether the utterance is translatable and should be handled by the Semparizer model 665. The label is associated with the translatability score. For example, if the translatability score exceeds a threshold, the utterance is labeled as translatable, and if the translatability score does not exceed the threshold, the utterance is labeled as untranslatable. In some instances, the encoded input is also consumed by a confusion span head, which predicts which part of the NL utterance is confusing or unclear and outputs the confusion span. Both the translatable score and the confusion span can be outputs of the OOCS model 660.

[0164] The semparizer model 665 transforms the text utterance input and viz input into a logical form 620 based on a first backend system schema 670 (e.g., a schema for relational databases from OracleDB) and a second backend system schema 675 (e.g., a schema for datasets from OAC). The logical form 620 (e.g., Oracle Meaning Representation Language (OMRL)) is an intermediate representation. The first subsystem 605 and the semparizer model 665 analyze the utterance input and schema, and determine which category (or categories) of output is requested. When the first subsystem 605 and the semparizer model 665 determine that the utterance input is text input and the user is requesting text output, the first subsystem 605 can route the request to a second subsystem 625 (e.g., an OMRL2SQL backend), which can translate the request into first programming code 680, such as SQL code, and execute the first programming code against one or more systems (or components of the systems, such as a database from OracleDB) to retrieve text output 645, which can be used to formulate a response to utterance 615. When the first subsystem 605 and the semparizer model 665 determine that the utterance input is a viz input and the user is requesting visualization output, the first subsystem 605 can translate the request into second programming code 685, such as OAC code, and execute the second programming code against one or more systems (or components of the systems, such as an analysis program from OAC) to retrieve the visualization output. The visualization output can be used to formulate a response to the utterance 615. The visualization output can take the form of a graph or chart, which the user can modify using additional utterance input. Advantageously, the semparizer model 665 can process the viz input without performing regressions on the effective processing of the text input.

[0165] Table 2 is a summary of four categories of visual discourse inputs that can be converted into visual outputs according to various embodiments. These four categories can include analytical queries, Viz-Creation queries, Viz-Incremental queries, and Viz-Manipulation queries. The visual discourse inputs for each category are described in the following paragraphs. About Figures 4-6C The described models (such as the semparizer model) can recognize, understand, and process visual utterance inputs from all four categories.

[0166] Table 2:

[0167]

[0168]

[0169] Analytical queries can be described as "pure select" statements. Visualization output can be associated with a "pure select" statement, but the visualization type may not be specified. A "pure select" statement for an analytical query can resemble text discourse input, but the wording can differ. For analytical queries, a computing platform (e.g., OAC) can determine the optimal chart type for rendering the associated visualization output. Processing analytical queries may involve grouping or filtering data at different granularities (e.g., year, quarter, month, week, millisecond, etc.) using date and time functions.

[0170] Examples of statements associated with analytical queries could include: “Show sales by customer segment”, “show profits and sales by month for 2019”, “show me profit and sales by shipment month and product category”, “show me profit by month of order date and product category”, “return the average sales by product category and shipping mode”, etc.

[0171] The second category of visualization input is called a Viz-Creation query. An example of a Viz-Creation query is "show me a map of sales by month in Europe". Viz-Creation queries can be similar to analytical queries but can include additional visualization clauses that request a specific type of visualization output. In this example, "a map" can be recognized as a visualization clause. With Viz-Creation queries, users can request specific visualizations for a dataset (e.g., the OAC dataset) with a specified type of chart. Users can also specify filters for the data to be plotted in the chart. In some examples, users can use Viz-Creation queries to request a set of charts. For example, a user can request a set of bar charts to display all measurements in the dataset across all dimensions, or to show a specific metric in the dataset across all dimensions.

[0172] A Viz-Creation query can request a detailed list of chart categories. The list of categories can include tables, bar charts, pie charts, and scatter plots. Each category can include subcategories. For example, the bar chart category can include many subcategories such as stacked bar charts, horizontal bar charts, horizontally stacked bar charts, 100% stacked bar charts, horizontally 100% stacked bar charts, etc. Models (such as Semparizer) can recognize categories or subcategories in a Viz-Creation query.

[0173] Each category or subcategory can have multiple synonyms. For example, the subcategory "horizontal stacked bar chart" can have several synonyms, including "horizontal stacked bar graph," "horizontal stacked bar plot," "stacked horizontal bar chart," "stacked horizontal bar graph," "stacked horizontal bar plot," "horizontal stacked rowchart," and "stacked horizontal row chart." The model can identify the chart category or subcategory based on synonyms identified in the Viz-Creation query. Table 3 shows a list of visualization output chart subcategories associated with the Viz-Creation query and example synonyms. Additional examples of chart subcategories can include overlay plots, correlation matrix plots, pictograms, tree diagrams, chord diagrams, tile plots, map plots, language narrative plots, list plots, etc. In some examples, the model can identify the appropriate chart category or subcategory even if a syntactic error, user error, or inaccurate voice2text occurs in the Viz-Creation query. For example, the model can recognize "stack horizontal bar plot" and "stakcedhorizontal bar chart" as requests for horizontally stacked bar charts.

[0174] Table 3:

[0175]

[0176]

[0177]

[0178]

[0179] Examples of Viz-Creation queries can include: "Create a stack bar chart of sales and profit by product category", "Using a pie chart, show the quantity ordered by transportation mode", "Show me a table of sales and profit by month and product category where country is Spain and Portugal and product name is '...'", "Show me a table of sales and profit by month and product category where profit is more than 10,000", "Create a horizontal bar chart of profit by region in Europe", "Create a set of horizontal bar charts that look at profit across all attributes", and "Create a set of bar charts for all measures by product category". "Category" (creates a set of bar charts for all measures by product category), "Create a line graph of all measures by product category and shipping mode", "Create a map of profit by country", etc.

[0180] The third category of visual discourse input is called a Viz-Incremental query. A Viz-Incremental query can include, or is similar to, "follow-up actions" that can change the nature of the analytical query on which the visualization is based. A Viz-Incremental query can include filters, modifications, alterations, etc., to be applied to the data included in the output visualization. Examples can be used to more easily explain the second category. Examples of Viz-Incremental queries can include "Add a filter for sales more than or equal to 2.5 million", "Add a filter to keep only rows that have profit more than 10 million", "Do not include the datapoints which have sales less than 10 million", "Add a filter to keep only Australia and New Zealand", "Do not filter by Portugal", "Show profit as well", "Remove sales and profit", and "Replace sales and profit".

[0181] The fourth category of visualization input is called a Viz-Manipulation query. A Viz-Manipulation query can involve modifications to the output visualization. A Viz-Manipulation query can be part of a meta-talk about the created output visualization. This output visualization can include many parts or components. While sales figures are measured values ​​included in the database, sales figures can also refer to a column as part of a visualization like "show me a map of sales by month in Europe". Filters can be promoted to first-class objects in a Viz-Manipulation query, such as in statements like "remove the Austria filter". Discourse within a Viz-Manipulation query can modify different parts of the visualization. The discourse in a Viz-Manipulation query may not be an analysis query or a change to a previous analysis query, but rather a modifier of the visualization itself.

[0182] Viz-Manipulation queries can include a wide range of modifiers, such as those for the type of chart, the fonts and colors used in the chart, the position / size of visualization elements, number and currency formats, axis ticks, and more. Examples of Viz-Manipulation queries could include "Change this to a line graph," "Convert this to a scatter plot," "Change the titlefont to Times New Roman with size 18," "Abbreviate the number up to 2 decimal places," "Make the titles blue and bold," "Color the background red and set transparency to 60%," "Disable the thousand separator," "Adjust the x-axis to logarithmic scale," "Move the legend to the right," and "Reset the background," among others.

[0183] A key aspect of the computing architecture used to support visualization output is to leverage NL2LF technology via entities (also referred to as visualization or display entities in this paper) used to represent the different actions of Viz-Creation, Viz-Incermental, and Viz-Manipulation queries, while retaining the existing advantages of text-based use cases such as SQL. Figure 7 Exemplary designs using entities targeting a given schema (SQL Dialogs) or dataset (OAC) are shown. According to various embodiments, entities represent different actions associated with Viz-Creation creation, Viz-Incermental, and Viz-Manipulation queries. Design guidelines include:

[0184] • Entities in a computing architecture environment can represent existing technologies that can perform actions involved in processing visual utterance input, while retaining existing strengths that support use cases associated with text utterance input.

[0185] • Entities (both visual and textual) can help make all visual utterance inputs intuitive for models (such as semparizer models).

[0186] • It can map the intermediate representation of the model to multiple backend programming languages.

[0187] • Queries can be intuitive to enable the collection of training data (e.g., with internal contractors and external suppliers).

[0188] In the context of both text-based use cases (such as SQLDialogs) and visualization-based use cases (such as OAC), the OOCS / semparizer model takes NL utterances and one or more schemas (entity names, attribute names, lists of links between entities, attribute types, other metadata, etc.) as input to generate a logical form of the NL utterances in a semantic representation language and / or a machine-oriented language (e.g., MRL logical form). A schema defines how data is organized within a system (such as a relational database); this includes logical constraints such as table names, fields, data types, and relationships between various entities. For example, a relational database may consist of one or more tables, where each of these tables includes one or more columns, and each of these columns includes one or more values. Each table and column in a relational database can be named using a unique identifier, and each unique identifier may include one or more words. In some instances, one or more columns in a relational database may be used as primary keys, where each value in the one or more columns used as primary keys is unique to each other. In some instances, one or more columns in a relational database may be used as foreign keys, which are used to link a table including these one or more columns to another table in the relational database. Schema information can include one or more data structures for storing unique identifiers for one or more tables, unique identifiers for one or more columns, and values ​​for each relational database. In some instances, the data structures storing schema information for a relational database can store a directed graph representing unique identifiers and values.

[0189] The metadata associated with a pattern includes additional information about the pattern, including synonyms for different words. For example, "car" is a synonym for "automobile." Using this additional information in the pattern, name-based pattern linking can be used to identify elements in the pattern representation based on recognizing synonyms and identifying exact matches. In other words, pattern-link relationships include metadata specifying synonyms for words (e.g., minimum, min, least, lowest, etc.). Pattern linking involves identifying values ​​used for identification in systems such as relational databases. For example, in the utterance "show invoices for customer Nike," the pattern link specifies that "Nike" is the value used to call "vendor" in the database. The metadata used for pattern linking is provided to the model, and this value linking helps the model make correct predictions. An example of this metadata used for pattern linking is content-based pattern linking (CBSL) matching offsets, which specify which part of the utterance matches the value. CBSL technology is described in more detail in U.S. Patent Application 18 / 065,387, filed December 13, 2022, entitled "Transforming Natural Language To Structured Query LanguageBased On Scalable Search And Content-Based Schema Linking," the entire contents of which are incorporated herein by reference for all purposes. In some instances, Name-Based Schema Linking (NBSL) is applied. NBSL is used to generate matches between lexical symbols in a natural language discourse and elements in a schema representation. NBSL matches entities such as table names and column names with words in the input discourse, which can be based on an exact or partial match between the main name and its synonyms and elements in the schema representation.

[0190] Existing schema specifications (e.g., schemas for relational databases) can be modified using an entity-based approach to adapt the schema associated with visualization by adding a set of additional entities for visualization. More specifically, actions associated with visualization queries can be represented as additional entities in the existing schema specification. Additional entities can include hub entities (e.g., the Display hub entity and the Display_Element hub entity) and subsidiary entities (e.g., action entities). Additional entities can be associated with attributes required to support visualization queries.

[0191] like Figure 7As shown, the exemplary design may include two hub entities. The Display hub entity is the central entity used to support Viz-Creation queries. The Display hub entity includes a main action entity called “visualize_act”. The “visualize_act” entity includes attributes for the output that the “visualization” model is intended to represent. The Display hub entity also includes two shared attributes called the “chart_type” attribute and the “chart_collection_type” attribute. The “chart_type” attribute specifies the type of visualization that can be filtered or conditional. The type of visualization can be specified from a list of predefined values ​​supported by the computing platform (e.g., the OAC platform).

[0192] The "chart_collection_type" attribute determines whether the output will involve a single chart or a collection of charts. Unless specified, the output can include a single chart by default. Details regarding collections of charts can be found in a predefined list supported by the computing platform.

[0193] The Display_Element hub entity is the central entity used to support other queries, including Viz-Incremental and Viz-Manipulation queries. The Display_Element hub entity includes multiple attributes and actions. These attributes can include common attributes that can be shared by supporting actions. Common attributes can include the "element_name" attribute. The "element_name" attribute can support actions involving manipulating different objects in a given visualization (e.g., titles, axes, legends, filters, totals, etc.). Other attributes can include properties such as "font," "font_name," and "font_size," which support actions manipulating the font of objects in a given visualization. The "color" attribute can support actions manipulating the color of objects in a visualization. Actions can be entities with multiple attributes that effectively support visualization actions (Viz-Creation, Viz-Incremental, Viz-Manipulation). Depending on the role of the action, actions can include attributes. As an example, the move action (called move_act) additionally has a "direction" attribute with a list of values: "left," "right," "up," "top," and "bottom." In some examples, the Display hub entity and the Display_Element hub entity can share some attributes. Examples of shared attributes could include include actions (referred to as the include_act entity), move actions (referred to as the move_act entity), and replace actions (referred to as the replace_act entity). These two hub entities are linked via link attributes to existing entities within the existing schema specification, referred to as fact entities. Fact entities maintain relationships via link attributes.

[0194] Figure 8 It is a visual representation of the schema specification enhanced with additional Display, Display_Element hub entities and associated attributes for the tabular OAC Superstore dataset, according to certain embodiments. The OAC Superstore dataset is shown in Table 4 below.

[0195] Table 4:

[0196]

[0197]

[0198] Figure 9This is a visual representation of a C2OMRL model pipeline flow for transforming natural language queries into a visual representation, according to various embodiments. The C2OMRL model pipeline comprises a combination of two deep learning models, referred to as OOCS and the Semparizer deep learning model. The OOCS model first checks whether the NL utterance is relevant to the Semparizer model. The OOCS model routes relevant NL utterances to the Semparizer model. The OOCS model also routes irrelevant NL utterances (e.g., NL utterances that are out of scope or domain for the Semparizer model) to another subsystem or system, such as an ODA system for intent analysis. When an NL utterance is relevant to the Semparizer model, the NL utterance and an enhanced schema specification are fed into the Semparizer model. Figure 10 A visual representation of an enhanced schema specification that supports analytical queries according to various embodiments is shown. Figure 11 A visual representation of the enhanced schema specification supporting Viz-Creation queries according to various embodiments is shown. Figure 12 A visual representation of the enhanced schema specification supporting Viz-Incremental and Viz-Manipulation queries, according to various embodiments, is shown.

[0199] The Semparizer model generates a first internal representation based on NL utterances and an enhanced schema specification, which can be referred to as OMRQL 1.0. In some instances, the first internal representation of the NL query can then be transformed into a second internal representation, which can be referred to as OMRQL 2.0, using an OMRL 2.0 transformer. Both the first and second internal representations are suitable for consumption by a backend translation system. However, the second internal representation can be configured for a specific backend system, allowing for more accurate and efficient translation from OMRL to a given language on a particular system. For example, in instances where Viz-Creation, Viz-Incremental, and Viz-Manipulation queries are executed on OAC, the second internal representation is particularly suitable for consumption by a backend translation system referred to as the OMRL2OAC backend. The OMRL2OAC backend generates OAC API code based on OMRQL 2.0, which is then executed to produce a final visualization that is ultimately returned to the user / client (e.g., as part of a dialogue or response to the user).

[0200] Table 5 shows examples of OAC analysis queries and the translated OMRL via the C2OMRL model pipeline.

[0201] Table 5:

[0202]

[0203]

[0204] Table 6 shows examples of OAC Viz-Creation queries and the translated OMRL via the C2OMRL model pipeline.

[0205] Table 6:

[0206]

[0207]

[0208]

[0209]

[0210] Table 7 shows examples of OAC Viz-Incremental queries and OMRL translated via the C2OMRL model pipeline.

[0211] Table 7:

[0212]

[0213]

[0214] Table 8 shows examples of OAC Viz-Manipulation queries and the translated OMRL via the C2OMRL model pipeline.

[0215] Table 8:

[0216]

[0217]

[0218] Figure 13 This is a flowchart illustrating a process 1300 that uses artificial intelligence-based techniques to transform NL into a visual representation according to various embodiments. Figure 13 The processing described herein can be implemented using software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding system, hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 13 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 13 Various processing steps that occur in a specific order or sequence are depicted, but this is not limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as those shown in Figure 6- Figure 12 In the embodiments depicted, Figure 13 The processing described herein can be performed by an NL2LF (e.g., C2OMRL) model pipeline to transform natural language discourse into an intermediate logical form (e.g., MRL logical form) that includes one or more visual actions.

[0219] Beginning at box 1305, the input string is generated by concatenating (a) a natural language utterance comprising a query, one or more visualization actions, or a combination thereof, with (b) a schema representation comprising a set of entities for the visualization action. In some instances, the input string is generated by concatenating (a) a natural language utterance comprising one or more visualization actions with (b) a schema representation comprising a set of entities for the visualization action. In some instances, the schema representation also includes a schema specification for a given schema and link attributes that connect one or more entities from the set of entities for the visualization action to one or more fact entities associated with the given schema.

[0220] At box 1310, the first encoder of the semparizer machine learning model generates one or more embeddings of the input string based on the input string.

[0221] At box 1315, a second encoder of the semparizer machine learning model encodes the relationships between elements in the pattern representation and words in the natural language discourse based on one or more embeddings. In some instances, pattern-link relationships are provided to the second encoder of the semparizer machine learning model, linking entities in the pattern representation to visual action words in the natural language discourse. Embeddings can also be generated based on pattern-link relationships, and pattern-link relationships can include metadata specifying synonyms for visual action words.

[0222] In some instances, before generating one or more embeddings of natural language utterances and pattern representations, the input string is fed to a filter machine learning model (e.g., an OOCS model), which generates a score based on the input string that indicates whether the natural language utterance is translatable into a command for a computing system and should be routed to a semparizer machine learning model. This score is then compared to a threshold, and in response to determining that the score exceeds the threshold, the input string is fed to the semparizer machine learning model.

[0223] At box 1320, an intermediate logical form representing at least a query, one or more visualization actions, or a combination thereof is generated by the semparizer machine learning model's syntax-based decoder based on encoded relations and one or more embeddings. In some instances, the intermediate logical form is transformed into an enhanced intermediate logical form, which includes a query, one or more visualization actions, or a combination thereof, translatable into a version of the MRL that can be translated into a systems programming language. In some instances, the semparizer machine learning model's syntax-based decoder is presented with a relational algebra grammar representing the intermediate logical form as a tree. The intermediate logical form is generated by the syntax-based decoder based on encoded relations, one or more embeddings, and relational algebra grammar.

[0224] At box 1325, a command is generated for the computing system based on an intermediate logic form. In some instances, the command is generated for the computing system based on an enhanced intermediate logic form. The command includes a query in the form of a system programming language, one or more visual actions, or a combination thereof. In some instances, the natural language discourse includes one or more visual actions, which are represented as one or more entities from a set of entities within a schema representation, and the command includes one or more visual actions in the form of a system programming language. The command can be executed on the computing system. This execution causes the computing system to obtain results for the query, generate, update, or modify one or more visualizations, or a combination thereof, based on the one or more visual actions. The results, one or more visualizations, or the combination thereof can then be communicated to the user who provided the natural language discourse.

[0225] Explanatory System

[0226] Figure 14 A simplified diagram of a distributed system 1400 is depicted. In the example shown, the distributed system 1400 includes one or more client computing devices 1402, 1404, 1406, and 1408 coupled to a server 1412 via one or more communication networks 1410. The client computing devices 1402, 1404, 1406, and 1408 can be configured to execute one or more applications.

[0227] In various examples, server 1412 may be adapted to run one or more services or software applications that enable the implementation of one or more embodiments described in this disclosure. In some examples, server 1412 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided as web-based services or cloud services, such as under a Software as a Service (SaaS) model, to users of client computing devices 1402, 1404, 1406, and / or 1408. Users operating client computing devices 1402, 1404, 1406, and / or 1408 may then interact with server 1412 using one or more client applications to utilize the services provided by these components.

[0228] exist Figure 14 In the configuration depicted, server 1412 may include one or more components 1418, 1420, and 1422 that implement the functions performed by server 1412. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. It should be recognized that various different system configurations are possible and may differ from the distributed system 700. Therefore, Figure 14 The example shown is an example of a distributed system used to implement the example system and is not intended to be limiting.

[0229] Users can use client computing devices 1402, 1404, 1406, and / or 1408 to execute one or more applications, models, or chatbots that can generate one or more events or models, which can then be implemented or serviced in accordance with the teachings of this disclosure. The client device can provide an interface that enables users of the client device to interact with it. The client device can also output information to the user via this interface. Although Figure 14 Only four client computing devices are depicted, but any number of client computing devices can be supported.

[0230] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstations, wearable devices, gaming systems, thin clients, various messaging devices, sensors, and other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux or Linux-like operating systems, such as Google Chrome). TMOS), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android). TM Portable handheld devices can include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices can include Google Glass® head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, internet-enabled gaming devices (e.g., Microsoft Xbox® consoles with or without Kinect® gesture input, Sony PlayStation® systems, various gaming systems provided by Nintendo®, and others), etc. Client devices can be capable of running a variety of applications, such as various internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and can use various communication protocols.

[0231] One or more networks 1410 may be any type of network familiar to those skilled in the art, which may use any of a variety of available protocols to support data communication, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. By way of example only, one or more networks 1410 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any IEEE 1002.11 protocol suite, Bluetooth®, and / or any other wireless protocol) and / or any combination of these and / or other networks.

[0232] Server 1412 may consist of one or more general-purpose computers, special-purpose server computers (as examples, including PC (personal computer) servers, UNIX® servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. Server 1412 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers. In various examples, server 1412 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0233] The computing system in server 1412 can run one or more operating systems, including any of the operating systems discussed above and any commercial server operating system. Server 1412 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA® servers, database servers, etc. Exemplary database servers include, but are not limited to, commercially available database servers from Oracle®, Microsoft®, Sybase®, IBM®, etc.

[0234] In some implementations, server 1412 may include one or more applications to analyze and integrate data feeds and / or event updates received from users of client computing devices 1402, 1404, 1406, and 1408. As an example, data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party information sources and continuous data streams, which may include real-time events related to sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. Server 1412 may also include one or more applications that display data feeds and / or real-time events via one or more display devices of client computing devices 1402, 1404, 1406, and 1408.

[0235] The distributed system 1400 may also include one or more data repositories 1414, 1416. In some examples, these data repositories may be used to store data and other information. For example, one or more of data repositories 1414, 1416 may be used to store information, such as information related to the performance of the chatbot or the generated model, for use by the chatbot when the server 1412 performs various functions according to various embodiments. Data repositories 1414, 1416 may reside in various locations. For example, the data repository used by the server 1412 may be local to the server 1412 or may be located remotely to the server 1412 and communicate with the server 1412 via a network-based connection or a dedicated connection. Data repositories 1414, 1416 may be of different types. In some examples, the data repository used by the server 1412 may be a database, such as a relational database, such as a database provided by Oracle Corporation® and other vendors. One or more of these databases may be adapted to store, update, and retrieve data in the database in response to commands in SQL format.

[0236] In some examples, the application may also use one or more of data stores 1414 and 1416 to store application data. The data store used by the application can be of different types, such as, for example, a key-value store, an object store, or a general-purpose storage store supported by a file system.

[0237] In some examples, the functionality described in this disclosure may be provided as a service via a cloud environment. Figure 15 This is a simplified block diagram of a cloud-based system environment, based on certain examples, where various services can be provided as cloud services. Figure 15 In the example depicted, cloud infrastructure system 1502 can provide one or more cloud services that can be requested by users using one or more client computing devices 1504, 1506, and 1508. Cloud infrastructure system 1502 may include one or more computers and / or servers, which may include those described above for server 1412. The computers in cloud infrastructure system 1502 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0238] One or more networks 1510 may facilitate communication and data exchange between clients 1504, 1506, and 1508 and cloud infrastructure system 1502. One or more networks 1510 may include one or more networks. The networks may be of the same or different types. One or more networks 1510 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0239] Figure 15 The example depicted is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be recognized that in some other examples, cloud infrastructure system 1502 may have more than... Figure 15 The components shown may have more or fewer components, may combine two or more components, or may have different component configurations or arrangements. For example, although Figure 15 Three client computing devices are depicted, but any number of client computing devices can be supported in alternative examples.

[0240] The term "cloud service" is generally used to refer to services made available to users on demand by a service provider's systems (e.g., cloud infrastructure system 1502) and via a communication network such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's systems are different from the customer's own internal servers and systems. The cloud service provider's systems are managed by the cloud service provider. Customers can therefore utilize cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's systems can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase the infrastructure resources needed to run the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle Corporation®, based in Redwood Shores, California, offers several cloud services, such as middleware services, database services, Java Cloud services, etc.

[0241] In some examples, cloud infrastructure system 1502 may use different models such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and other models including hybrid service models to provide one or more cloud services. Cloud infrastructure system 1502 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0242] The SaaS model enables applications or software to be delivered to customers as a service via communication networks such as the internet, without requiring customers to purchase the underlying hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by cloud infrastructure systems. Examples of SaaS services provided by Oracle Corporation® include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0243] The IaaS model is typically used to provide customers with infrastructure resources (such as servers, storage devices, hardware, and networking resources) as cloud services to provide elastic computing and storage capabilities. Oracle Corporation® offers a variety of IaaS services.

[0244] The PaaS model is typically used to provide platform and environment resources as a service, enabling customers to develop, run, and manage applications and services without having to purchase, build, or maintain such resources. Examples of PaaS services provided by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solutions, and other services.

[0245] Cloud services are typically delivered on an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure basis. For example, a customer may subscribe to one or more services provided by cloud infrastructure system 1502 via a subscription order. Cloud infrastructure system 1502 then performs processing to provide the service requested in the customer's subscription order. For instance, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent), as described above, and / or to provide services to the chatbot system described herein. Cloud infrastructure system 1502 may be configured to provide one or more cloud services.

[0246] Cloud infrastructure system 1502 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 1502 can be owned by a third-party cloud service provider, and the cloud services are provided to any general public customer, who can be an individual or a business. In some other examples, under a private cloud model, cloud infrastructure system 1502 can be operated within an organization (e.g., within a business organization) and services can be provided to customers within the organization. For example, customers can be various departments within the enterprise, such as human resources, payroll, or even individuals within the enterprise. In some other examples, under a community cloud model, cloud infrastructure system 1502 and the services provided can be shared by several organizations in the relevant community. Various other models, such as a hybrid of the models mentioned above, can also be used.

[0247] Client computing devices 1504, 1506, and 1508 can be of different types (such as...) Figure 14 The client computing devices 1402, 1404, 1406, and 1408 depicted in this disclosure may be capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 1502, such as requesting services provided by the cloud infrastructure system 1502. For example, a user can use the client device to request information or actions from a chatbot, as described in this disclosure.

[0248] In some examples, the processing performed by cloud infrastructure system 1502 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, potentially processing data in parallel, using the data to perform simulations, etc. For example, big data analysis may be performed by cloud infrastructure system 1502 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0249] As in Figure 15 As depicted in the examples, cloud infrastructure system 1502 may include infrastructure resources 1530 for facilitating the provisioning of various cloud services offered by cloud infrastructure system 1502. Infrastructure resources 1530 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to serve storage requested from an application may be part of cloud infrastructure system 1502. In other examples, the storage virtual machine may be part of a different system.

[0250] In some examples, to facilitate the efficient provisioning of these resources to support the various cloud services provided by cloud infrastructure system 1502 to different customers, resources can be bundled into groups of resources or resource modules (also known as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods may be pre-provisioned for different types of cloud services. For example, a first pod may be provisioned for a database service, and a second pod may be provisioned for a Java service, where the second pod may include a different combination of resources than the pods in the first pod. For some services, resources allocated for provisioning services may be shared between services.

[0251] The cloud infrastructure system 1502 can internally use service 1532, which is shared by different components of the cloud infrastructure system 1502 and facilitates the service provisioning of the cloud infrastructure system 1502. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0252] Cloud infrastructure system 1502 may include multiple subsystems. These subsystems may be implemented using software or hardware, or a combination thereof. Figure 8As shown, the subsystem may include a user interface subsystem 1512 that enables users or customers of the cloud infrastructure system 1502 to interact with it. The user interface subsystem 1512 may include various interfaces, such as a web interface 1514, an online store interface 1516 (where cloud services provided by the cloud infrastructure system 1502 are advertised and available for purchase by consumers), and other interfaces 1518. For example, a customer may use a client device to request (service request 1534) one or more services provided by the cloud infrastructure system 1502 using one or more of interfaces 1514, 1516, and 1518. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 1502, and place an order for one or more services provided by the cloud infrastructure system 1502 that the customer wishes to subscribe to. A service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place an order for services provided by the cloud infrastructure system 1502. As part of the order, the customer can provide information identifying the chatbot system for which they wish to provide services, as well as one or more optional credentials for the chatbot system.

[0253] In some examples, such as Figure 15 As illustrated in the example, cloud infrastructure system 1502 may include an order management subsystem (OMS) 1520 configured to process new orders. As part of this process, OMS 1520 may be configured to: create accounts for customers (if not already created); receive invoices and / or accounting information from customers, which will be used to bill customers for the requested services; verify customer information; and, upon verification, place an order for the customer; and have orchestrated various workflows to prepare the order for supply.

[0254] Once correctly verified, OMS 1520 can invoke the Order Provisioning Subsystem (OPS) 1524, which is configured to provision resources for the order, including processing resources, storage resources, and networking resources. Provisioning may include allocating resources for the order and configuring resources to facilitate the service requested by the customer's order. The method and type of resource provisioning for the order may depend on the type of cloud service the customer has subscribed to. For example, according to a workflow, OPS 1524 may be configured to determine the specific cloud service requested and identify multiple clusters that may have been pre-configured for that specific cloud service. The number of clusters allocated to the order may depend on the size / quantity / level / scope of the requested service. For example, the number of clusters to be allocated may be determined based on the number of users supported by the service, the duration of the requested service, etc. The allocated clusters can then be customized for the specific requesting customer to provide the requested service.

[0255] In some examples, the setup phase processing described above can be performed by cloud infrastructure system 1502 as part of the provisioning process. Cloud infrastructure system 1502 can generate application IDs and select storage virtual machines for applications from storage virtual machines provided by cloud infrastructure system 1502 itself or from storage virtual machines provided by other systems besides cloud infrastructure system 1502.

[0256] Cloud infrastructure system 1502 may send a response or notification 1544 to the requesting client to indicate that the requested service is now ready for use. In some cases, information (e.g., a link) may be sent to the client, enabling the client to begin using and taking advantage of the benefits of the requested service. In some examples, for the client requesting the service, the response may include a chatbot system ID generated by cloud infrastructure system 1502 and information identifying the chatbot system selected by cloud infrastructure system 1502 for the chatbot system corresponding to the chatbot system ID.

[0257] Cloud infrastructure system 1502 can provide services to multiple customers. For each customer, cloud infrastructure system 1502 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1502 can also collect usage statistics about customers' use of subscribed services. For example, statistics can be collected on storage usage, data transfer volume, number of users, system uptime, and system downtime. This usage information can be used for billing customers. Billing can be, for example, on a monthly basis.

[0258] Cloud infrastructure system 1502 can provide services to multiple customers in parallel. Cloud infrastructure system 1502 can store information about these customers, including potentially proprietary information. In some examples, cloud infrastructure system 1502 includes an Identity Management Subsystem (IMS) 1528, which is configured to manage customer information and provide separation of the managed information, making information associated with one customer inaccessible to another. IMS 1528 can be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles and related capabilities, and so on.

[0259] Figure 16 An example of computer system 1600 is illustrated. In some examples, computer system 1600 can be used to implement any digital assistant or chatbot system within a distributed environment and within the various servers and computer systems described above. Figure 16As shown, computer system 1600 includes various subsystems, including a processing subsystem 1604 that communicates with multiple other subsystems via a bus subsystem 1602. These other subsystems may include a processing acceleration unit 1606, an I / O subsystem 1608, a storage subsystem 918, and a communication subsystem 1624. Storage subsystem 918 may include a non-transitory computer-readable storage medium, including storage medium 1622 and system memory 1610.

[0260] Bus subsystem 1602 provides a mechanism for enabling various components and subsystems of computer system 1600 to communicate with each other as desired. While bus subsystem 1602 is schematically shown as a single bus, alternative examples of bus subsystems may utilize multiple buses. Bus subsystem 1602 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, local buses using any of the various bus architectures, etc. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as mezzanine buses manufactured according to the IEEE P1386.1 standard, etc.

[0261] Processing subsystem 1604 controls the operation of computer system 1600 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Processors may include single-core or multi-core processors. The processing resources of computer system 1600 may be organized into one or more processing units 1632, 1634, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 1604 may include one or more dedicated coprocessors, such as graphics processors, digital signal processors (DSPs), etc. In some examples, some or all of processing subsystem 1604 may be implemented using custom circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0262] In some examples, the processing units in processing subsystem 1604 can execute instructions stored in system memory 1610 or on computer-readable storage medium 1622. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in system memory 1610 and / or on computer-readable storage medium 1622, including possibly on one or more storage devices. With appropriate programming, processing subsystem 1604 can provide the various functions described above. In the case where computer system 1600 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0263] In some examples, a processing acceleration unit 1606 may optionally be provided for performing custom processing or for offloading some processing performed by the processing subsystem 1604, thereby accelerating the overall processing performed by the computer system 1600.

[0264] I / O subsystem 1608 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 1600. Generally, the term "input device" is used to include all possible types of devices and mechanisms for inputting information to computer system 1600. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens incorporated into a display, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices such as the Microsoft Kinect® motion sensor, the Microsoft Xbox® 360 game controller, and devices that provide an interface for receiving input using gestures and spoken commands, enabling users to control and interact with the input device. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., "blinking" when taking pictures and / or making menu selections) and translates eye gestures into input to an input device (e.g., Google Glass®). In addition, user interface input devices may include voice recognition sensing devices that enable users to interact with voice recognition systems (such as the Siri® navigator) via voice commands.

[0265] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointers, gamepads and graphics tablets, and audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission computed tomography, and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0266] Generally, the term "output device" is used to encompass all possible types of devices and mechanisms for outputting information from computer system 1600 to a user or other computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those utilizing liquid crystal displays (LCDs) or plasma displays, projection devices, touch screens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0267] Storage subsystem 1618 provides a repository or data storage facility for storing information and data used by computer system 1600. Storage subsystem 1618 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Storage subsystem 1618 may store software (e.g., programs, code modules, instructions) that provides the aforementioned functionality when executed by processing subsystem 1604. The software may be executed by one or more processing units of processing subsystem 1604. Storage subsystem 1618 may also provide certification according to the teachings of this disclosure.

[0268] The storage subsystem 1618 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. For example... Figure 16As shown, the storage subsystem 1618 includes system memory 1610 and computer-readable storage medium 1622. System memory 1610 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some embodiments, the basic input / output system (BIOS) may typically be stored in ROM, containing basic routines that facilitate the transfer of information between elements within the computer system 1600, for example, during startup. RAM typically contains data and / or program modules currently operated and executed by the processing subsystem 1604. In some embodiments, system memory 1610 may include various different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0269] For example, rather than a limitation, such as Figure 16 As shown, system memory 1610 can load an application 1612, program data 1614, and operating system 1616 that is being executed, which may include various applications such as web browsers, middleware applications, relational database management systems (RDBMS), etc. As an example, operating system 1616 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercial UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS, and other operating systems.

[0270] Computer-readable storage medium 1622 can store programming and data structures that provide some example functionality. Computer-readable storage medium 1622 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 1600. When executed by processing subsystem 1604, the software (programs, code modules, instructions) providing the above functionality can be stored in storage subsystem 1618. As an example, computer-readable storage medium 1622 may include non-volatile memory such as hard disk drives, disk drives, optical disc drives such as CD ROMs, DVDs, Blu-ray® discs, or other optical media. Computer-readable storage medium 1622 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital video tapes, etc. Computer-readable storage medium 1622 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs), and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs.

[0271] In some examples, storage subsystem 1618 may also include a computer-readable storage medium reader 1620, which may also be connected to a computer-readable storage medium 1622. Reader 1620 may receive and be configured to read data from a memory device such as a disk, flash drive, etc.

[0272] In some examples, computer system 1600 may support virtualization technologies, including but not limited to the virtualization of processing and memory resources. For example, computer system 1600 may provide support for executing one or more virtual machines. In some examples, computer system 1600 may execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Memory, computing (e.g., processor, cores), I / O, and networking resources may be allocated to each virtual machine. Each virtual machine typically runs independently of other virtual machines. Virtual machines typically run their own operating systems, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 1600. Thus, computer system 1600 may potentially run multiple operating systems simultaneously.

[0273] The communication subsystem 1624 provides an interface to other computer systems and networks. The communication subsystem 1624 serves as an interface for receiving data from the computer system 1600 and transmitting data to other systems. For example, the communication subsystem 1624 enables the computer system 1600 to establish a communication channel via the Internet to one or more client devices for receiving and sending information to client devices. For example, when the computer system 1600 is used to implement... Figure 1 When the robot system 120 is described, the communication subsystem can be used to communicate with the chatbot system selected for the application.

[0274] The communication subsystem 1624 may support both wired and / or wireless communication protocols. In some examples, the communication subsystem 1624 may include radio frequency (RF) transceiver components, global positioning system (GPS) receiver components, and / or other components for accessing wireless voice and / or data networks, for example, using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 2402.XX family of standards), or other mobile communication technologies, or any combination thereof. In some examples, as an addition to or alternative to the wireless interface, the communication subsystem 1624 may provide a wired network connection (e.g., Ethernet).

[0275] The communication subsystem 1624 can receive and transmit data in various forms. In some examples, among others, the communication subsystem 1624 can also receive input communications in the form of structured and / or unstructured data feeds 1626, event streams 1628, event updates 1630, etc. For example, the communication subsystem 1624 can be configured to receive (or send) data feeds 1626 in real time from users of social media networks and / or other communication services such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0276] In some examples, the communication subsystem 1624 can be configured to receive data that may be continuous or unbounded without a definite end, in the form of a continuous data stream, wherein the continuous data stream may include an event stream 1628 of real-time events and / or event updates 1630. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0277] The communication subsystem 1624 can also be configured to transmit data from computer system 1600 to other computer systems or networks. Data can be transmitted to one or more databases in various forms, such as structured and / or unstructured data feeds 1626, event streams 1628, event updates 1630, etc., and these one or more databases can communicate with one or more streaming data sources coupled to computer system 1600.

[0278] The computer system 1600 can be of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), personal computers, workstations, mainframes, information stations, server racks, or any other data processing systems. Due to the constantly evolving nature of computers and networks, [the following applies]. Figure 16 The description of the computer system 1600 shown in the illustration is intended only as a concrete example. It has more than Figure 16 Many other configurations of the system with more or fewer components are possible. Based on the disclosures and teachings provided herein, it should be recognized that other ways and / or methods exist to implement the various examples.

[0279] While specific examples have been described, various modifications, alterations, alternative constructions, and equivalents are possible. The examples are not limited to operations within certain specific data processing environments but can operate freely within multiple data processing environments. Furthermore, although some examples have been described using a specific series of transactions and steps, it will be clear to those skilled in the art that this is not intended to be limiting. While some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Moreover, the order of operations can be rearranged. Processes may have additional steps not included in the diagrams. Various features and aspects of the above examples can be used individually or in combination.

[0280] Furthermore, while certain examples have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. Some examples can be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same or different processors.

[0281] When a device, system, component, or module is described as being configured to perform certain operations or functions, such configuration can be achieved, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors), such as by executing computer instructions or code, or by programming a processor or core to execute code or instructions stored on a non-transitory storage medium, or any combination thereof. Processes can communicate using various technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes can use different technologies, or the same pair of processes can use different technologies at different times.

[0282] Specific details are set forth in this disclosure to provide a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides exemplary examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the prior description of the examples will provide those skilled in the art with enabling descriptions for implementing the various examples. Various changes can be made to the function and arrangement of the elements.

[0283] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be clear that additions, omissions, deletions, and other modifications and alterations may be made therein without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0284] In the foregoing specification, aspects of this disclosure have been described with reference to specific examples therein; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used alone or in combination. Furthermore, the examples may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the accompanying drawings are to be considered illustrative rather than restrictive.

[0285] In the foregoing description, the methods have been described in a specific order for illustrative purposes. It should be recognized that, in alternative examples, these methods may be performed in a different order than described. It should also be recognized that the methods described above can be executed by hardware components or can be implemented in a sequence of machine-executable instructions that can be used to cause a machine, such as a general-purpose or special-purpose processor, or logic circuits programmed with instructions, to execute the methods. These machine-executable instructions can be stored on one or more machine-readable media, such as a CD-ROM or other types of optical discs, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic cards or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, these methods can be executed by a combination of hardware and software.

[0286] When a component is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits), or any combination thereof.

[0287] While illustrative examples of this application have been described in detail herein, it should be understood that the inventive concept may be implemented and adopted in other ways, and the appended claims are intended to be construed as including such variations, other than those limited by the prior art.

Claims

1. A computer-implemented method, comprising: The input string is generated by concatenating (a) a natural language discourse comprising a query, one or more visual actions, or a combination thereof, with (b) a pattern representation comprising a set of entities for the visual actions; The first encoder of the semparizer machine learning model generates one or more embeddings of the input string based on the input string; The second encoder of the semparizer machine learning model encodes the relationship between elements in the pattern representation and words in the natural language discourse based on the one or more embeddings; The semparizer machine learning model's syntax-based decoder generates an intermediate logical form that at least represents the query, the one or more visualization actions, or a combination thereof, based on the encoded relations and the one or more embeddings. as well as Commands for the computing system are generated based on the intermediate logic, wherein the commands include the query in the form of a system programming language, one or more visual actions, or a combination thereof.

2. The computer-implemented method according to claim 1, further comprising: Execute the command on the computing system, wherein the execution causes the computing system to obtain results for the query, and generate, update or modify one or more visualizations, or a combination thereof, based on the one or more visualization actions; as well as The results, one or more visualizations, or a combination thereof, are transmitted to the user who provided the natural language discourse. The natural language discourse includes the one or more visual actions, and the one or more visual actions are represented as one or more entities from the set of entities within the pattern representation.

3. The computer-implemented method according to claim 1 further includes: The input string is provided to the filter machine learning model before generating the natural language utterance and the one or more embeddings of the pattern representation; The filter machine learning model generates a score based on the input string, the score indicating whether the natural language utterance can be translated into a command for the computing system and should be routed to the semparizer machine learning model; The score is compared with a threshold. as well as In response to determining that the score exceeds the threshold, the input string is provided to the semparizer machine learning model.

4. The computer-implemented method according to claim 1, further comprising: The intermediate logic form is converted into an enhanced intermediate logic form, which includes the query, the one or more visual actions, or a combination thereof, that can be translated into a version of the Meaning Representation Language (MRL) form of the system programming language. as well as The commands for the computing system are generated based on the enhanced intermediate logic, wherein the commands include the queries in the form of the system programming language, one or more visual actions, or a combination thereof.

5. The computer-implemented method of claim 1, further comprising providing a pattern-link relationship to the second encoder of the semparizer machine learning model, the pattern-link relationship linking entities in the pattern representation to visual action words in the natural language discourse, wherein the embedding is further generated based on the pattern-link relationship, and wherein the pattern-link relationship includes metadata specifying synonyms for visual action words.

6. The computer-implemented method of claim 1, wherein the pattern representation further includes a pattern specification for a given pattern, and a link attribute that connects one or more entities from the set of entities used to visualize actions to one or more fact entities associated with the given pattern.

7. The computer-implemented method of claim 1, further comprising providing a relational algebra grammar to the syntax-based decoder of the semparizer machine learning model, the relational algebra grammar representing the intermediate logical form as a tree, wherein the intermediate logical form is generated by the syntax-based decoder based on encoded relations, the one or more embeddings, and the relational algebra grammar.

8. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a system to perform operations including: The input string is generated by concatenating (a) a natural language discourse comprising a query, one or more visual actions, or a combination thereof, with (b) a pattern representation comprising a set of entities for the visual actions; The first encoder of the semparizer machine learning model generates one or more embeddings of the input string based on the input string; The second encoder of the semparizer machine learning model encodes the relationship between elements in the pattern representation and words in the natural language discourse based on the one or more embeddings; The semparizer machine learning model's syntax-based decoder generates an intermediate logical form that at least represents the query, the one or more visualization actions, or a combination thereof, based on the encoded relations and the one or more embeddings. as well as Commands for the computing system are generated based on the intermediate logic, wherein the commands include the query in the form of a system programming language, one or more visual actions, or a combination thereof.

9. The one or more non-transitory computer-readable media according to claim 8, further comprising: Execute the command on the computing system, wherein the execution causes the computing system to obtain results for the query, and generate, update or modify one or more visualizations, or a combination thereof, based on the one or more visualization actions; as well as The results, one or more visualizations, or a combination thereof, are transmitted to the user who provided the natural language discourse. The natural language discourse includes the one or more visual actions, and the one or more visual actions are represented as one or more entities from the set of entities within the pattern representation.

10. The one or more non-transitory computer-readable media according to claim 8, further comprising: The input string is provided to the filter machine learning model before generating the natural language utterance and the one or more embeddings of the pattern representation; The filter machine learning model generates a score based on the input string, the score indicating whether the natural language utterance can be translated into a command for the computing system and should be routed to the semparizer machine learning model; The score is compared with a threshold. as well as In response to determining that the score exceeds the threshold, the input string is provided to the semparizer machine learning model.

11. The one or more non-transitory computer-readable media according to claim 8, further comprising: The intermediate logic form is converted into an enhanced intermediate logic form, which includes the query, the one or more visual actions, or a combination thereof, that can be translated into a version of the Meaning Representation Language (MRL) form of the system programming language. as well as The commands for the computing system are generated based on the enhanced intermediate logic, wherein the commands include the queries in the form of the system programming language, one or more visual actions, or a combination thereof.

12. The one or more non-transitory computer-readable media of claim 8, further comprising providing a pattern-link relationship to the second encoder of the semparizer machine learning model, the pattern-link relationship linking entities in the pattern representation to visual action words in the natural language discourse, wherein the embedding is further generated based on the pattern-link relationship, and wherein the pattern-link relationship includes metadata specifying synonyms for visual action words.

13. One or more non-transitory computer-readable media according to claim 8, wherein the schema representation further includes a schema specification for a given schema, and a link attribute that connects one or more entities from the set of entities used to visualize actions to one or more fact entities associated with the given schema.

14. The one or more non-transitory computer-readable media of claim 8, further comprising providing a relational algebra grammar to the syntax-based decoder of the semparizer machine learning model, the relational algebra grammar representing the intermediate logical form as a tree, wherein the intermediate logical form is generated by the syntax-based decoder based on encoded relations, the one or more embeddings, and the relational algebra grammar.

15. A system comprising: One or more processors; as well as One or more computer-readable media storing instructions that, when executed by the one or more processors, cause a system to perform methods including: The input string is generated by concatenating (a) a natural language discourse comprising a query, one or more visual actions, or a combination thereof, with (b) a pattern representation comprising a set of entities for the visual actions; The first encoder of the semparizer machine learning model generates one or more embeddings of the input string based on the input string; The second encoder of the semparizer machine learning model encodes the relationship between elements in the pattern representation and words in the natural language discourse based on the one or more embeddings; The semparizer machine learning model's syntax-based decoder generates an intermediate logical form that at least represents the query, the one or more visualization actions, or a combination thereof, based on the encoded relations and the one or more embeddings. as well as Commands for the computing system are generated based on the intermediate logic, wherein the commands include the query in the form of a system programming language, one or more visual actions, or a combination thereof.

16. The system of claim 15, further comprising: Execute the command on the computing system, wherein the execution causes the computing system to obtain results for the query, and generate, update or modify one or more visualizations, or a combination thereof, based on the one or more visualization actions; as well as The results, one or more visualizations, or a combination thereof, are transmitted to the user who provided the natural language discourse. The natural language discourse includes the one or more visual actions, and the one or more visual actions are represented as one or more entities from the set of entities within the pattern representation.

17. The system of claim 15, further comprising: The input string is provided to the filter machine learning model before generating the natural language utterance and the one or more embeddings of the pattern representation; The filter machine learning model generates a score based on the input string, the score indicating whether the natural language utterance can be translated into a command for the computing system and should be routed to the semparizer machine learning model; The score is compared with a threshold. as well as In response to determining that the score exceeds the threshold, the input string is provided to the semparizer machine learning model.

18. The system of claim 15, further comprising: The intermediate logic form is converted into an enhanced intermediate logic form, which includes the query, the one or more visual actions, or a combination thereof, that can be translated into a version of the Meaning Representation Language (MRL) form of the system programming language. as well as The commands for the computing system are generated based on the enhanced intermediate logic, wherein the commands include the queries in the form of the system programming language, one or more visual actions, or a combination thereof.

19. The system of claim 15, further comprising providing a pattern-link relationship to the second encoder of the semparizer machine learning model, the pattern-link relationship linking entities in the pattern representation to visual action words in the natural language discourse, wherein the embedding is further generated based on the pattern-link relationship, and wherein the pattern-link relationship includes metadata specifying synonyms for visual action words.

20. The system of claim 15, wherein the pattern representation further includes a pattern specification for a given pattern, and a link attribute that connects one or more entities from the set of entities used to visualize actions to one or more fact entities associated with the given pattern.

21. An apparatus comprising: Components for implementing the operation of the method according to any one of claims 1-7.

22. A computer program product comprising computer instructions that, when executed by a processor, implement the operation of the method according to any one of claims 1-7.

Images