Maintaining state and context of a conversation between a user and a digital assistant using threads

Abstract

Techniques for maintaining state and context of a conversation between a user and a digital assistant using threads. In one aspect, a method includes receiving, during a session, a natural language utterance from a user, obtaining a topic context instance for the natural language utterance, and generating, by a GenAI model, a list including executable actions based on candidate actions associated with the topic context instance. The executable actions are then executed to produce an output. The executing includes determining that no thread associated with the topic context instance, the executable actions, or both is running, and creating a thread associated with the topic context instance, the executable actions, or both in response to determining that no thread is running, and executing the executable actions using the thread to obtain the output. The output or a communication derived from the output is then sent to the user.

Description

[0001] Priority requirements

[0002] This application claims the interest and priority of U.S. Provisional Application No. 63 / 583,161, filed September 15, 2023, pursuant to 35 USC 119(e), the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure generally relates to artificial intelligence technology, and more specifically to techniques for using threads to maintain the state and context of conversations between a user and a digital assistant. Background Technology

[0004] Artificial intelligence (AI) has a variety of applications, and one example is its use in providing instant responses in instant messaging or chat platforms. Organizations leverage these platforms to engage in real-time conversations with customers, but hiring human service bots for this purpose would be extremely expensive. To address this issue, chatbots—automated programs designed to simulate human conversation—were developed, particularly for internet use. These chatbots can be integrated into existing messaging applications that users are already familiar with. Initially, chatbots were simple programs designed to simulate conversations with users through text-based interactions, often following predefined scripts with limited capabilities. These early chatbots were primarily used for basic customer service tasks, such as answering common questions or providing information about products and services.

[0005] The evolution of chatbots into more sophisticated chatbot systems, such as digital assistants, is driven by advancements in AI and the growing demand for more complex and interactive user experiences. More specifically, with advancements in AI technologies, particularly Natural Language Processing (NLP) and Machine Learning (ML), chatbots are evolving into more intelligent and context-aware systems. NLP enables chatbots to understand and process human language more effectively, allowing them to understand context, manage ambiguity, and handle diverse linguistic nuances. This shift allows chatbots to engage in more natural and meaningful conversations, moving beyond simple keyword-based interactions to understand user intent and provide more relevant responses. ML enables chatbots to understand voice commands, interact with various applications and services, manage schedules, control smart devices, and provide personalized recommendations. AI's continuous learning and adaptive capabilities ensure that chatbots can evolve with user needs and preferences, providing a more seamless and intuitive user experience. This evolution from simple chatbots to complex chatbot systems represents a significant leap forward in AI's ability to enhance daily life and business operations. Summary of the Invention

[0006] In various embodiments, a computer-implemented method includes: receiving, at a digital assistant, natural language utterances from a user during a session between the user and the digital assistant; obtaining a topic context instance for the natural language utterances; generating, by a generative artificial intelligence (GenAI) model, a list of executable actions based on one or more candidate actions associated with the topic context instance; and executing the executable action to produce output, wherein the execution includes: determining that no thread is running within the session associated with the topic context instance, the executable action, or both; in response to determining that no thread is running, creating a thread associated with the topic context instance, the executable action, or both, wherein the thread maintains the topic context instance, the executable action, or both as a context separate from and associated with any other conversation, for a distinctly different conversation including the natural language utterances between the user and the digital assistant; and using the thread to execute the executable action to obtain output; and sending the output or communications derived from the output to the user.

[0007] In some embodiments, the computer-implemented method further includes: at a digital assistant, receiving subsequent natural language utterances from the user during a session between the user and the digital assistant; obtaining a topic context instance for the natural language utterances; generating, by a GenAI model, another list including another executable action based on one or more candidate actions associated with the topic context instance; executing the other executable action, wherein execution includes: identifying a thread running within the session associated with the topic context instance, the executable action, or both; determining, based on an identifier associated with the topic context instance, the executable action, or both, that the other executable action should be executed using the thread; and using the thread to execute the other executable action to obtain another output; and sending the other output or communication derived from the other output to the user.

[0008] In some embodiments, the computer-implemented method further includes: at a digital assistant, receiving subsequent natural language utterances from a user during a session between the user and the digital assistant; obtaining a topic context instance for the subsequent natural language utterances; generating, by a GenAI model, another list including another executable action based on one or more candidate actions associated with the topic context instance for the subsequent natural language utterances; and executing the other executable action, wherein execution includes: identifying a thread running within the session that is associated with the topic context instance for the natural language utterances, the executable action, or both; determining that no thread running within the session is associated with the topic context instance for the subsequent natural language utterances, the other executable action, or both. In response to determining that no thread is running within the session associated with a topic context instance for subsequent natural language utterances, another thread is created having an identifier associated with the topic context instance for subsequent natural language utterances, the other executable action, or both, wherein the other thread maintains the topic context instance for subsequent natural language utterances, the other executable action, or both as a separate context for another distinct conversation between the user and the digital assistant, and performs the other executable action using the other thread to obtain another output; and sends the other output or communication derived from the other output to the user.

[0009] In some embodiments, the computer-implemented method further includes: in response to identifying that a thread running within a session is associated with a topic context instance for a natural language utterance, an executable action, or both, and determining that no thread is running within the session associated with a topic context instance for a subsequent natural language utterance, sending an acknowledgment communication to the user requesting confirmation of the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both; and at the digital assistant, receiving an acknowledgment from the user confirming the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both, wherein in response to determining that no thread is running within the session associated with a topic context instance for the subsequent natural language utterance and receiving the acknowledgment, the other thread is created.

[0010] In some embodiments, the computer-implemented method further includes: receiving a message payload comprising (i) an indication of a subsequent utterance or event associated with an interaction between a user and an application, and (ii) an application context; identifying an identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both is the same identifier associated with a topic context instance, an executable action, or both; and processing the message payload using a thread in response to determining that the identifier is the same.

[0011] In some embodiments, the computer-implemented method further includes: receiving a message payload comprising: (i) an indication of a subsequent utterance or event associated with an interaction between a user and an application, and (ii) an application context; identifying an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both is different from an identifier associated with a topic context instance, an executable action, or both; and in response to determining that the identifier is different, creating another thread having an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both, wherein the other thread maintains the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both as a separate context for another distinctly different conversation between the user and the digital assistant; and processing the message payload using the other thread.

[0012] In some embodiments, the computer-implemented method further includes: in response to determining that the identifier is different, sending a confirmation communication to the user requesting confirmation of (i) a subsequent statement or interface action, (ii) an application context, or (iii) both; and at the digital assistant, receiving confirmation from the user of (i) a subsequent statement or interface action, (ii) an application context, or (iii) both, wherein in response to determining that the identifier is different and receiving the confirmation message, the other thread is created.

[0013] In various embodiments, an apparatus is provided that includes components for implementing some or all of the operations and / or methods disclosed herein.

[0014] In various embodiments, a computer program product is provided that includes computer instructions that, when executed by a processor, implement some or all of the operations and / or methods disclosed herein.

[0015] In some embodiments, a system is provided that includes one or more processors and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0016] In some embodiments, a non-transitory computer-readable medium is provided that stores one or more instructions that, when executed by one or more processors, cause a system to perform some or all of the operations and / or methods disclosed herein.

[0017] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided in the diagram below, as described in more detail below. However, the implementations and contexts below are only a few of the many. Attached Figure Description

[0018] Figure 1 It is a simplified block diagram of a distributed environment according to various embodiments.

[0019] Figure 2 This is a simplified block diagram of a computing system for implementing a master robot according to various embodiments.

[0020] Figure 3 This is a simplified block diagram of a computing system for implementing a skill-based robot according to various embodiments.

[0021] Figure 4 These are simplified example system architectures based on various embodiments for enabling multiple users to interact with a set of large language models (LLMs).

[0022] Figure 5 This is a simplified block diagram of a distributed environment incorporating a chatbot system according to various embodiments.

[0023] Figure 6 This is an exemplary architecture of an LLM-based digital assistant according to various embodiments.

[0024] Figure 7 The illustration shows a side panel chat as part of the Copilot UI according to various embodiments.

[0025] Figure 8 The illustration shows micro-conversation modes as part of the Copilot UI according to various embodiments.

[0026] Figure 9 This is a simplified block diagram of the workflow for implementing copilot according to various embodiments.

[0027] Figure 10The illustrations depict the recognition of prompts (with one or more prompt parameters) based on flow and intent according to various embodiments.

[0028] Figure 11 The illustrations depict exemplary schema payloads (e.g., JSON payloads) required for generating digital assistant artifacts according to various embodiments.

[0029] Figure 12 This is a flowchart, according to various embodiments, of the process for maintaining the state and context of a conversation between a user and a digital assistant using threads.

[0030] Figure 13 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0031] Figure 14 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0032] Figure 15 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0033] Figure 16 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0034] Figure 17 This is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Implementation

[0035] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of some embodiments. However, it will be apparent, however, that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word “exemplary” as used herein means “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other embodiments or designs.

[0036] introduction

[0037] Artificial intelligence has numerous applications. For example, digital assistants are AI-powered interfaces that help users complete various tasks by conversing using natural language. For each digital assistant, a customer can assemble one or more skills. Skills (also described in this document as chatbots, bots, or skill bots) are individual bots focused on specific types of tasks, such as tracking inventory, submitting timecards, and creating expense reports. When an end user interacts with a digital assistant, the assistant evaluates the end user's input and routes the conversation to or from the appropriate chatbot. Digital assistants can be delivered to end users through various channels, such as FACEBOOK® Messenger, SKYPE MOBILE® Messenger, or the Short Message Service (SMS). Channels relay chat back and forth between the end user and the digital assistant and its various chatbots on various messaging platforms. Channels can also support user agent upgrades, event-initiated conversations, and testing.

[0038] Intents allow AI-based technologies, such as chatbots, to understand what a user wants the chatbot to do. An intent is a user's intention conveyed to the chatbot via user requests and statements (also known as utterances, e.g., getting account balance, making a purchase, etc.). As used in this paper, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. Intents can be created by providing a name that describes a certain user action (e.g., ordering pizza) and compiling a set of real-life user statements or utterances typically associated with triggering that action. Because the chatbot's cognition stems from these intents, each intent can be created from a robust (one to a dozen or so utterances) and diverse dataset so that the chatbot can interpret ambiguous user input. A rich set of utterances allows the chatbot to understand what the user wants when receiving messages such as "Forgot this order!" or "Cancel delivery!"—these messages mean the same thing but are expressed differently. In general, intents and the utterances associated with them constitute the chatbot's training corpus. By training the model with the corpus, the client can essentially turn the model into a reference tool for parsing end-user input into single intents. Customers can improve the chatbot's cognitive acuity through several rounds of intent testing and intent training.

[0039] Once the chatbot understands the user's intent, it can execute a conversational flow. A flow is a segment of a conversation that defines interactions with the user to accomplish a task or part of a task the user wants to perform. Typical examples of flows include:

[0040] Intent-driven flows, where each intent defined in a skill has an associated flow, such as “order pizza,” “remit money,” or “create fee.”

[0041] Support or utility flows used for tasks such as user authorization, new user onboarding, logging, or providing user assistance. Such flows can be invoked from multiple flows. For example, you might have a create account subflow invoked from a flow such as ordering pizza or making a money transfer.

[0042] Generally, flows can be broken down into the following types: mainstream, intent flow, flows for built-in events and system transitions, and sub-flows that can be used by the top-level flow. The mainstream isn't really such a flow. More accurately, it's the control center for skills, from which users are directed to specialized flows mapped to intents. Within intents and sub-flows, various actions can be defined to support what the user wants the chatbot to do based on the understood intent. These actions can include conversational dialogue, performing database queries, sentiment analysis, data analysis, API and REST calls, and more.

[0043] Recently, large language models (LLMs) have been integrated into digital assistants to leverage generative AI capabilities to enhance skills. These capabilities include handling small talk with users, generating written summaries of data, automating challenging or repetitive business tasks (such as those required for talent acquisition), and providing sentiment analysis of a given text fragment to determine whether it reflects positive, negative, or neutral opinions. Using callable large language model components (LLM components), skill robot developers can insert these capabilities anywhere in the conversation flow where they require them. This conversation flow component is the primary integrator of generative AI because it contacts the LLM via a call (e.g., a REST call), then sends the LLM prompts (natural language instructions to the LLM) along with relevant parameters. It then returns the results generated by the model (also known as completion) and manages the state of the LLM-user interaction so that its responses remain in context after successive rounds of user queries and feedback. An LLM component can call any LLM. Users can add one or more LLM component states (or LLM blocks) to the flow. Users can also chain LLM calls so that the output of one LLM request can be passed to subsequent LLM requests.

[0044] Copilot represents a new generation of innovative user experiences designed to enhance applications, such as business applications. Leveraging the digital assistant framework, developers can create customized digital assistants incorporating a variety of skills, including specialized functionalities such as out-of-the-box digital assistants with skills like expense management, project management, and human capital management (HCM). As discussed in this article, skill developers now have the ability to integrate generative AI components, such as LLM, into their skill streams, allowing them to introduce generative AI capabilities into their existing digital assistants. With the hybrid capabilities of digital assistants, copilot can seamlessly integrate with applications, providing both conversational and generative AI experiences tailored to user needs while applications are used for a wide range of tasks. The challenge is developing / configuring copilot so that it can be used regardless of which application it is integrated into. If copilot cannot be used across various applications, it will break if the application or its version changes.

[0045] To address this and other challenges, the copilot platform was developed / configured to employ various techniques to achieve a relatively loose coupling between the digital assistant (DA) and the application. These techniques include context passing, interaction modalities (side panel chat and micro-chat chat), and copilot actions. Context passing involves obtaining context from the application, the pages displayed by the application, and the user using the application, so that copilot can better understand the situation and collaborate with the DA to provide the user with the most relevant information about the application. Interaction modalities are interfaces that allow users to interact with copilot while still viewing the application, and include side panel chat and micro-chat chat. Copilot actions are actions performed or triggered by copilot and used to control views and information within the application. These techniques (described in more detail herein) enable a context-aware, personalized journey for the application's users, proactively assisting them with specific suggestions or actions, assisting with content generation and formatting, and providing users with smooth application navigation and dynamic updates. This improves the technological landscape for facilitating digital interaction between users and computational resources such as chatbots.

[0046] Threads are used to allow or maintain the state and context of conversations between a user and a digital assistant (DA). Threading is the maintenance of multiple isolated conversations within the same session between a user and a DA. A thread identifier is the key concept for switching between threads and determining which thread is being used or should be used. Threads are initiated as part of an executable action identified via a GenAI model such as LLM. Alternatively, threads are initiated as part of an executable action directly identified via a UI event. Executable actions have a thread ID configuration, such as thread ID = patient name[-] patient ID when specific to a patient record. Threads maintain state with a conversation or task context for a specific dialogue or flow, which includes all utterances, responses, actions performed, etc. The conversation or task context is stored in a context storage repository associated with the thread ID. A thread manager handles all thread IDs, states, and switching. Threading has the benefits of ensuring that all utterances and actions are grouped based on a given domain (such as a patient identifier) ​​(e.g., the doctor does not incorrectly mix data / information between patients). Threads can also be separated based on multiple domains (such as by patient and by action). One assistive technical challenge is knowing when to switch threads, especially when the utterance doesn't match the context. Therefore, the dialogue can include confirmations sent to the user to ensure they wish to switch threads.

[0047] In an exemplary embodiment, a computer-implemented method includes: receiving natural language utterances from a user during a session between a user and a digital assistant at a digital assistant; obtaining a topic context instance for the natural language utterances; generating a list of executable actions by a generative artificial intelligence (GenAI) model based on one or more candidate actions associated with the topic context instance; and executing the executable actions to produce output, wherein the execution includes: determining that no thread is running within the session associated with the topic context instance, the executable action, or both; in response to determining that no thread is running, creating a thread associated with the topic context instance, the executable action, or both, wherein the thread maintains the topic context instance, the executable action, or both as a context separate from any other conversation for a distinctly different conversation including the natural language utterances between the user and the digital assistant; and using the thread to execute the executable actions to obtain output; and sending the output or communications derived from the output to the user.

[0048] As used herein, the articles “a” and “an” are used to refer to one or more (i.e., at least one) grammatical object of the article. For example, an element means at least one element and may include more than one element.

[0049] As used herein, the terms “approximately,” “similarly,” “substantially,” and “approximately” are defined as substantially, but not necessarily, fully specified (and include fully specified), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms “approximately,” “similarly,” “substantially,” or “approximately” may be replaced by “within [a certain percentage] of the specified value,” where percentages include 0.1%, 1%, 5%, and 10%, etc. Moreover, the terms “approximately,” “similarly,” “substantially,” and “approximately” are used to provide flexibility for the endpoints of a numerical range by specifying that a given value can be slightly above or slightly below the endpoint without affecting the desired result.

[0050] As used in this article, when an action is “based on” something, it means that the action can be based at least partially on at least a portion of said thing.

[0051] The use of the terms “comprising,” “including,” or “having,” and variations thereof herein is intended to cover the elements listed thereafter and their equivalents, as well as any additional elements. Embodiments described as including, containing, or having certain elements are also considered to be substantially composed of and comprised of those elements. As used herein, “and / or” means and covers any and all possible combinations of one or more associated listed items, and, in alternative cases (“or”), the absence of such combinations.

[0052] Robotics and analytics systems (skills tracking)

[0053] A bot (also known as a skill, chatbot, chattterbot, or talkbot) is a computer program that can converse with an end user. Bots typically respond to natural language messages (e.g., asking questions or commenting) through messaging applications that use natural language messaging. Businesses can use one or more bot systems to communicate with end users via messaging applications. The messaging application (which may be referred to as a channel) can be a messaging application that the end user already has installed and is familiar with, and that is the end user's preferred messaging application. Therefore, the end user does not need to download and install a new application to chat with the bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), mobile and web application extensions that leverage chat capabilities to extend native or hybrid / responsive mobile or web applications, or voice-based input (such as devices or applications with interfaces that allow interaction using Siri, Cortana, Google Voice, or other voice input).

[0054] In some examples, a robot system may be associated with a Uniform Resource Identifier (URI). A URI can be used as a string to identify the robot system. A URI can be used as a webhook for one or more messaging applications. A URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The robot system may be designed to receive messages from messaging applications (e.g., Hypertext Transfer Protocol (HTTP) POST request messages). HTTP POST request messages can be directed from the messaging application to the URI. In some embodiments, the message may differ from an HTTP POST request message. For example, the robot system may receive messages from a Short Message Service (SMS). While the discussion herein may refer to communication received by the robot system as a message, it should be understood that a message can be an HTTP POST request message, an SMS message, or any other type of communication between the two systems.

[0055] End users can interact with robotic systems through conversational interactions (sometimes called conversational user interface (UI)), much like human-to-human interactions. In some cases, the interaction may involve the end user saying "hello" to the robot, and the robot responding with "hi" and asking how it can help the end user. In other cases, the interaction may be transactional with, for example, a banking robot, such as transferring funds from one account to another; informational with, for example, an HR robot, such as checking holiday balances; or with, for example, a retail robot, such as discussing returning purchased goods or seeking technical support.

[0056] In some embodiments, a bot system can intelligently handle end-user interactions without interacting with the bot system's administrator or developers. For example, an end-user can send one or more messages to the bot system to achieve a desired goal. Messages may include content such as text, emojis, audio, images, video, or other methods of conveying the message. In some embodiments, the bot system can convert the content into a standardized form (e.g., a Representational State Transition (REST) ​​call to an enterprise service with the correct parameters) and generate a natural language response. The bot system may also prompt the end-user for additional parameters or request additional information. In some embodiments, the bot system may also initiate communication with the end-user rather than passively responding to end-user utterances. This document describes various techniques for identifying explicit calls to a bot system and determining the input of the called bot system. In some embodiments, explicit call analysis is performed by the master bot based on detecting the call name in the utterance. In response to the detection of the call name, the utterance can be refined to input to the skill bot associated with that call name.

[0057] Conversations with a chatbot can follow a specific conversation flow that includes multiple states. The flow can be defined based on input, indicating what will happen next. In some embodiments, a state machine that includes user-defined states (e.g., end-user intentions) and actions to be taken within or from one state to another can be used to implement the chatbot system. Based on end-user input, the conversation can take different paths, which affects the decisions the chatbot makes regarding the flow. For example, in each state, based on end-user input or utterances, the chatbot can determine the end-user's intention in order to determine the appropriate next action to take. As used herein and in the context of utterances, the term "intention" refers to the intention of the user providing the utterance. For example, a user might intend to have a conversation with the chatbot about ordering pizza, so the user's intention could be represented by the utterance "order pizza." User intentions can be directed to specific tasks that the user wants the chatbot to perform on their behalf. Therefore, utterances can be worded as questions, commands, requests, etc., that reflect the user's intention. Intentions can include goals that the end-user wishes to accomplish.

[0058] In the context of chatbot configuration, the term "intent" is used herein to refer to configuration information used to map user utterances to specific tasks / actions or categories of tasks / actions that the chatbot can perform. To distinguish the intent of a utterance (i.e., user intent) from the intent of the chatbot, the latter is sometimes referred to herein as "bot intent." A bot intent can include one or more utterances associated with that intent. For example, the intent to order pizza can be conveyed through various permutations of utterances expressing the desire to order pizza. These associated utterances can be used to train the chatbot's intent classifier so that the intent classifier can subsequently determine whether an input utterance from the user matches the pizza-ordering intent. A bot intent can be associated with one or more dialogue flows used to initiate a conversation with the user in a specific state. For example, the first message for an pizza-ordering intent could be the question "What kind of pizza do you want?". In addition to associated utterances, a bot intent can also include named entities associated with that intent. For example, an pizza-ordering intent can include variables or parameters used to perform the pizza-ordering task, such as toppings 1, toppings 2, pizza type, pizza size, number of pizzas, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0059] Figure 1 This is a simplified block diagram of an environment 100 incorporating a chatbot system according to some embodiments. Environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistant or chatbot systems. DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as... Figure 1 As shown, user 104 of a specific enterprise can use DABP 102 to create and deploy digital assistants 106 for that specific enterprise's users. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform can be used by multiple enterprises to create digital assistants. As another example, the owner of a restaurant (e.g., a pizzeria) can use DABP 102 to create and deploy digital assistants that enable the restaurant's customers to order food (e.g., order pizza).

[0060] For the purposes of this disclosure, a "digital assistant" is an entity that assists its user in performing various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. Digital assistants can be implemented or realized in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Accordingly, for the purposes of this disclosure, the terms "digital assistant" and "chatbot system" are used interchangeably.

[0061] Digital assistants, such as digital assistant 106 built using DABP 102, can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and retrieve a response 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and responses 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested task and respond to the user with an appropriate response.

[0062] User input 110 is generally in the form of natural language and is referred to as speech. User speech 110 can be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user speech 110 can be in the form of audio input or speech, such as when a user speaks or utters content that is provided as input to the digital assistant 106. Speech is typically in the language spoken by user 108. For example, speech can be English or some other language. When speech is in speech form, the speech input is converted into speech in text form in that particular language, and the text speech is then processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert speech or audio input into text speech, which is then processed by the digital assistant 106. In some embodiments, speech-to-text conversion can be performed by the digital assistant 106 itself.

[0063] Discourse (which can be text discourse or spoken discourse) can be fragments, sentences, multiple sentences, one or more words, one or more questions, combinations of the types mentioned above, etc. Digital Assistant 106 can be configured to apply Natural Language Understanding (NLU) technology to discourse to understand the meaning of user input. As part of NLU processing for discourse, Digital Assistant 106 can be configured to perform processing to understand the meaning of discourse, which involves identifying one or more intentions and one or more entities corresponding to the discourse. After understanding the meaning of the discourse, Digital Assistant 106 can perform one or more actions or operations in response to the understood meaning or intention. For the purposes of this disclosure, it is assumed that the discourse is either text discourse directly provided by user 108 of Digital Assistant 106 or the result of converting input spoken discourse into text form. However, this is not intended to limit or constrain in any way.

[0064] For example, user 108 can request to order pizza by providing a statement such as "I want to order pizza." Upon receiving such a statement, digital assistant 106 is configured to understand the meaning of the statement and take an appropriate action. The appropriate action may involve responding to the user's input, for example, by asking questions about the type of pizza the user wishes to order, the size of the pizza, any toppings, etc. The responses provided by digital assistant 106 can also be in natural language and typically use the same language as the input statement. As part of generating these responses, digital assistant 106 can perform natural language generation (NLG). For a user ordering pizza, through conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the necessary information for the pizza order, and then allow the pizza to be ordered at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0065] At the conceptual level, the digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a sequence or pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as Natural Language Understanding (NLU)), determining the action to be performed in response to the utterance, causing the action to be performed where appropriate, generating a response to be output to the user in response to the user's utterance, and outputting the response to the user, etc. NLU processing may include parsing the received input utterance to understand the structure and meaning of the utterance, refining and reorganizing the utterance to develop a form (e.g., logical form) or structure for better understanding of the utterance. Generating a response may include using NLG techniques.

[0066] NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related processes, such as sentence parsing (e.g., lexicalization, lemmatization, identifying part-of-speech tags for sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing individual clauses, parsing anaphora, performing chunking, etc.). In some embodiments, NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of NLU processing. For example, the syntax and structure of an input utterance sentence can be identified by processing the sentence using a parser, a part-of-speech tagger, and / or a named entity discriminator. In one implementation, for the English language, parsers, part-of-speech taggers, and named entity discriminators, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0067] While the various examples provided in this disclosure illustrate utterances in the English language, this is merely intended as an example. In some embodiments, the digital assistant 106 is also capable of handling utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes NLU processing flexible and scalable for each language, including allowing for different processing orders. Language packs may be provided for various languages, wherein the language packs may register a list of subsystems that can be served from the NLU core server.

[0068] It is possible to provide or allow its users 108 to access digital assistants (such as...) through various channels (such as, but not limited to, certain applications, social media platforms, various messaging services and applications, and other applications or channels). Figure 1 The digital assistant 106 is depicted in the image. Several channels can be configured for a single digital assistant, allowing it to run on and be accessed by different services simultaneously.

[0069] Digital assistant or chatbot systems typically contain one or more skills or are associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill-based bots) configured to interact with users and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc. For example, for Figure 1In the embodiments depicted, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the terms "skill" and "various skills" are used synonymously with the terms "skilled robot" and "various skilled robots," respectively.

[0070] Each skill associated with the digital assistant helps the user complete tasks through conversation, which may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may take the form of text or audio messages given to the user, and / or use simple user interface elements (e.g., a selection list) presented to the user for selection.

[0071] There are various ways to associate or add skills or skill bots to a digital assistant. In some instances, businesses can develop skill bots and then add them to a digital assistant using DABP 102. In other instances, skill bots can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. Still others, DABP 102 provides an online digital store (referred to as the "Skill Store") offering a variety of skills for a wide range of tasks. Skills offered through the Skill Store can also expose various cloud services. To add a skill to a digital assistant created using DABP 102, a DABP 102 user can access the Skill Store via DABP 102, select the desired skill, and indicate which skill to add to the digital assistant created using DABP 102. Skills from the Skill Store can be added to the digital assistant as is or in a modified form (for example, a DABP 102 user can select and clone a specific skill bot offered by the Skill Store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0072] Various architectures can be used to implement digital assistant or chatbot systems. For example, in some embodiments, digital assistants created and deployed using DABP 102 can be implemented using a master-bot / sub (or slave) bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more sub-bots that act as skill-based bots. For example, in... Figure 1 In the embodiments depicted, the digital assistant 106 includes a main robot 114 and skill robots 116-1, 116-2, etc., which are sub-robots of the main robot 114. In some embodiments, the digital assistant 106 itself is considered to act as the main robot.

[0073] Digital assistants implemented using a master-slave robot architecture enable users to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user interacts with the digital assistant, user input is received by the master robot. The master robot then processes the input to determine its meaning. It then determines whether the task requested in the user's utterance can be handled by the master robot itself; otherwise, it selects the appropriate skill robot to handle the user's request and routes the conversation to that selected skill robot. This allows users to interact with the digital assistant through a shared, single interface while still providing the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot could interface with skill robots with specific functionalities, such as a CRM robot for performing functions related to Customer Relationship Management (CRM), an ERP robot for performing functions related to Enterprise Resource Planning (ERP), an HCM robot for performing functions related to Human Capital Management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access it through a shared master robot interface, while multiple skill robots are available behind the scenes to handle user requests.

[0074] In some embodiments, in a master robot / sub-robot infrastructure, a master robot is configured to be aware of a list of available skill robots. The master robot can access metadata identifying the various available skill robots, and for each skill robot, the master robot can access the capabilities of that skill robot, including tasks that the skill robot can perform. Upon receiving a user request in the form of a utterance, the master robot is configured to identify or predict from among the multiple available skill robots that a particular skill robot can best serve or handle the user request. The master robot then routes the utterance (or a portion of the utterance) to that particular skill robot for further handling. Control thus flows from the master robot to the skill robot. The master robot can support multiple input and output channels. In some embodiments, routing can be performed using processing performed by one or more available skill robots. For example, as discussed below, a skill robot can be trained to infer the intent of a utterance and determine whether the inferred intent matches an intent configured on the skill robot. Therefore, routing performed by the master robot can involve the skill robot conveying to the master robot an indication of whether it is configured with an intent suitable for handling the utterance.

[0075] Although Figure 1The embodiments illustrated in the text depict a digital assistant 106 including a master robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations using a combination of software and hardware.

[0076] DABP 102 provides the infrastructure, services, and features that enable DABP 102 users to create digital assistants that include one or more skill bots associated with the digital assistant. In some instances, skill bots can be created by cloning existing skill bots, such as those offered by a skill store. As previously indicated, DABP 102 provides a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP 102 users can clone skill bots from the skill store. Cloned skill bots can be modified or customized as needed. In other instances, DABP 102 users create skill bots from scratch using the tools and services provided by DABP 102. As previously indicated, a skill store or skill catalog provided by DABP 102 can offer multiple skill bots for performing various tasks.

[0077] In some embodiments, creating or customizing a skill robot at a higher level involves the following steps:

[0078] (1) Configure settings for new skill robots

[0079] (2) Configure one or more intentions for the skill robot

[0080] (3) Configure one or more entities for one or more intents.

[0081] (4) Training skills robots

[0082] (5) Create dialogue flow for skill robots

[0083] (6) Add custom components to the skill robot as needed.

[0084] (7) Testing and deploying skills robots

[0085] Each of the above steps is briefly described below.

[0086] (1) Configure settings for new skill robots – Various settings can be configured for skill robots. For example, skill robot designers can specify one or more invocation names for the skill robot being created. These invocation names can then be used by the digital assistant user to explicitly invoke the skill robot. For example, the user can enter the invocation name in their speech to explicitly invoke the corresponding skill robot.

[0087] (2) Configuring one or more intents and associated example utterances for the skill robot – The skill robot designer assigns one or more intents (also called robot intents) to the skill robot being created. The skill robot is then trained based on these assigned intents. These intents represent the categories or classes in which the skill robot is trained to infer input utterances. Upon receiving a utterance, the trained skill robot infers the intent of that utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then responds to the utterance with an appropriate action based on the intent inferred for that utterance. In some instances, the intents of the skill robot represent tasks that the skill robot can perform for a digital assistant user. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents assigned to the skill robot could include “CheckBalance”, “TransferMoney”, “DepositCheck”, etc.

[0088] For each intent defined for a skill bot, the skill bot designer can also provide one or more example statements that represent and illustrate that intent. These example statements are intended to represent the statements a user would input into the skill bot for that intent. For example, for the CheckBalance intent, example statements could include “What is my savings account balance?”, “How much money is in my checking account?”, “How much money is in my account?”, etc. Accordingly, various substitutions of typical user statements can be specified as example statements for the intent.

[0089] Intents and associated example utterances are used as training data to train the skill robot. Various training techniques can be used. This training generates a predictive model, which is configured to take utterances as input and output the intent inferred by the predictive model for those utterances. In some instances, the input utterance is provided to an intent analysis engine, which is configured to use the trained model to predict or infer the intent of the input utterance. The skill robot can then take one or more actions based on the inferred intent.

[0090] (3) Configuring Entities for One or More Intents of a Skill Robot – In some instances, additional context may be required for the skill robot to respond appropriately to user utterances. For example, there may be cases where user input utterances are resolved to the same intent in the skill robot. For example, in the example above, the utterances “What is my savings account balance?” and “How much money do I have in my checking account?” are both resolved to the same CheckBalance intent, but these utterances are different requests asking for different things. To clarify such requests, one or more entities are added to the intent. Using the banking skill robot example, an entity called AccountType, which defines values ​​called “checking” and “saving”, allows the skill robot to resolve user requests and respond appropriately. In the example above, although the utterances are resolved to the same intent, the values ​​associated with the AccountType entity are different for the two utterances. This allows the skill robot to perform potentially different actions for the two utterances, even though they are resolved to the same intent. One or more entities can be specified for certain intents configured for the skill robot. Entities are thus used to add context to the intent itself. Entities help to describe the intent more comprehensively and enable the skill robot to fulfill the user request.

[0091] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, cycle time, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for banking skills, the skill bot designer can define an AccountType entity that enables various banking transactions by examining user input for keywords such as checks, savings, and credit cards.

[0092] (4) Training the Skill Robot – The skill robot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select an intent associated with the received user input. As indicated above, the skill robot must be trained for this purpose. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example utterances associated with those intents (collectively, training data) so that the skill robot can parse user input utterances as one of its configured intents. In some embodiments, the skill robot uses a predictive model that is trained using the training data and allows the skill robot to discern what the user said (or, in some cases, what they were trying to say). DABP 102 provides a variety of different training techniques that can be used by skill robot designers to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion (e.g., 80%) of the training data is used to train the skill robot model, while another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as the trained skill robot) can then be used to process and respond to user utterances. In some cases, the user's utterances may be questions that only require a single answer and do not require further conversation. To handle such situations, Q&A (question and answer) intents can be defined for skill bots. This allows skill bots to output responses to user requests without having to update the dialogue definition. Q&A intents are created in a similar manner to regular intents. The dialogue flow of a Q&A intent can differ from that of a regular intent.

[0093] (5) Creating a Dialogue Flow for a Skilled Bot – The dialogue flow specified for a skilled bot describes how the bot reacts as it is parsed and interpreted in response to different intentions of the bot in relation to received user input. The dialogue flow defines the actions or behaviors the bot will take, such as how it responds to user utterances, prompts for user input, or returns data. The dialogue flow is like a flowchart followed by the bot. Skilled bot designers use languages ​​such as Markdown to specify the dialogue flow. In some embodiments, a version of YAML called OBotML can be used to specify the dialogue flow for the bot. The dialogue flow definition for the bot acts as a model of the conversation itself, allowing the bot designer to orchestrate the interactions between the bot and the users for whom it provides services.

[0094] In some embodiments, the dialogue flow definition for a skill-based robot comprises three sections:

[0095] (a) Context section

[0096] (b) Default transition section

[0097] (c) Status segment

[0098] Context Sections – Skill bot designers can define variables used in the conversation flow within context sections. Other variables that can be named in context sections include, but are not limited to: variables for error handling, variables for built-in or custom entities, and user variables that enable the skill bot to identify and persist user preferences.

[0099] Default Transition Section – Transitions for skill bots can be defined in the dialogue flow state section or in the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when no suitable transition is defined within the state or when the conditions required to trigger a state transition are not met. The default transition section can be used to define routes that allow skill bots to gracefully handle unexpected user actions.

[0100] State segments – A dialogue flow and its associated operations are defined as a series of transient states that manage the logic within the dialogue flow. Each state node within the dialogue flow definition names the component that provides the required functionality at that point in the dialogue. Therefore, states are built around components. States contain component-specific properties and define transitions to other states that are triggered after a component executes.

[0101] Special scenarios can be handled using state segments. For example, sometimes you might want to offer users the option to temporarily leave the first skill they are engaged in to do something in a second skill within the digital assistant. For instance, if a user is engaged in a conversation with the shopping skill (e.g., the user has made some purchase choices), then the user might want to jump to the banking skill (e.g., the user might want to ensure he / she has enough money to make the purchase), and then return to the shopping skill to complete the user's order. To address this, the action in the first skill can be configured to initiate an interaction with a second, different skill within the same digital assistant, and then return to the original flow.

[0102] (6) Adding Custom Components to the Skill Robot – As described above, the states specified in the dialogue flow for the skill robot are named to provide the required functionality corresponding to those states. Components enable the skill robot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured components for performing a wide range of functions. The skill robot designer can select one or more of these pre-configured components and associate them with states in the dialogue flow for the skill robot. The skill robot designer can also use the tools provided by DABP 102 to create custom or new components and associate custom components with one or more states in the dialogue flow for the skill robot.

[0103] (7) Testing and Deploying Skilled Robots – DABP 102 provides several features that enable skilled robot designers to test the features of the skilled robots being developed. The skilled robots can then be deployed and included in digital assistants.

[0104] While the above description illustrates how to create skill robots, similar techniques can be used to create digital assistants (or master robots). At the master robot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the master robot) can handle without invoking the skill robots associated with the digital assistant. Examples of system intents defined for the master robot include: (1) Exit: applicable when the user signals that they wish to exit the current conversation or context within the digital assistant; (2) Help: applicable when the user requests help or orientation; and (3) UnresolvedIntent: applicable to user input that doesn't quite match the exit and help intents. The digital assistant also stores information about one or more skill robots associated with it. This information allows the master robot to select a specific skill robot to handle the utterance.

[0105] At the master chatbot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant uses a routing model that can be rule-based, AI-based, or a combination thereof to determine this. The routing model determines whether the conversation corresponding to the user's input utterance should be routed to a specific skill for handling, handled by the digital assistant or master chatbot itself according to built-in system intent, or handled as a different state within the current conversation flow.

[0106] In some embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If an invocation name is present in the user input, it is considered an explicit invocation of the skill bot corresponding to that invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. If there is no specific or explicit invocation, in some embodiments, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and the skill bot associated with the digital assistant. The score calculated for the skill bot or system intent represents the probability that the user input represents a task that the skill bot is configured to perform, or the probability that it represents a system intent. Any system intent or skill bot with an associated calculated confidence score exceeding a threshold value (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In some embodiments, after identifying one or more skill bots as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill), and a confidence score is determined for each intent. Generally, any intent with a confidence score exceeding a threshold (e.g., 70%) is considered a candidate intent. If a specific skill bot is selected, the user's utterance is routed to that skill bot for further processing. If a system intent is selected, the main bot itself performs one or more actions based on the selected system intent.

[0107] Figure 2 This is a simplified block diagram of a master robot (MB) system 200 according to some embodiments. The MB system 200 may be implemented in software only, in hardware only, or in a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, a multi-intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data repository 250. Figure 2 The MB system 200 depicted is merely an example of the arrangement of components in the main robot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have more than Figure 2 The systems or components shown may include more or fewer systems or components, may combine two or more subsystems, or may have different configurations or arrangements of subsystems.

[0108] The preprocessing subsystem 210 receives the utterance “A” 202 from the user and processes the utterance through a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation marks. For example, if the utterance 202 is provided as audio, the preprocessing subsystem 210 can use a speech-to-text converter (not shown) that inserts punctuation marks (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio into text.

[0109] The language detector 212 detects the language of discourse 202 based on the text of discourse 202. The way discourse 202 is processed depends on the language, because each language has its own syntax and semantics. Differences between languages ​​must be considered when analyzing the syntax and structure of discourse.

[0110] Language parser 214 parses utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. Language parser 214 can also lexically represent the linguistic units of utterance 202 (e.g., converting each word into a separate lexical) and perform lemmatization on the words. A lexical is the primary form in which a set of words are represented in a dictionary (e.g., "run" is a lexical of run, runs, ran, running, etc.). Other types of preprocessing that language parser 214 can perform include chunking of compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". Language parser 214 can also identify relationships between words in utterance 202. For example, in some embodiments, language parser 214 generates a dependency tree indicating which parts of the utterance (e.g., specific nouns) are direct objects, which parts of the utterance are prepositions, etc. The result of the processing performed by the language parser 214 forms the extracted information 205, which, together with the utterance 202 itself, is provided as input to the MIS 220.

[0111] As indicated above, utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, even if utterance 202 comprises multiple sentences, it can be treated as a single unit. However, in some embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210 to identify a single sentence from multiple sentences for multi-intent analysis and explicit call analysis. Generally, the results produced by MIS 220 and EIS 230 are substantially the same regardless of whether utterance 202 is processed as a hierarchy of single sentences or as a single unit comprising multiple sentences.

[0112] MIS 220 determines whether utterance 202 represents multiple intentions. While MIS 220 can detect the presence of multiple intentions in utterance 202, the processing performed by MIS 220 does not involve determining whether the intention of utterance 202 matches any intention already configured for the robot. Instead, the processing to determine whether the intention of utterance 202 matches robot intentions can be performed by the intention classifier 242 of MB system 200 or by the intention classifier of the skill robot (e.g., ...). Figure 3 (As shown in the embodiment). The processing performed by MIS 220 assumes the existence of a bot that can handle utterance 202 (e.g., a specific skill bot or the main bot itself). Therefore, the processing performed by MIS 220 does not require knowledge of which bots exist in the chatbot system (e.g., the identity of skill bots registered with the main bot) or knowledge of which intents have been configured for a particular bot.

[0113] To determine that utterance 202 includes multiple intentions, MIS 220 applies one or more rules from rule set 252 in data repository 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include sentence patterns indicating the presence of multiple intentions. For example, a sentence pattern may include a coordinating conjunction connecting two parts of a sentence (e.g., parallel items), where both parts correspond to separate intentions. If utterance 202 matches this sentence pattern, then it can be inferred that utterance 202 represents multiple intentions. It should be noted that utterances with multiple intentions do not necessarily have different intentions (e.g., intentions pointing to different robots or different intentions within the same robot). Instead, utterances may have separate instances of the same intention, such as “order pizza using payment account X, then order pizza using payment account Y”.

[0114] As part of determining that discourse 202 represents multiple intentions, MIS 220 also determines which parts of discourse 202 are associated with each intention. MIS 220 constructs a new discourse for each intention represented in a discourse containing multiple intentions, so as to process it separately in place of the original discourse, for example, regarding... Figure 2The two utterances are utterance “B” 206 and utterance “C” 208. Therefore, the original utterance 202 can be broken down into two or more individual utterances, processed one at a time. MIS 220 uses the extracted information 205 and / or analyzes utterance 202 itself to determine which of the two or more utterances should be processed first. For example, MIS 220 may determine that utterance 202 contains a marked word indicating a specific intention that should be processed first. The newly formed utterance corresponding to this specific intention (e.g., one of utterances 206 or 208) will be the first utterance sent for further processing by EIS 230. After the conversation triggered by the first utterance has ended (or has been temporarily suspended), the next highest priority utterance (e.g., another of utterances 206 or 208) can then be sent to EIS 230 for processing.

[0115] EIS 230 determines whether a received utterance (e.g., utterance 206 or utterance 208) contains a call name for a skill bot. In some embodiments, a unique call name is assigned to each skill bot in the chatbot system, distinguishing that skill bot from other skill bots in the chatbot system. A list of call names may be maintained as part of skill bot information 254 in a data repository 250. When a utterance contains words that match a call name, the utterance is considered an explicit call. If no bot is explicitly called, the utterance received by EIS 230 is considered a non-explicit call utterance 234 and is fed into the master bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to handle the utterance. In some instances, intent classifier 242 will determine that the master bot should handle non-explicit call utterances. In other instances, intent classifier 242 will determine the skill bot to route the utterance for handling.

[0116] The explicit invocation functionality provided by EIS 230 has several advantages. It reduces the amount of processing the master robot must perform. For example, when explicit invocation is present, the master robot may not need to perform any intent classification analysis (e.g., using intent classifier 242), or may only need to perform a simplified intent classification analysis to select a skill robot. Therefore, explicit invocation analysis enables the selection of a specific skill robot without resorting to intent classification analysis.

[0117] Furthermore, there can be situations where the functionality of multiple skill robots overlaps. This can occur, for example, if the intentions handled by two skill robots overlap or are very similar to each other. In such cases, the master robot may struggle to identify which of the multiple skill robots to select based solely on intention classification analysis. In these scenarios, explicit invocation eliminates ambiguity regarding the specific skill robot to use.

[0118] In addition to determining that a utterance is an explicit invocation, EIS 230 is also responsible for determining whether any part of the utterance should be used as input to the explicitly invoked skill robot. Specifically, EIS 230 can determine whether a portion of the utterance is not associated with the invocation. EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. EIS 230 can send the portion of the utterance not associated with the invocation to the invoking skill robot instead of sending the entire utterance received by EIS 230. In some instances, the input to the invoked skill robot is formed by simply removing any portion of the utterance associated with the invocation. For example, “I want to order pizza using the pizza robot” can be shortened to “I want to order pizza” because “using the pizza robot” is associated with invoking the pizza robot but not with any processing the pizza robot is to perform. In some instances, EIS 230 can reformat the portion to be sent to the invoked robot, for example, to form a complete sentence. Therefore, EIS 230 not only determines that an explicit invocation exists but also determines what to send to the skill robot when an explicit invocation exists. In some instances, no text input may be sent to the invoked robot. For example, if the utterance is "pizza robot," then EIS 230 can determine that a pizza robot is being invoked, but there is no text to be processed by the pizza robot. In such scenarios, EIS 230 can indicate to the skill robot invoker 240 that nothing needs to be sent.

[0119] Skill robot invoker 240 invokes skill robots in various ways. For example, skill robot invoker 240 may invoke a robot in response to receiving instruction 235 indicating that a specific skill robot has been selected due to explicit invocation. Instruction 235 may be sent by EIS 230 along with input to the explicitly invoked skill robot. In this scenario, skill robot invoker 240 will transfer control of the conversation to the explicitly invoked skill robot. The explicitly invoked skill robot will determine the appropriate response to the input from EIS 230 by treating the input from EIS 230 as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, where the initial state of the new conversation depends on the input sent from EIS 230.

[0120] Another way the skill robot invoker 240 can invoke a skill robot is through implicit invocation using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that a utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different classes, one class per skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. The parameters resulting from this training (e.g., a set of values ​​for the parameters of a machine learning model) can be stored as part of the skill robot information 254.

[0121] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described further in detail herein. Training the machine learning model may involve inputting at least a subset of example utterances associated with various skill robots to generate an inference, as the output of the machine learning model, about which robot is the correct robot to use for any particular training utterance. For each training utterance, an indication of the correct robot to be used for that training utterance can be provided as basic fact information. The behavior of the machine learning model can then be adapted (e.g., via backpropagation) to minimize the discrepancy between the generated inference and the basic fact information.

[0122] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot. This confidence score indicates the likelihood that the skill robot can handle a utterance (e.g., a non-explicit invocation utterance 234 received from EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score meets one or more conditions, the skill robot invoker 240 will invoke the robot associated with that particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is either an identifier of a system intent or an identifier of a particular skill robot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must also exceed the second-highest confidence score by a certain margin. Imposing this condition will enable routing to a specific skill robot when the confidence scores of multiple skill robots each exceed the threshold confidence score value.

[0123] After identifying a robot based on an assessment of its confidence score, the skill robot invoker 240 hands over processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is a skill robot. Additionally, the skill robot invoker 240 determines what content to provide as input 247 to the identified robot. As indicated above, in the case of explicit invocation, input 247 may be based on a portion of the utterance not associated with the invocation, or input 247 may be empty (e.g., an empty string). In the case of implicit invocation, input 247 may be the entire utterance.

[0124] Data repository 250 includes one or more computing devices storing data used by various subsystems of the master bot system 200. As explained above, data repository 250 includes rules 252 and skill bot information 254. Rule 252 includes, for example, rules for determining, by MIS 220, when a utterance represents multiple intents and how to break down utterances representing multiple intents. Rule 252 also includes rules for determining, by EIS 230, which parts of a utterance that explicitly invokes a skill bot should be sent to that skill bot. Skill bot information 254 includes the invocation names of skill bots in the chatbot system, for example, a list of invocation names of all skill bots registered with a particular master bot. Skill bot information 254 may also include information used by intent classifier 242 to determine the confidence score of each skill bot in the chatbot system, for example, parameters of a machine learning model.

[0125] Figure 3 This is a simplified block diagram of a skill robot system 300 according to some embodiments. The skill robot system 300 is a computing system that can be implemented solely in software, solely in hardware, or in a combination of hardware and software. In, for example... Figure 1 In some embodiments, such as the one depicted, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0126] The skill robot system 300 includes a MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to... Figure 2 MIS 220 provides similar functionality, including operability to determine, using rule 352 in data repository 350: (1) whether a utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by MIS 310 to detect multiple intents and split utterances are the same as those applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to... Figure 1 The extracted information 205 can be generated using the language parser 214 or the local language parser of the skill robot system 300.

[0127] The intent classifier 320 can be combined with the above. Figure 2 The intent classifier 242 is trained in a manner similar to that discussed in the embodiments and described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a specific skill robot, using at least a subset of example utterances associated with that specific skill robot as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with that training utterance.

[0128] Message 302 can be received directly from the user or supplied through the main robot. When message 302 is supplied through the main robot, for example, as... Figure 2 The results of processing performed by MIS 220 and EIS 230 in the embodiments depicted can bypass MIS 310 to avoid duplication of processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a conversation after routing to a skill bot, then MIS 310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If so, then MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, for example, utterance “D” 306 and utterance “E” 308. If utterance 302 does not represent multiple intents, then MIS 310 forwards utterance 302 to intent classifier 320 for intent classification and does not split utterance 302.

[0129] Intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) to intents associated with the skill robot system 300. As explained above, the skill robot can be configured with one or more intents, each intent including at least one example utterance associated with that intent and used to train the classifier. Figure 2In one embodiment, the intent classifier 242 of the main robot system 200 is trained to determine the confidence scores of individual skill robots and the system intent. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and therefore more granular. The intent classifier 320 can access intent information 354. The intent information 354 includes, for each intent associated with the skill robot system 300, a list of utterances that represent and explain the meaning of the intent and are generally associated with a task that can be performed by the intent. The intent information 354 may also include parameters generated as a result of training on this list of utterances.

[0130] The conversation manager 330 receives a specific intent instruction 322 as the output of the intent classifier 320, which identifies the specific intent as the best match for the utterance input to the intent classifier 320. In some instances, the intent classifier 320 cannot determine any match. For example, if the utterance is directed to a system intent or an intent of a different skill robot, the confidence score calculated by the intent classifier 320 may drop below a threshold confidence score. When this occurs, the skill robot system 300 may submit the utterance to the master robot for processing, such as routing it to a different skill robot. However, if the intent classifier 320 successfully identifies an intent within a skill robot, the conversation manager 330 will initiate a conversation with the user.

[0131] The conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 can be implemented using a state machine configured to execute a dialogue flow in response to the identified intent. The state machine may include a default starting state (e.g., for invoking the intent when no additional input is provided) and one or more additional states, each with an associated action to be performed by the skill robot (e.g., performing a purchase transaction) and / or a dialogue to be presented to the user (e.g., a question, a response). Thus, the conversation manager 330 can determine an action / dialogue 335 upon receiving an instruction 322 to identify the intent, and can determine additional actions or dialogues in response to subsequent utterances received during the conversation.

[0132] The data repository 350 includes one or more computing devices for storing data used by the various subsystems of the skill robot system 300. About Figure 3 The data repository 350 includes rules 352 and intent information 354. In some embodiments, the data repository 350 may be integrated into the data repository of the master robot or digital assistant (e.g., Figure 2 The data is stored in repository 250.

[0133] Digital assistants that utilize large language model blocks for processing (with LLM-supported skill tracking)

[0134] Figure 4 The diagram illustrates the use of large language models (LLMs) with digital assistants (e.g., regarding...). Figure 1-3 The example system 400 describes the integration of a Data Automation (DA) system to enhance skills with generative AI capabilities. These capabilities include handling casual conversation with users, generating written summaries of data, automating challenging or repetitive business tasks (such as those required for talent acquisition), and providing sentiment analysis of a given text fragment to determine whether it reflects positive, negative, or neutral opinions. Skill robot developers can insert these capabilities into any part of the conversation flow where they need them by invoking a Large Language Model (LLM) component. This conversation flow component is the primary integrator of generative AI because it contacts the LLM hosted on one or more computing platforms (such as cloud platforms, on-premises platforms, edge platforms, etc.) via invocation (e.g., REST calls), then sends prompts (natural language instructions to the LLM) along with relevant parameters. It then returns the results generated by the model (also known as completion) and manages the state of the LLM-user interaction so that its responses remain in context after successive rounds of user queries and feedback. The LLM component can invoke any LLM. Users can add one or more LLM component states (or LLM blocks) to the flow. Users can also chain LLM calls so that the output of one LLM request can be passed to subsequent LLM requests.

[0135] System 400 includes a cloud computing platform 405 (e.g., such as regarding...) Figure 13-17The Oracle Cloud Infrastructure (IaaS platform) and other computing platforms 410 configured to provide services include the ability to interact with a digital assistant enhanced by an LLM component 415, which invokes an LLM 415 hosted by one or more LLM service providers. In some embodiments, a cloud computing platform 405 hosts an LLM 415 for one or more LLM service providers. In other embodiments, one or more other computing platforms 410 (such as one or more other cloud platforms, one or more on-premises platforms, one or more edge platforms, etc.) host an LLM 415 for one or more LLM service providers. In other embodiments, a cloud computing platform 405 hosts some of the LLM 415 for some of the one or more LLM service providers, and one or more other computing platforms 410 host some of the LLM 415 for some of the one or more LLM service providers. The LLM 415 includes a collection of multiple LLMs (e.g., GPT-4, LaMDA, etc.) that can be used to handle various generative artificial intelligence tasks (e.g., natural language understanding and processing) to provide efficient, user-centric, and context-aware responses to one or more users.

[0136] In addition to the LLM components, other key components of LLM integration include endpoints for one or more LLM service providers, and transformation handlers for converting request and response payloads between digital assistant formats using the Common LLM Interface (CLMI) (also described in this document as GenAI Interface 420). The following are the advanced steps for adding these and other components to create LLM integration for your skills:

[0137] Register API services for LLM endpoints (e.g., REST endpoints) in the digital assistant instance.

[0138] For the skill, create an LLM transformation event handler to use the GenAI interface 420 to transform LLM request and response payloads between CLMIs.

[0139] In some instances, pre-built processors are provided, for example, if a user integrates their skills with a Cohere model or Oracle Generative AI services. If they are accessing other models, such as Azure OpenAI, then the user can update the starting transformation code provided using declarative processing.

[0140] Define an LLM service for the skill, which maps to the service that the user has registered with the instance using the LLM transformation handler.

[0141] In the flow of skills that users want to use LLM, LLM components are inserted and configured by adding prompt text and setting other parameters.

[0142] You can use the hint builder (accessible via LLM components) to add hint text and / or templates to refine and test hints.

[0143] Once LLM integration is configured, the CLMI or GenAI interface 420 enables LLM components to handle these request and response payloads (e.g., translating, reviewing, and validating responses and requests). Generally, the GenAI interface 420 consists of various components for handling the following:

[0144] Request subject specification

[0145] The successful response body specification applies when an LLM call returns a standard response (such as an HTTP 200 status).

[0146] Error response body specifications apply when an LLM call returns a status other than a standard response and the LLM call itself succeeds. For example, in the case of HTTP status codes 401 (Unauthorized) or 500 (Internal Server Error), the LLM call failed, and therefore no response body is expected. The VFD LLM component will handle these status codes separately.

[0147] Subject of review request

[0148] Audit response entity

[0149] System 400 also includes one or more client devices 425 (e.g., personal computing devices, mobile devices, kiosks, IoT devices, etc.) that can be used by one or more users to interact with services provided by cloud computing platform 405 and other computing platforms 410 (optionally via one or more networks 430, which may host applications or websites for interacting with the services). Users of client devices 425 can send utterances (e.g., queries) or commands to the digital assistant / chatbot system for processing, including LLM integration via LLM components and CLMI or GenAI interface 420.

[0150] Digital assistants and knowledge dialogue (agent tracking)

[0151] Figure 5This is a simplified block diagram of an environment 500 incorporating a generative AI-based digital assistant system (also referred to herein as a digital assistant, or more specifically as an agent assistant in implementations involving agents) according to certain embodiments. Environment 500 includes a Digital Assistant Builder Platform (DABP) 505, which enables a user 510 to create and deploy a digital assistant system 515. For the purposes of this disclosure, a digital assistant is an entity that assists its user in performing various tasks through natural language conversation. The DABP and digital assistant can be implemented using software only (e.g., a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. In some instances, environment 500 is part of an Infrastructure as a Service (IaaS) cloud service (described in detail below) and can be implemented as part of the IaaS by leveraging scalable computing resources and storage capabilities provided by the IaaS provider to handle and manage large amounts of data and complex computations. This setup allows the DABP and digital assistant to provide real-time responsive interaction while ensuring high availability, security, and performance scalability to meet evolving demand levels. Digital assistants can be implemented or realized in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Accordingly, for the purposes of this disclosure, the terms digital assistant and chatbot system are used interchangeably.

[0152] The DABP 505 can be used to create one or more digital assistant systems (or DAs). For example, such as... Figure 5 As shown, a user 510 representing a specific business can use DABP 505 to create and deploy a digital assistant 515A for that specific business's users. For example, a bank can use DABP 505 to create one or more digital assistants for its customers to use, for example, to change 401k payments. Multiple businesses can use the same DABP 505 platform to create digital assistants. As another example, a restaurant owner, such as a pizzeria, can use DABP 505 to create and deploy a digital assistant 515B that enables the restaurant's customers to order food (e.g., order pizza).

[0153] To create one or more digital assistant systems 515, the DABP 505 is equipped with a set of tools 520 that enable the acquisition of LLMs, the creation of agents, the identification of assets, and the communication via computing platforms (such as those related to...). Figure 13-17(A detailed description of the cloud computing platform) deploys digital assistant systems within a user-facing service architecture. In some instances, tools 520 can be used to access pre-trained and / or fine-tuned LLMs from data repositories or computing systems. Pre-trained LLMs serve as foundational elements, possessing extensive language understanding derived from massive datasets. This capability enables models to generate coherent responses across a wide range of topics, facilitating transfer learning. Pre-trained models offer cost-effectiveness and flexibility, allowing for scalable improvements and continuous pre-training with new data, typically establishing benchmarks in natural language processing (NLP) tasks. Conversely, fine-tuned models are trained specifically for tasks or industries (e.g., leveraging the contextual learning capabilities of LLMs for plan creation, knowledge or information retrieval on behalf of agents, generating responses for human-like conversations, etc.), enhancing their performance on specific applications and enabling efficient learning from smaller, specialized datasets. Fine-tuning offers advantages such as task specialization, data efficiency, faster training times, model customization, and resource efficiency. In some embodiments, fine-tuning may be particularly advantageous for niche applications and continuous enhancement.

[0154] In other instances, Tool 520 can be used for pre-training and / or fine-tuning LLMs. Tool 520, or any subset thereof, can be a standalone machine learning operational framework, including hardware components such as processors (e.g., CPUs, GPUs, TPUs, FPGAs, or any combination thereof), memory, and storage devices. This framework operates software or computer program instructions (e.g., TensorFlow, PyTorch, Keras, etc.) to execute arithmetic, logical, and input / output commands for training, validating, and deploying machine learning models in production environments. In some instances, Tool 520 uses cloud platforms such as Oracle Cloud Infrastructure (OCI) for model training, validation, and deployment. Leveraging cloud platforms makes machine learning more accessible, flexible, and cost-effective, enabling developers to develop and deploy models more quickly.

[0155] Tool 520 also includes a prompt-based agent composition unit for creating agents and their associated actions that end users can ultimately invoke. An agent is a container for agent actions and can be part of one or more digital assistants. Each digital assistant can contain one or more agents through digital assistant relationships, which are cross-entities linking agents to digital assistants. Agents and digital assistants are implemented as robot subtypes and can be persisted to an existing BOTS table. This offers advantages in reusing design-time code (e.g., Java code) and UI artifacts.

[0156] Agent actions belong to a specific action type (e.g., knowledge, service, or API, LLM, etc.) and contain a schema (e.g., a JSON schema) that describes and defines action parameters. The action description and parameter schema are indexed by a semantic index and sent to the planner LLM to select one or more appropriate actions to execute. Action parameters are key-value pairs that serve as inputs to the action execution. They originate from properties in the schema but may also include additional UI / dialogue properties for slot-filling dialogues. Actions can be part of one or more classes. For example, some actions can be part of an application event subscription class that defines the agent actions to be performed when an application event is received. Application events can be received as command messages that update the application context. An application event property mapping class (part of the application event subscription class) specifically maps application event payload properties to corresponding agent action parameters. Actions can optionally be part of an action group. Action groups can be used when importing plugin manifests or external API specifications (such as the Open API specification). Action groups are particularly useful when re-importing plugins or open API specifications, allowing you to add new actions, update existing actions, or remove actions from new manifests or open API specifications that no longer exist. At runtime, action groups can be used to restrict the application context groups sent to the LLM as conversation contexts by looking up the action group name that corresponds to the context group context.

[0157] An agent (e.g., a 401k "change payment" agent) can be primarily defined as an assembly of agent artifacts using natural language within a prompt-based agent composition unit. User 510 can quickly create functional agents by providing agent artifact information, parameters, and configuration, and by pointing to assets. Assets can be or include resources such as APIs for interfacing with applications, files and / or documents for retrieving knowledge, data repositories for interacting with data, etc., which can be used by the agent to perform actions. Once assets are imported, User 510 can then again provide additional API customizations using natural language for dialogue and routing / reasoning. Much of what the agent performs likely involves executing actions. Actions can be explicit actions written in natural language (similar to creating agent artifacts—for example, the "What impact does XYZ have on my 401k contribution limit?" action in the "401k Contribution Agent" diagram below), or implicit actions created when importing assets (automatically imported based on metadata and / or specifications associated with the asset when pointing to a given asset—for example, actions created for the "Change Contribution" and "Get Contribution" APIs in the "401k Contribution Agent" diagram below). Users can easily create explicit actions at design time. For example, when a user learns that a new FAQ needs to be added (because it is not currently in the knowledge document (asset) referenced by the agent (and therefore not implicitly added as an action)), the user can select a "Rich Text" action type (see the list of exemplary action types in Table 1) and create the artifact named "What impact does XYZ have on my 401k contribution limit?".

[0158] Table 1:

[0159]

[0160] There are various ways to associate or add agents and assets to a digital assistant 515. In some instances, agents can be developed by an enterprise and then added to the digital assistant using DABP 505. In other instances, agents can be developed and created using DABP 505 and then added to a digital assistant created using DABP 505. In still other instances, DABP 505 provides an online digital store (called the "Agent Store") that offers a variety of pre-created agents for a wide range of tasks and actions. Agents offered through the Agent Store can also expose various cloud services. To add an agent to a digital assistant created using DABP 505, a DABP 505 user 510 can access assets via tool 520, select specific assets for the agent, initiate several simulated chat conversations with the agent, and instruct the agent to be added to the digital assistant created using DABP 505.

[0161] Once deployed in a production environment (such as regarding...) Figure 2In the described architecture, a digital assistant (such as a digital assistant 515A built using a DABP 505) can be used to perform various tasks via natural language-based conversation between the digital assistant 515A and its user 525. As described above, this enables... Figure 5 The digital assistant 515A shown is available for use or access by its user 525 through a variety of different channels, such as, but not limited to, certain applications, social media platforms, various messaging services and applications, and other applications or channels. A single digital assistant may have several channels configured for it, allowing it to run and be accessed by different services simultaneously.

[0162] As part of a conversation, user 525 can provide one or more user inputs 530 to digital assistant 515A via a user interface element such as a chat window and receive a response 535 from digital assistant 515A. A conversation may include one or more of the user inputs 530 and responses 535. Through these conversations, user 525 can request one or more tasks to be performed by digital assistant 515A, and in response, digital assistant 515A is configured to perform the user-requested task and respond to user 525 with an appropriate response using one or more LLMs 540. Conversations displayed in the chat window can be organized by thread. For example, in some applications, conversations related to one page of the application should not be mixed with conversations related to another page of the application. Applications and / or application plugins define thread boundaries (e.g., a set of (nested) plugins can run within their own threads). In practice, the chat window will only display the history of messages belonging to the same thread. Thread settings and changes can be performed via applications and / or plugins using update application context command messages. Additionally or alternatively, when a user query matches a plugin semantic action and the plugin is running in a thread different from the current thread, the thread can be changed via the execution plan orchestrator. In this case, the planner changes the thread so that any messages sent in response to the executed action are displayed in the correct new thread. For each agent dialogue thread, the digital assistant can maintain the following information: application context, LLM conversation history, conversation history with the user, and agent execution context that keeps information about one or more (stacked) execution plans associated with that thread.

[0163] User input 530 is typically in natural language form and is referred to as a utterance (which may also be referred to as a prompt, query, request, etc.). User input 530 can be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides it as input to the digital assistant 515A. In some embodiments, user input 530 can be in audio input or speech form, such as when a user speaks or utters something provided as input to the digital assistant 515A. User input 530 is typically in the language spoken by the user 525. For example, user input 530 might be English or some other language. When user input 530 is in speech form, the speech input is converted into text form of user input 530 in that particular language, and then the text utterance is processed by the digital assistant 515A. Various speech-to-text processing techniques can be used to convert speech or audio input into text utterance, which is then processed by the digital assistant 515A. In some embodiments, the speech-to-text conversion can be performed by the digital assistant 515A itself. For the purposes of this disclosure, it is assumed that user input 530 is either text speech directly provided by user 525 of digital assistant 515A, or the result of inputting voice speech converted into text form. However, this is not intended to limit or constrain in any way.

[0164] User input 530 can be used by digital assistant 515A to determine a list of candidate agents 545A-N. The list of candidate agents (e.g., 545A-N) includes agents configured to perform one or more actions that may potentially facilitate a response 535 to user input 530. This list can be determined by running searches (such as semantic searches) on a context and memory repository having one or more indices containing metadata of all agents 545 available to digital assistant 515A. The metadata of candidate agents 545A-N in the list of candidate agents is then combined with the user input to construct an input prompt for one or more LLMs 540.

[0165] The digital assistant 515A is configured to apply NLP techniques to text and / or speech using one or more LLM 540s to understand input prompts and to apply Natural Language Understanding (NLU), including syntactic and semantic analysis of the text and / or speech, to determine the meaning of user input 530. Determining the meaning of utterances may involve identifying the user's goal, one or more of the user's intentions, the context surrounding various words or phrases or sentences, one or more entities corresponding to the utterance, etc. NLU processing may include parsing the received user input 530 to understand the structure and meaning of the utterance, refining and reshaping the utterance to develop a more understandable form (e.g., logical form) or structure. The NLU processing performed may include various NLP-related processes such as sentence parsing (e.g., lexicalization, lemmatizing, identifying part-of-speech tags of sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing individual clauses, parsing pronouns, performing chunking, etc.). In some instances, NLU processing, or any part thereof, is performed by the LLM 540 itself. In other instances, the LLM 540 uses additional resources to perform parts of the NLU processing. For example, the syntax and structure of the input utterance can be identified by processing sentences using parsers, part-of-speech taggers, named entity recognition models, pre-trained language models (such as BERT), etc.

[0166] After understanding the meaning of the utterance, one or more LLMs 540 generate an execution plan that identifies one or more agents (e.g., agent 545A) from a list of candidate agents to perform and conduct one or more actions or operations in response to the understood user meaning or goal. The one or more actions or operations are then executed by the digital assistant 515A on one or more assets (e.g., asset 550A—knowledge, API, SQL operations, etc.) and / or a context and memory repository. The execution of the one or more actions or operations generates output data from the one or more assets and / or generates relevant context and memory information from a context and memory repository that includes the context of the current conversation with the digital assistant 515A. The output data and relevant context and memory information are then combined with user input 530 to construct an output prompt for one or more LLMs 540. The LLM 540 synthesizes a response 535 to user input 530 based on the output data, relevant context and memory information, and user input 530. The response 535 is then sent to user 525 either as a separate response or as part of a conversation with user 525.

[0167] For example, user input 530 could request to order pizza by providing a utterance such as “I want to order pizza.” Upon receiving such a utterance, digital assistant 515A is configured to understand the meaning or goal of the utterance and take an appropriate action. The appropriate action might involve, for example, providing the user with a response 535, which has a question requesting the user to input information about the type of pizza the user wishes to order, the size of the pizza, any toppings, etc. The user's request for the question can be generated by performing an action on a knowledge asset (e.g., a pizza restaurant's menu) via an agent (e.g., agent 545A) to retrieve information related to ordering pizza (e.g., to order pizza, the user must provide the type, size, toppings, etc.). The response 535 provided by digital assistant 515A can also be in natural language form and is typically the same language as user input 530. As part of generating these responses 535, digital assistant 515A may perform natural language generation (NLG) using one or more LLMs 540. For a user ordering pizza, the digital assistant 515A can guide the user to provide all the necessary information for the pizza order through a conversation between the user and the digital assistant 515A, and then the pizza is ordered at the end of the conversation. The order can be executed by an agent (e.g., agent 545A) performing an action on an API asset (e.g., an API for ordering pizza) to upload or provide the pizza order to the restaurant's ordering system. The digital assistant 515A can end the conversation by generating a final response 535 that provides the user 525 with information indicating that the pizza has been ordered.

[0168] While the various examples provided in this disclosure describe and / or illustrate utterances in the English language, these are merely examples. In some embodiments, the digital assistant 515 may also handle utterances in languages ​​other than English. The digital assistant 515 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes NLU processing flexible and scalable for each language, including allowing different processing sequences. Language packs may be provided for each language, wherein the language packs may register a list of subsystems that can be serviced by the NLU core server.

[0169] Although Figure 5 The embodiments illustrated herein include a digital assistant 515A comprising one or more LLMs 540 and one or more agents 545A-N, but this is not intended to be limiting. A digital assistant may include various other components that provide the functionality of a digital assistant (e.g., as per [reference to...]). Figure 6Other systems and subsystems described in more detail. The Digital Assistant 515A and its systems and subsystems may be implemented using software only (e.g., code or instructions stored on a computer-readable medium and executable by one or more processors), hardware only, or a combination of software and hardware.

[0170] Figure 6 This is an example of the architecture of a computing environment 200 using generative artificial intelligence, according to various embodiments. For example... Figure 6 As shown, the infrastructure, along with various services and features, can be used to enable users to interact with digital assistants (e.g., regarding...) at least in part based on a series of prompts (such as conversation) and / or a series of actions (such as interaction with the user interface). Figure 5 The interaction takes place with the digital assistant 515A described below. The following is a detailed walkthrough of the conversation flow, and the roles and responsibilities of the components, services, models, etc., of the computing environment 600 within the conversation flow. In this walkthrough, assume that user "David" is interested in making changes to his 401k contribution, and in utterance 602, David provides the following input: Hi, how are you? I want to change my 401k contribution. utterance 602 can be transmitted to the digital assistant (e.g., via a digital assistant user interface such as a text dialog box or microphone). Upon receiving utterance 602 at this stage, the sessionist creates a new session or retrieves the current session context, and the user message publisher updates the session transcript and LLM message history with the new user message (e.g., utterance 602).

[0171] In instances where a user provides utterances (602) and / or performs actions while using an application supported by a digital assistant, the application issues an update application context command when the user interacts with the application (e.g., providing utterances via text or audio, triggering user interface elements, navigating between pages in the application, etc.). Whenever the digital assistant receives an update application context command message from the application, an application context handler (part of the context manager) is implemented. The application context handler performs the following tasks: (i) managing conversation threads based on the application context message, for example, creating a new conversation thread and making it the current thread if the threadId specified in the message does not already exist, and making the corresponding conversation thread the current thread if the threadId already exists; (ii) creating or updating application context objects for the current conversation thread; and (iii) if a service call ID such as a REST request ID is included, the application context can be enriched (as described in more detail herein). It should be understood that the application context only contains information reflecting the state of the application's user interface and plugins (if available); it does not contain other state information (e.g., user or page state information / context).

[0172] In some instances, when an update application context command message is received, the application event handler checks whether the update application context command message includes an event definition. An event is uniquely identified by the following attributes in the message payload: (i) context: the context path and / or plugin path (for top-level workspace plugins, the context is set to the plugin name; for nested plugins, it includes the plugin paths separated by slashes, e.g., Patient / Vitalschart); (ii) eventType: the type of the event, which can be either a built-in event or a custom event; and (iii) semantic object: the semantic object to which the event is applied. Events can be mapped to one or more actions, and message payload attributes can be mapped to action parameters. This mapping is done through the application event subscription. Each attribute in the message payload can be mapped to an agent action parameter using the application event attribute map.

[0173] In some instances, utterance 602 and / or user-performed actions are provided directly as input to planner 608. In other instances implementing an application event processor, utterance 602 and / or user-performed actions are provided as input to planner 608 when the application event processor determines that an event such as receiving utterance 602 maps to an agent or action associated with the digital assistant. Planner 608 is used by the digital assistant to create execution plan 610 using specified parameters from utterance 602, user-performed actions, context, or any combination thereof. Execution plan 610 identifies one or more agents and / or one or more actions to be performed by one or more agents in response to utterance 602 and / or user-performed actions.

[0174] An execution plan 610 can be generated via a two-step process via planner 608. First, a search 612 can be performed to identify a list of candidate agents and / or actions. Search 612 involves running queries against an index 613 (e.g., a semantic index) of context and memory repository 614 based on utterances 602 and / or user-performed actions. In some instances, search 612 is a semantic search performed using words in utterances 602 and / or that represent user-performed actions. Semantic search uses NLP and optional machine learning techniques to understand the meaning of utterances 602 and / or user-performed actions and retrieves relevant information from context and memory repository 614. Compared to traditional keyword-based searches that rely on exact matches between words in the query and data in context and memory repository 614, semantic search takes into account relationships between words, the context of the query and / or actions, synonyms, and other linguistic nuances. This allows the digital assistant to provide more accurate and context-sensitive results, making it more effective in understanding user intent in utterances 602 and / or user-performed actions.

[0175] To run the query, planner 608 invokes context and memory repository 614 (e.g., the semantic index of context and memory repository 614) to obtain a list of candidate agents and / or actions. The following information is passed in the call: (i) the digital assistant's ID (which defines the set of agents and / or actions the semantic index will search, and therefore the agents and / or actions must be part of the digital assistant), and (ii) the last X user messages and / or actions (e.g., X can be set to the last 5 rounds), which can be configured via the digital assistant settings.

[0176] The context and memory repository 614 is implemented using a data framework that connects external data to the LLM 616, allowing users to easily plug in custom data sources. The data framework provides a rich and efficient retrieval mechanism on top of data from various sources, such as files, documents, data repositories, APIs, etc. Data can be external (e.g., enterprise assets) and / or internal (e.g., user preferences, memories, digital assistant and agent metadata, etc.). In some instances, the data includes metadata extracted from artifacts 617 associated with the digital assistant and its agents 618 (e.g., 618a and 618b). The digital assistant's artifacts 617 include information about the general capabilities of the digital assistant, as well as specific information about the capabilities (e.g., actions) of each agent 618 (e.g., agent artifacts) available to the digital assistant. Additionally or alternatively, artifacts 617 may contain parameters or information associated with artifacts 617 and that can be used to define agents 618, wherein the parameters or information associated with artifacts 617 may include a name, description, one or more actions, one or more assets, one or more customizations, etc. In some instances, the data also includes metadata extracted from assets 619 associated with the digital assistant and its agents 618 (e.g., 618a and 618b). Assets 619 can be resources that agents 618 can use to perform actions (e.g., actions 625a, 625b, and 625c), such as APIs 620, files and / or documents 622, data repositories 623, etc. The data is indexed in a context and memory repository 614 as an index 613, which is a data structure that provides a fast and efficient way to find and retrieve specific data records within the data. Thus, the context and memory repository 614 provides a searchable and comprehensive record of all the capabilities available to the digital assistant to respond to requests and / or actions.

[0177] The response from the context and memory repository 614 is transformed into a list of agent and / or action instances that are not only available to the digital assistant to respond to requests but also potentially facilitate the generation of responses to utterance 602 and / or actions performed by the user. The list of candidate agents and / or actions contains metadata associated with each candidate agent and / or action in the context and memory repository 614 (e.g., metadata extracted from artifact 617 and asset 619). This list may be limited to a predetermined number of candidate agents and / or actions that satisfy the query (e.g., the top 10), or it may include all agents and / or actions that satisfy the query. The list of candidate agents and / or actions with associated metadata is appended to utterance 602 and / or actions performed by the user to construct input prompts 627 for LLM 616. Search 612 is important for the digital assistant because it filters out agents and / or actions that are unlikely to facilitate the generation of responses to utterance 602 and / or actions performed by the user. This filtering ensures that the number of lexical units (e.g., word lexical units) generated from input prompt 627 remains within the maximum lexical unit limit or contextual limit set for LLM 616. The lexical unit limit represents the maximum amount of text that can be input into the LLM. This limit is technical in nature and arises from computational constraints such as memory and processing resources, and thus ensures that the LLM can accept input prompts as input.

[0178] In some instances, one or more knowledge actions are additionally appended to the list of candidate agents and utterance 602. Knowledge actions allow for the acquisition of additional knowledge related to utterance 602 and / or actions performed by the user (this knowledge is typically outside the knowledge scope of the LLM used to train the digital assistant). There are two types of knowledge action sources: (i) structured: the knowledge source defines a predefined list of questions the user might ask and exposes it as some API (e.g., Multum), and (ii) unstructured: with the knowledge source, the user has an unlimited number of ways to ask questions, and the knowledge source exposes a generic query interface (e.g., medical documents (SOAP notes, discharge summaries, etc.)).

[0179] In some instances, conversation context 629 for utterance 602 is additionally appended to the list of candidate agents and utterance 602. Conversation context 629 can be retrieved from one or more sources, including context and memory repositories 614, and includes user session information, conversation state, conversation or context history, application context, page context, user information, or any combination thereof. For example, conversation context 629 may include: the current date and time (required to resolve time references in user queries, such as "yesterday" or "next Thursday"), additional context (which contains information such as user profile attributes and application context groups with semantic object attributes), and chat history with the digital assistant (and / or other digital assistants or systems internal or external to computing environment 600).

[0180] The second step of the two-step process involves LLM 616 generating an execution plan 610 based on input prompt 627. The invocation of LLM 616 can be performed as follows: creating an LLM chat message with a role system, receiving input prompt 627, converting candidate agents and / or actions into LLM function definitions, retrieving the correct LLM client based on LLM configuration options, optionally transforming input prompt 627, the LLM chat message, etc., into the correct format for the LLM client, and sending the LLM chat message to the LLM client to invoke LLM 616. The LLM client then sends back an LLM success response in CLMI format, or uses an adapter such as OpenAIAdapter to convert a provider-specific response back to a CLMI format LLM success response (or sends back or converts to an LLM error response in case of unexpected errors). An LLM invocation instance is created and added to the conversation history, which captures all request and response details, including execution time.

[0181] LLM 616 employs a deep generative model architecture (e.g., an invertible or autoregressive architecture) to generate execution plans 610. In some instances, LLM 616 has over 100 billion parameters and uses autoregressive language modeling within a transformer architecture to generate execution plans 610, allowing LLM 616 to capture complex patterns and dependencies in the input cue 627. LLM 616's ability to generate execution plans 610 is a result of its training on diverse and extensive textual data, enabling LLM to understand human language in various contexts. During training, LLM 616 learns to predict the next word in a sequence based on the context of the preceding word. This process involves adjusting the model's parameters (weights and biases) based on the error between its prediction and the actual next word in the training data. When LLM 616 receives input such as the input cue 627, it breaks the text into smaller units, such as words or sub-words. Each word is then represented as a vector in a multi-dimensional space. LLM 616 processes the input sequence word by word, maintaining an internal representation of the context. LLM 616's attention mechanism allows it to measure the importance of different words in the context of generating the next word. For each word in the vocabulary, LLM 616 computes a probability distribution based on its learned parameters. This probability distribution represents the likelihood that each word is the next word in a given context. For example, to generate execution plan 610, LLM 616 samples words from the computed probability distribution. The sampled word becomes the next word in the generated sequence. This process is repeated iteratively, with each newly generated word influencing the context for generating subsequent words. LLM 616 can continue generating words until a predefined length or stopping condition is reached.

[0182] In some instances, such as Figure 6As shown, LLM 616 may fail to generate a complete execution plan 610 because it lacks information such as whether more information is needed to determine the appropriate agent for the response, to perform one or more actions, etc. In this particular instance, LLM 616 has determined that, in order to change a 401k contribution at the user's request, it is necessary to know whether the user is willing to change the contribution by a percentage or a specific monetary amount. To obtain this information, LLM 616 (or another LLM such as LLM 636) generates an end-user response 635 to input prompt 627 (I'm fine. Would you like to change your contribution by a percentage or an amount? [Percentage] [Amount]), which obtains the missing information, allowing LLM 616 to generate a complete execution plan 610. In some instances, the response can be rendered within a dialog box in the UI, which has one or more UI elements to allow the user to respond more easily. In other instances, the response can be rendered within a dialog box in the UI, allowing the user to respond using the dialog box (or an alternative, such as a microphone). In this particular instance, the user responds with an additional query 638 (What is my current 401k contribution? And can you tell me my contribution limit?) to gather additional information, allowing the user to reply with response 635. The subsequent response—additional query 638—is fed into planner 608 and performs the same processing described above for utterance 602, but this time with context from the user's conversation with the digital assistant regarding the previous utterance / response (e.g., utterance 602 and response 635). This time, as... Figure 6 As shown, LLM 616 is able to generate a complete execution plan 610 because it has all the information it needs.

[0183] In some instances, a user's utterance 602 can be identified by LLM 616 as an illogical utterance (i.e., an utterance that logically does not follow the previous utterance in a conversation or dialogue). In such instances, an execution plan orchestrator can be used to handle the switching between different conversation paths. The execution plan orchestrator is configured to: track all ongoing conversation paths; if a new conversation path is created, create a new entry and pause the currently ongoing conversation (if any); if the conversation concludes based on metadata or user preferences for a new action, remove the entry; it may generate a prompt message when an illogical utterance begins or continues a previous utterance; manage the conversation for which a prompt message is sent and choose to continue or resume the current conversation; confirm or cancel when the user responds to a prompt for an illogical utterance; and manage cancellation or exit from the conversation.

[0184] Execution plan 610 includes an ordered list of agents and / or actions that can be used and / or executed to adequately respond to requests (such as additional query 638). For example, and as... Figure 6As shown, execution plan 610 may be an ordered list including a first agent 642a capable of performing a first action 644a via an associated asset and a second agent 642b capable of performing a second action 644b via the associated asset. The agents and the actions extended therefrom may be ordered such that the first agent 642a performs the first action 644a before the second agent 642b performs the second action 644b. In some instances, execution plan 610 may be ordered based on dependencies indicated by the agents and / or actions included in it. For example, if the execution of second agent 642b depends on or otherwise requires the output generated by first agent 642a performing first action 644a, then execution plan 610 may command first agent 642a and second agent 642b to comply with that dependency. It should be understood that other examples of dependencies are also possible.

[0185] The execution plan 610 is then transmitted to the execution engine 650 for implementation. The execution engine 650 includes multiple engines, including a natural language to programming language translator 652, a knowledge engine 654, an API engine 656, a prompting engine 658, etc., for executing the agent's actions and implementing the execution plan 610. For example, the natural language to programming language translator 652 (such as a conversation to Oracle Meaning Representation Language (C2OMRL) model) can be used by the agent to translate natural language into (e.g., OMRL) intermediate logic, convert that intermediate logic into a system programming language (e.g., SQL), and execute that system programming language (e.g., execute an SQL query) on an asset 619 (such as a data repository 623) to perform actions and / or obtain data or information. The knowledge engine 654 can be used by the agent to obtain data or information from a context and memory repository 614 or asset 619 (such as a file / document 622). The API engine 656 can be used by the agent to call API 620 and interface with applications (such as a retirement account management application) to perform actions and / or obtain data or information. The prompting engine 658 can be used by an agent to construct prompts for input into an LLM (such as an LLM in a context and memory repository 614 or an asset 619) to perform actions and / or obtain data or information.

[0186] Execution engine 650 implements execution plan 610 by using one or more appropriate engines to run each agent and execute each action sequentially based on an ordered list of agents and / or actions. To facilitate this implementation, execution engine 650 communicates with agents (e.g., 642a, 642b, etc.), context and memory repository 614, and assets 619 (e.g., via public and / or private networks). Figure 6As shown, when execution engine 650 implements execution plan 610, it will first use API engine 656 to execute agent 642a and action 644a to call API 620 and interface with the retirement account management application to retrieve the user's current 401k contributions. Subsequently, execution engine 650 can use knowledge engine 654 to execute agent 642b and action 644b to retrieve knowledge about 401k contribution limits. In some instances, knowledge engine 654 retrieves knowledge from asset 619 (e.g., file / document 622). In other instances (as in this particular instance), knowledge is retrieved by knowledge engine 654 from context and memory repository 614. Various techniques, including internal task mapping and / or machine learning models, such as attached LLM models, can be used to implement knowledge retrieval and action execution using context and memory repository 614. For example, a query for “What is the 401k contribution limit?” and the associated agent can be mapped to a “semantic search” knowledge task type, used to search indexes 613 within the context and memory repository 614 in response to a given query. As another example, a request such as “Can you summarize the key points related to 401k contributions?” can be or include a “summarize” knowledge task type, which can be mapped to different indexes within the context and memory repository 614, which have LLMs trained to create natural language responses (e.g., summaries of key points related to 401k contributions) for a given query. Over time, a library of general end-user task or action types (e.g., semantic search, summarization, comparison / contrast, heterogeneous data synthesis, etc.) can be built to ensure that the indexes and models within the context and memory repository 614 are optimized for various task or action types.

[0187] The result of executing execution plan 610 is output data 669 (e.g., the result of an action, data, information, etc.), which is passed to output pipeline 670 to generate end-user response 672. For example, output data 669 from assets 619 (knowledge, APIs, conversation history, etc.) and relevant information from context and memory repository 614 can be passed to output pipeline 670. Output data 669 is appended to utterance 602 to construct output prompt 674 for input to LLM 636. In some instances, context 629 about utterance 602 is additionally appended to output data 669 and utterance 602. Context 629 can be retrieved from context and memory repository 614 and includes user session information, conversation state, conversation or context history, user information, or any combination thereof. LLM 636 generates response 672 based on output prompt 674. In some instances, LLM 636 is the same or similar model to LLM 616. In other instances, LLM 636 differs from LLM 616 (e.g., trained on different datasets, with different architectures, trained for one or more different tasks, etc.). In any instance, LLM 636 has a deep generative model architecture (e.g., with an invertible or autoregressive architecture) for generating responses 672 using training and generative processing similar to that described for LLM 616. In some instances, LLM 636 has over 100 billion parameters and uses autoregressive language modeling to generate responses 672 within the transformer architecture, allowing LLM 636 to capture complex patterns and dependencies in the output cues 674.

[0188] In some instances, the end-user response 672 can be formatted using the Conversational Message Model (CMM) and output as a rich, multimodal response. The CMM defines various message types that a digital assistant can send to a user (outbound) and that a user can send to a digital assistant (inbound). In some instances, the CMM recognizes the following message types:

[0189] • Text: Basic text messages

[0190] • Cards: Card representations that include a title and optional descriptions, images, and links.

[0191] • Attachment: A message containing a media URL (file, image, video, or audio).

[0192] Location: A message containing geographic location coordinates.

[0193] • Sendback: A message with a sendback payload

[0194] Messages defined in CMM are channel-independent and can be created using CMM syntax. Channel-specific connectors transform CMM messages into the format required for a specific channel, allowing users to run digital assistants on multiple channels without having to create separate message formats for each channel.

[0195] Finally, output pipeline 670 (e.g., via user device or interface) transmits response 672 to the end user. In some instances, response 672 is rendered within a dialog box in the GUI, allowing the user to view and respond using the dialog box (or alternative means, such as a microphone). In other instances, response 672 is rendered within a GUI dialog box with one or more GUI elements, allowing for easier user responses. In this particular instance, the first response 672 to the additional query 638 (What is my current 401k contribution? Also, could you tell me my contribution limit?) is rendered within a dialog box in the GUI. Furthermore, to follow up on information still needed to obtain the initial statement 602, LLM 636 generates another response 672 prompting the user to provide the missing information (Would you like to change your contribution by percentage or amount? [Percentage] [Amount]).

[0196] Although Figure 6 The embodiment of computing environment 600 illustrates the interaction of a digital assistant in a specific conversation flow, but this is not intended to be limiting, but merely to facilitate a better understanding of the role and responsibilities of the components, services, models, etc. of computing environment 600 within the conversation flow.

[0197] Digital assistant with Copilot support

[0198] Copilot represents a new generation of innovative user experiences designed to enhance applications, such as business applications. Utilizing technologies like... Figure 1-6 The described digital assistant framework allows developers to create customized digital assistants incorporating a variety of skills, including specialized functionalities such as ready-to-use digital assistants with skills like expense management, project management, and human capital management (HCM). As discussed in this paper, skill developers now have the ability to integrate generative AI components such as LLM into their skill streams, allowing them to introduce generative AI capabilities into their existing digital assistants. Leveraging the hybrid capabilities of digital assistants, copilot can seamlessly integrate with applications, providing both conversational and generative AI experiences tailored to user needs while using the application for a variety of tasks.

[0199] Copilot implementation

[0200] To integrate copilot into applications such as HCM or skill robot design applications, developers access the application development environment and configure the Copilot SDK integration endpoint and credentials along with the DA. This simplifies the integration process, enabling developers to efficiently incorporate copilot's digital assistant capabilities into their applications.

[0201] For context passing integration: Developers enable context-aware framework features, including passing application context, page context, and user context from applications and other sources to copilot and ultimately to the DA. The passing of context and the ability to execute DA skills or LLM components using that context can be achieved via a set of REST APIs exposed to developers targeting copilot support (the Copilot SDK provides convenient wrappers around these REST APIs). This allows REST API calls to the digital assistant conversation pipeline by sending CMM messages using new types of CommandMessagePayloads (start thread and update context). The CMM defines various message types that DA skills or LLMs can send to users (outbound) and that users can send to DA skills or LLMs (inbound). For start thread, the command attribute value in the CommandMessagePayload used for Invoke Flow commands can be set to a key-value pair such as startThread. The following attributes from Table 2 can be used:

[0202] Table 2:

[0203]

[0204] The threadId (thread identifier) ​​of a new thread can be passed in as part of the channel conversation metadata.

[0205] To update the context, the Copilot SDK provides methods for setting the user, application, page (including context widgets – for example, context obtained from user interactions with UI elements), and field context within a context variable data structure (e.g., one or more JSON conversation objects stored in association with a JSON session object). This context variable data structure can be associated with a threadId. These methods are translated into CMM messages sent by the chat server. Command attribute values ​​in the CommandMessagePayload used for updating the context can be set as key-value pairs, such as updateContext. The following attributes from Table 3 can be used:

[0206] Table 3:

[0207]

[0208] For user context, using the SDK API, developers can configure the CommandMessagePayload to deliver user preferences such as first and last name, email, job title, department, work location, language, time zone, and date format. For application context, using the SDK API, developers can configure the CommandMessagePayload to deliver navigation such as "Navigation: Home". For page context, leveraging available controls, developers can choose to deliver the entire page content to copilot, or select options as needed. For example, a developer might decide not to deliver the entire homepage because it may not be the proper fit. Therefore, developers configure copilot to deliver page context (such as the application's current page view, i.e., the homepage view) using the SDK API. These contexts and CMM messages enable the declarative definition of copilot's functionality to obtain and synchronize application context (application, page, field) within the context variable data structure. Based on the context within the context variable data structure, routing rules are used by copilot to invoke the conversation flow between the DA / skill and the user (supporting routing to various skills or LLMs within the DA system) and trigger actions based on the current context (e.g., assisting with application navigation based on the indicated context). Advantageously, this provides personalized conversation start prompts and dialogues, which are conveyed to the user through Copilot's interface.

[0209] To support context-aware routing, the copilot routing section can be added to the DA's dialogue engine via skill settings. When an updateContext CommandMessagePayload is received, the copilot routing section defines routing rules. These rules map various applications, application components, pages, page content, etc., to various skills, LLMs, and / or dialogue flows. For example, a routing section can contain a table with the following fields (see Example Table 4):

[0210] Table 4:

[0211]

[0212] Here is an example of how to populate this table in an HCM use case (see example table 5) (where the three context attribute values ​​from tables 3 and 4 can be hcm, hcm:jobDescription, and hcm:benefits):

[0213] Table 5:

[0214]

[0215] Once this route configuration is ready, the updateContext command message can be processed as follows:

[0216] 1. Internally, the concept of threads can be used (similar to the concept used in interactive modals with Copilot). Each stream triggered via the updateContext command message will run on its own thread. This allows state to be maintained for each running stream, so when the user switches between pages, the correct thread can be resumed, and the state associated with that thread can be restored. State that needs to be shared across threads can be stored in skill-scoped variables. The Copilot SDK can simply omit the threadId that will be included in the channel conversation metadata in the bot's response.

[0217] 2. Use a table, and optionally find the flow that matches the applicationName as specified in the message.

[0218] 3. Then check if an existing thread with a threadId set to the stream name is already running in the session.

[0219] a. If no such thread exists, then invoke the stream using a new thread with a threadId set to the stream name.

[0220] b. If such a thread exists, then resume that thread.

[0221] 4. Introduce a new `system.coPilotContext` variable, which holds a mapping with application name, page name, and field names.

[0222] 5. Optional variables passed along with the updateContext CommandMessagePayload are stored in a new skill / DA scope variable named system.coPilotContext. This variable can be a JSONObject, where each key contains the variable name and the value is the variable's value.

[0223] As a verification step, developers can test the context-aware features within the application UI by manually navigating to a page (such as the benefits section in an HCM application) and initiating a side panel chat. Developers will see a personalized and context-sensitive response from Copilot, such as "Hi Michael, do you need more information about the benefits plan?", along with suggested actions like "Summary medical benefits," "What is my cap?" and "How much do I need to co-pay?" Developers can then select the "Summary medical benefits" action and see the summary.

[0224] Here is an example of an end user:

[0225] To initiate the hiring process, Sarah logged into the Oracle HCM system. It had been 12 months since she last hired an employee, and she didn't quite remember how the process worked, plus the system had been upgraded since then. Sarah was a little apprehensive because she would spend 30 minutes figuring out what she needed to do, rather than actually posting the job opening.

[0226] Sarah was overwhelmed by the numerous options and menus on the HCM login page, but a small welcome button in the lower right corner caught her eye. She had seen similar chat buttons on her favorite online retailer's website, so she clicked it. The chat window expanded into a side panel chat, and Sarah received a greeting from HCM Copilot:

[0227] "Hello Sarah, I am your HCM Copilot. I can help you with:"

[0228] • Your employee self-service tasks

[0229] • Manage your team

[0230] • Hiring and bringing in new team members

[0231] Sarah breathed a sigh of relief; this copilot was helping her complete her recruitment task. Behind the scenes, copilot perceived Sarah's profile and current application view (homepage) within the context. It knew she was an employee, a manager, and currently in the application's homepage view, thus providing the appropriate points in its greeting.

[0232] Since Sarah's goal was to create job applications for senior application developers, she navigated to the job application creation page herself.

[0233] Sarah saw that there were a total of 6 sections she needed to fill in. Copilot obtained the application and page context regarding the different section and field types.

[0234] Sarah saw that in the open side panel chat, Copilot was now responding, "How can I help you with your job application creation?". She could type, or better yet, use her microphone to talk to Copilot. She clicked the microphone icon and used Copilot.

[0235] Sarah said, "I need to create a new job application for a senior application developer. Can you assist me?"

[0236] Copilot confirmed Sarah's request and asked targeted questions about the new job opening, such as the job level, location, and required qualifications.

[0237] Copilot: "To create a job application, I need some details. What will the job level, location, and minimum qualifications be?"

[0238] Sarah: "This will be a remote full-time employee based in the United States, working on a project for our client that requires comprehensive and proficient Java technical skills."

[0239] Copilot confirmed Sarah's response. Behind the scenes, Copilot was gradually learning the page context of Sarah's current view, including the available elements in the form. It was also able to update these fields with the data provided by Sarah. But before that, Copilot sought confirmation from Sarah.

[0240] Copilot: "Thank you, Sarah. I will guide you through the job application process step by step. Would you like me to automatically populate some of the data that has already been collected?" (With options "Yes" or "No")

[0241] Interface Modal

[0242] The side panel chat is a chat interface where users can interact with copilot while still maintaining full visibility of the application. It's a context-aware UI embedded within the application, displaying responses generated by the DA's skills and LLM components. For example, the side panel chat can (i) determine the context of the task to be performed using the application when a user opens a specific page (such as a benefits page or job application page), (ii) route CMM messages to the appropriate DA's skills and LLMs based on that context and routing rules, and (iii) display context-appropriate responses from the DA. The side panel chat also provides copilot with the ability to understand and correlate the information being queried / entered with the content available on the page view. It also gives users the ability to update page input areas or field values ​​either from generated content or based on user input.

[0243] Here is an example of an end-user:

[0244] To initiate the hiring process, Sarah logged into the Oracle HCM system. It had been 12 months since she last hired an employee, and she didn't entirely remember how the process worked, plus the system had been upgraded since then. Sarah was a little apprehensive because she would spend 30 minutes figuring out what she needed to do, rather than actually posting the job requirements.

[0245] Sarah was overwhelmed by the numerous options and menus on the HCM login page, but a small welcome button in the lower right corner caught her eye. She had seen similar chat buttons on her favorite online retailer's website, so she clicked it. The chat window expanded (as shown in the image). Figure 7 (As shown in the image) is the side panel chat, where Sarah receives a greeting from HCMPilot:

[0246] "Hello Sarah, I am your HCM Copilot. I can help you with:"

[0247] • Your employee self-service tasks

[0248] • Manage your team

[0249] • Hiring and bringing in new team members

[0250] Sarah breathed a sigh of relief; this copilot was helping her complete her recruitment task. Behind the scenes, copilot perceived Sarah's profile and current application view (homepage) within the context. It knew she was an employee, a manager, and currently in the application's homepage view, thus providing the appropriate points in its greeting.

[0251] Micro-conversation mode is a chat-style user interface designed to facilitate fast and focused one-turn interactions. It provides a simple and concise chat-like interface where users can enter queries and receive responses one at a time. For example... Figure 8 As shown, the micro-chat mode offers: Fast interaction: For simple and fast interactions where users need to receive a concise response at a time, the micro-chat UI view is suitable. It focuses on delivering essential information without overwhelming the user with a full chat interface. Space constraints: In scenarios where the application's user interface has limited space (such as narrow sidebars), the micro-chat UI can be more efficient in utilizing available display space.

[0252] There are four types of micro-conversation interactions:

[0253] 1. Users initiate a chat by clicking a chat icon button located in a position such as the bottom right corner, which expands into a micro-chat style view to greet the user in a context-sensitive way and provide ways to enter queries via text or microphone.

[0254] 2. Automatically activate micro-chat when an event is detected (such as specific content being highlighted on an app page). The app then passes the selected page content to the micro-chat and provides context-sensitive and proactive assistance.

[0255] 3. Developers want to explicitly invoke micro-conversation UI for specific sections when users click buttons such as help or information tooltips.

[0256] 4. Inline Context Widgets for Input Fields: Inline context widgets are components that appear within input fields or near user interactions, providing real-time suggestions and content generation based on context. They assist users as they interact with various elements on the page, guiding them through input processing. For example, when a user focuses on the job description text area, an inline context widget is activated, providing context about the field / element and suggesting actions such as "automatically generate content." Once content is generated, it offers predefined actions to format or refine the output as the user desires, such as improving and restating (refining), shortening, detailing, correcting syntax, or undoing.

[0257] Here is an example of an end-user:

[0258] Excited about the content generation features, Michael decided to use the Copilot SDK to enable an inline context widget for the job description field on the "Create Job Application" page.

[0259] Therefore, starting with the configuration, he used the SDK API to enable the inline context widget, which enables a micro-UI view inlined with the JD text area field. This is activated once the user clicks on the JD text area.

[0260] At this point, the SDK will pass the element's content to the dialog stream when it is invoked. Initially, the element is empty.

[0261] When the stream is invoked, the SDK passes the contents of the text region to the current_job_description parameter.

[0262] JD_Flow responds with a payload describing the state of the context widget:

[0263] • 'Automatic Generation' Action

[0264] • Stop action (disabled)

[0265] Therefore, Michael can now click to automatically generate an action and invoke a stream, which then returns another payload:

[0266] • Generated job description (“streamed”)

[0267] • Free-form message input

[0268] To support micro-conversations on a single page, the concept of multi-turn chat using text input or action buttons within an inline context widget can be introduced as follows: (1) When a user begins to interact with a widget button, all the necessary information is passed in, see the following points. (2) When configuring a widget for a field, the co-pilot SDK allows developers to configure a list of initial button actions when the widget is activated, which can be a static list of actions. When a user begins to interact with the widget by clicking one of the buttons, the LLM robot response includes a new list of dynamic actions that replaces the initial static list of actions. (3) When configuring buttons for a context widget, developers can configure the following properties:

[0269] 1. buttonLabel: The label of the button.

[0270] 2. flowName: The name of the flow to be invoked.

[0271] 3. skillName: Optional, can be used when the copilot skill is DA.

[0272] 4. fieldInputParameters: Key-value pairs, where the key is the name of the stream input parameter, and the value is the name of the field from which its value should be passed as a parameter. For example:

[0273] a.role→jobRoleField

[0274] 5. additionalInputParameters: Key-value pairs, where the key is the name of the stream input parameter and the value is the parameter's value. For example:

[0275] a.language -> 'english'

[0276] (4) When the user clicks the initial button on the context widget, a Copilot SDK method named, such as startWidgetInteraction, is invoked. This method retrieves the button configuration and sends a startThread command message. (The message can be sent directly or via a REST API wrapper.) The bot response text is displayed in the field associated with the widget. The bot response action is displayed as a new list of buttons with the widget. (5) Additional flags can be provided in the button configuration to inform the SDK whether the bot response should be streamed back to the field or to the context widget mini-chat window. (6) After starting the thread using one of the INITIAL actions, the widget button is replaced by an action returned as part of the bot message. These can be implemented using CMM actions, such as postback actions with payloads that can be sent back to the dialogue engine as is, similar to what happens when a button is clicked in a “normal” conversation. The only difference is that the postback message should include the threadId in the channel conversation metadata. When a user navigates away from a page with a context widget and then returns to it, and the fields associated with the widget already have values ​​because the user previously interacted with the widget, then the SDK should display the last set of action buttons returned by the skill from the last interaction with the widget.

[0277] Threading

[0278] Threads are used to allow or maintain the state and context of conversations between the user and the digital assistant (DA). Threading is the maintenance of multiple isolated conversations within the same session between the user and the DA. A thread identifier is the key concept for switching between threads and determining which thread is being used or should be used. Threads are initiated as part of an executable action identified via a GenAI model such as LLM. Alternatively, threads are initiated as part of an executable action directly identified via a UI event. Executable actions have a thread ID configuration, such as thread ID = patient name[-] patient ID when specific to a patient record. Threads maintain state with a conversation or task context for a specific dialogue or flow, which includes all utterances, responses, actions performed, etc. The conversation or task context is stored in a context storage repository associated with the thread ID. The thread manager handles all thread IDs, states, and switching. Threading has the benefits of ensuring that all utterances and actions are grouped based on a given domain (such as the patient identifier) ​​(e.g., the doctor does not incorrectly mix data / information between patients). Threads can also be separated based on multiple domains (such as by patient and by action). One assistive technical challenge is knowing when to switch threads, especially when the utterance doesn't match the context. Therefore, the dialogue can include confirmations sent to the user to ensure they wish to switch threads.

[0279] Specifically, to support side panel chat widgets and multiple micro-chats on a single page, the concept of in-session threads can be implemented as follows: Each context UI widget launches a micro-chat on its own thread. For example, if a user navigates to a specific page to perform a task—Copilot uses a side panel chat supported by a first thread to assist with this first task. And if the user has to switch to another part of the application to perform a second task while completing the first task, the user can get help on a second thread that is separate from the first thread. This allows DA to maintain separate states for different tasks or chats within the same session.

[0280] Threads have a unique name (identifier) ​​that the client should pass in when creating the thread. Each thread has its own flow stack (e.g., when one flow calls another, or when an illogical inference occurs), current flow, and current state. Each thread has its own flow scope context variable (associated with the flow executing within the thread, without introducing a new thread-level context). All threads within a session share the same skill context, user context, and profile context. Threads can be started by specifying a new threadId using the UpdateApplicationContextCommand message, and specific flows can be started. Flow input parameters can be specified to pass additional context from the page. Once a thread is started, conversations within the thread can continue as usual by sending CMM messages, with the added benefit of passing the threadId as part of the channel conversation metadata, in addition to the userId used as the session key. All robot response messages created in response to user messages that include the threadId will include the same threadId in the channel conversation metadata, allowing the Co-Pilot SDK to filter responses based on threads. As the user moves between various tasks or conversations, copilot and DA can switch between threads within the same session.

[0281] In the example of the inline context widget (one of the micro-conversation interaction types), threading can be implemented as follows. A "thread map" is stored on the SessionContext, with the threadId as the key and the ExecutionContext as the value. By default, this map has only one entry for system.mainThread, the internal thread ID of which can be used for "normal" conversation. ThreadPreProcessor and ThreadPostProcessor can then be added to the pipeline in the skill tracking implementation. The preprocessor executes before the conversation engine and selects the correct execution context for the requested thread (system.mainThread if no threadId is specified in the incoming user message) and stores it as the execution context in the session. The postprocessor executes after the conversation engine and stores the updated execution context in the thread map. In this way, the conversation engine can remain unaware of the thread concept.

[0282] It should be understood that while the threaded implementation is described for specific use cases of side panel chat and inline context widgets, threading can also be used in a similar way to maintain isolation between other objects within the same session, such as multiple conversations within an LLM-based DA.

[0283] Copilot's Design and Implementation

[0284] Artificial intelligence has diverse applications, with a notable evolution in the field of digital assistants or chatbots. Initially, many users sought immediate responses through instant messaging or chat platforms. Organizations recognized the potential for engagement and leveraged these platforms to interact with entities such as end users in real-time conversations. However, maintaining real-time communication channels with entities through human service personnel proved costly for organizations. In response to this challenge, digital assistants or chatbots (also known as bots) emerged as solutions that simulate conversations with entities, particularly on the internet. Bots enable entities to interact with users through messaging applications they already use or other applications with messaging capabilities. However, creating bots can be complex, requiring tedious, repetitive, and time-consuming tasks for training, testing, validating, and deploying them.

[0285] To address this challenge, Design-Time (DT) copilot was developed to pipeline the creation of end-to-end job skills and LLM agents for bots (e.g., DA bots), providing a solution for bot or skill developers seeking to improve the efficiency of their development processes through bot development applications (e.g., DA applications). This feature enables the rapid generation of end-to-end job skills / LLM agents and facilitates their review and modification, covering the essential functionality critical to bot development. DT copilot empowers developers to adopt a more task-oriented approach during job skill implementation, allowing them to focus on application requirements without getting bogged down in the functional complexities of the bot platform. For example, consider the process of creating intents. Typically, bot developers must generate intents and add 5 to 10 examples (each intent having at least 12 recommended utterances). After this, they need to create a stream and map the intents to the stream. However, with DT copilot, the repetitive and time-consuming task of generating multiple variations of similar sentences to train the intent model is eliminated. Developers can now automate this process by fully leveraging the intelligence of DT Copilot to skip the tedious step of adding the exact same sentences in various ways. All that robot developers need to provide is the task they intend to accomplish within the skill or intelligent entity. The DT Copilot skill then generates intents, intent utterances, flows, and flow maps. This not only accelerates the skill development lifecycle but also improves the overall productivity of robot developers.

[0286] This functionality is delivered using the aforementioned aspects of copilot, comprising two channels: a chat widget (a type of micro-conversation chat) and a side panel chat. When a developer logs into the UI of the bot development application containing the chat widget, the developer can observe the widgets on each page and activate them to obtain assistance for specific aspects of the application identified on the page. Furthermore, as the developer navigates to each page of the application, the navigation triggers an embedded help menu in the side panel chat to guide the developer through various tasks to be performed in bot development. Advantageously, there is no disconnect between these two channels. When a developer updates something from the chat widget, the modified variable is seamlessly sent to the side panel chat context, ensuring that it does not retain outdated values. This integration guarantees a smooth and consistent user experience across both channels.

[0287] More specifically, when a user communicates utterances or events using the side panel chat, the skill begins with a given stream, such as the 'GetOperation' stream, which uses a machine learning model, such as a generative artificial intelligence model (e.g., LLM), to determine the intent (including actions and artifacts) of the user's utterance or event and stores the intent in variables. Once the intent is resolved, the next step in the process is dynamic, where stored cues are retrieved and a machine learning model, such as a generative artificial intelligence model (e.g., LLM), is invoked. Each of the artifact and action combinations may require different cues. Along with the cues, a pattern (e.g., a JSON pattern or an object pattern) is also retrieved from a resource pack. This pattern has slot information, which is then populated by the machine learning model for a given artifact and action combination. Slot population is the task of filling the independent variable slots in the pattern with parameters (e.g., date / time values) or variations of parameters (e.g., September 1st compared to today or September 1st). Again, each artifact and action may require different information or parameters from the context or the user's utterance. Once the slots in the schema are filled, another machine learning model call is initiated and the machine learning model generates the final schema so that internal service calls can be made to the API (e.g., REST API).

[0288] For example and such Figure 9As shown, a message payload (e.g., a CMM message payload) can initially be received as described herein. The message payload may include: (i) an indication of a utterance or interface action (e.g., an event or user interaction with a UI element), and (ii) an application context. The message payload is associated with an interaction between the user and the application (e.g., a robotics development application). The message payload can be used to trigger a stream, for example, triggering a 'GetOperation' stream that uses a machine learning model to determine the intent (including actions and artifacts) of a user's utterance or event. In some instances, the context variable data structure is generated by storing variables from the application context as at least a portion of the data stored within a context variable data structure to maintain the context associated with the thread (e.g., the application context).

[0289] Then, a machine learning model is used to parse the intent (including the action and artifact). For example, when the user's utterance is 'create intent', the machine learning model can return the action: CREATE and the artifact: INTENT. Retrieve prompts associated with the action: CREATE and the artifact: INTENT. Figure 10 As shown, cues can be used in a stream, and intent resolution is used to obtain the correct cues for the determined intent. In some instances, multiple resource bundle entries may exist, allowing retrieval of one or more cues for each of the operations and artifacts. Cues include executable actions and fields that need to be performed by a system (e.g., a cue generator), and a corresponding pattern for each cue also exists and is retrieved. To obtain patterns, similar techniques can be used to dynamically retrieve the correct pattern from the resource bundle. In some instances, skills may be stored in the resource bundle instead of patterns, where the skill includes the pattern.

[0290] Each service skill call uses a reusable stream to resolve slots, passing hints and patterns defining the slots that need to be populated to a machine learning model. Slot resolution may include using the machine learning model to generate a list of one or more executable actions based on the hints. Each executable action includes an argument slot to be populated. This slot population process includes: deriving one or more parameters (e.g., data and time values, name values, location values, currency values, etc.) from the message payload, data in a context variable data structure, or any combination thereof, and populating the argument slots of one or more executable actions with versions of the parameters that conform to the pattern (e.g., the original version derived from the utterance or context, or a normalized or normalized value of the parameter).

[0291] The machine learning model used to resolve slots validates the schema against a pattern validation framework and ensures that any missing information is prompted to the user. For example, if the current utterance and context include information or parameters for `skillName` and `skillPurpose`, the machine learning model fills in those argument slots in the schema. Otherwise, the machine learning model can return a message to the user to fill in the missing information. During runtime, if the `ForMissingInformation` message is not empty, the machine learning model's event handler can send that message back to the user for additional information or parameters. Once all the necessary slots are filled, the schema can be sent to the machine learning model, and the machine learning model generates the final schema so that internal service calls can be made to management APIs (e.g., REST APIs). Figure 11 As shown in the diagram. Specifically, a machine learning model is invoked to create a schema payload (e.g., a JSON payload) required to generate the DA artifact. An LLM component or similar component can be used to construct the schema payload required for the final invocation of the management API that creates the DA artifact for performing a task associated with the intent (e.g., creating a cancelPizza intent for a skill). The appropriate management API is then invoked via this invocation, and the task associated with the intent is applied. As part of the task execution, another component of the management API or DA (e.g., a dialogue engine) can communicate to the user the output derived from the actions performed to complete the task, or the communication derived from that output.

[0292] In instances where utterances, events, contexts, or any combination thereof conflict with each other, the machine learning model is trained to still determine the user's correct intent. For example, if a user is on the orderPizza intent creation page of a skills development app, and the user informs the DA via copilot that they want to create a "cancelPizza" intent, then the machine learning model should disregard the current context of orderPizza and return cancelPizza as the user's correct intent.

[0293] Figure 12 This is a flowchart of a process 1200 for maintaining the state and context of a conversation between a user and a digital assistant using threads, according to various embodiments. Figure 12 The processing described herein can be implemented using software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores), hardware, or a combination thereof of the corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 12 The processes presented and described below are intended to be illustrative and non-limiting. Although Figure 12The illustrations depict various processing steps that occur in a specific order or sequence, but this is not intended to be limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed at least partially in parallel. Example method 1200 can be derived from the descriptions herein. Figures 1-11 and Figures 13-17 Any device, system, or apparatus illustrated and described may be performed using some or all of its components.

[0294] Processing 1200 can begin at step 1205, in which natural language speech from the user is received (at the digital assistant) during the session between the user and the digital assistant.

[0295] At step 1210, a topic context instance for natural language discourse is obtained.

[0296] At step 1215, a list is generated by the GenAI model. This list includes executable actions based on one or more candidate actions associated with a topic context instance.

[0297] At step 1220, an executable action is performed to produce output. This execution includes: determining that no thread is running within the session associated with the topic context instance, the executable action, or both; in response to determining that no thread is running, creating a thread associated with the topic context instance, the executable action, or both, wherein the thread maintains the topic context instance, the executable action, or both as a context separate from and associated with any other conversation, for a distinctly different conversation including natural language utterances between the user and the digital assistant; and using the thread to perform the executable action to obtain output.

[0298] At step 1225, the output or communication derived from the output is sent to the user.

[0299] In some instances, processing 1200 further includes: at the digital assistant, receiving subsequent natural language utterances from the user during a session between the user and the digital assistant; obtaining a topic context instance for the natural language utterances; generating another list including another executable action by a GenAI model based on one or more candidate actions associated with the topic context instance; executing the other executable action, wherein execution includes: identifying a thread running within the session that is associated with the topic context instance, the executable action, or both; determining, based on an identifier associated with the topic context instance, the executable action, or both, that the other executable action should be executed using the thread; and using the thread to execute the other executable action to obtain another output; and sending the other output or communication derived from the other output to the user.

[0300] In some instances, processing 1200 further includes: at the digital assistant, receiving subsequent natural language utterances from the user during a session between the user and the digital assistant; obtaining a topic context instance for the subsequent natural language utterances; generating, by a GenAI model, another list including another executable action based on one or more candidate actions associated with the topic context instance for the subsequent natural language utterances; and executing said other executable action, wherein execution includes: identifying a thread running within the session that is associated with the topic context instance for the natural language utterances, the executable action, or both; and determining that there is no thread within the session associated with the topic context instance for the subsequent natural language utterances, said other executable action, or both. In response to determining that no thread is running within the session associated with a topic context instance for subsequent natural language utterances, another thread is created having an identifier associated with the topic context instance for subsequent natural language utterances, the other executable action, or both, wherein the other thread maintains the topic context instance for subsequent natural language utterances, the other executable action, or both as a separate context for another distinct conversation between the user and the digital assistant, and performs the other executable action using the other thread to obtain another output; and sends the other output or communication derived from the other output to the user.

[0301] In some instances, processing 1200 further includes: in response to identifying that a thread running within the session is associated with a topic context instance for a natural language utterance, an executable action, or both, and determining that no thread is running within the session associated with a topic context instance for a subsequent natural language utterance, sending an acknowledgment communication to the user requesting confirmation of the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both; and at the digital assistant, receiving confirmation from the user of the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both, wherein in response to determining that no thread is running within the session associated with a topic context instance for the subsequent natural language utterance and receiving the confirmation message, the other thread is created.

[0302] In some instances, processing 1200 further includes: receiving a message payload that includes (i) an indication of a subsequent utterance or event associated with an interaction between the user and the application, and (ii) an indication of an application context; identifying an identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both is the same identifier associated with a topic context instance, an executable action, or both; and processing the message payload using a thread in response to determining that the identifier is the same.

[0303] In some instances, processing 1200 further includes: receiving a message payload comprising (i) an indication of a subsequent utterance or event associated with an interaction between the user and the application, and (ii) an application context; identifying an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both is different from the identifier associated with a topic context instance, an executable action, or both; and in response to determining that the identifier is different, creating another thread having an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both, wherein the other thread maintains the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both as a separate context for another distinctly different conversation between the user and the digital assistant; and processing the message payload using the other thread.

[0304] In some instances, processing 1200 further includes: in response to determining that the identifier is different, sending a confirmation communication to the user requesting confirmation of (i) a subsequent utterance or interface action, (ii) an application context, or (iii) both; and at the digital assistant, receiving confirmation from the user of (i) a subsequent utterance or interface action, (ii) an application context, or (iii) both, wherein in response to determining that the identifier is different and receiving the confirmation message, the other thread is created.

[0305] Example:

[0306] (A) A computer-implemented method includes: receiving natural language utterances from a user during a session between a user and a digital assistant at a digital assistant; obtaining a topic context instance for the natural language utterances; generating a list of executable actions by a generative artificial intelligence (GenAI) model based on one or more candidate actions associated with the topic context instance; executing the executable actions to produce output, wherein the execution includes: determining that no thread associated with the topic context instance, the executable actions, or both is running within the session; in response to determining that no thread is running, creating a thread associated with the topic context instance, the executable actions, or both, wherein the thread maintains the topic context instance, the executable actions, or both as a context separate from and associated with any other conversation for a distinctly different conversation including the natural language utterances between the user and the digital assistant; and using the thread to execute the executable actions to obtain output; and sending the output or communication derived from the output to the user.

[0307] (B) The computer implementation of embodiment (A) further includes: at a digital assistant, receiving subsequent natural language utterances from the user during a session between the user and the digital assistant; obtaining a topic context instance for the natural language utterances; generating another list including another executable action by a GenAI model based on one or more candidate actions associated with the topic context instance; executing the other executable action, wherein execution includes: identifying a thread running within the session associated with the topic context instance, the executable action, or both; determining, based on an identifier associated with the topic context instance, the executable action, or both, that the other executable action should be executed using the thread; and using the thread to execute the other executable action to obtain another output; and sending the other output or communication derived from the other output to the user.

[0308] (C) The computer implementation of embodiment (A) further includes: at a digital assistant, receiving subsequent natural language utterances from a user during a session between the user and the digital assistant; obtaining a topic context instance for the subsequent natural language utterances; generating, by a GenAI model, another list including another executable action based on one or more candidate actions associated with the topic context instance for the subsequent natural language utterances; and executing the other executable action, wherein execution includes: identifying a thread running within the session associated with the topic context instance for the natural language utterances, the executable action, or both; determining that there is no associated topic context instance for the subsequent natural language utterances, the other executable action, or both within the session. The associated threads are running, and in response to determining that no thread is running within the session associated with a topic context instance for subsequent natural language utterances, another thread is created having an identifier associated with the topic context instance for subsequent natural language utterances, the other executable action, or both, wherein the other thread maintains the topic context instance for subsequent natural language utterances, the other executable action, or both as a separate context for another distinct conversation between the user and the digital assistant, and performs the other executable action using the other thread to obtain another output; and sends the other output or communication derived from the other output to the user.

[0309] (D) A computer-implemented method of embodiment (C): in response to identifying that a thread running in a session is associated with a topic context instance for a natural language utterance, an executable action, or both, and determining that no thread is running in the session associated with a topic context instance for a subsequent natural language utterance, an acknowledgment communication requesting confirmation of the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both; and at the digital assistant, receiving confirmation from the user of the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both, wherein in response to determining that no thread is running in the session associated with a topic context instance for the subsequent natural language utterance and receiving the confirmation message, the other thread is created.

[0310] (E) The computer-implemented method of embodiment (A) further includes: receiving a message payload comprising (i) an indication of a subsequent utterance or event associated with an interaction between a user and an application, and (ii) an application context; identifying an identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both is the same identifier associated with a topic context instance, an executable action, or both; and processing the message payload using a thread in response to determining that the identifier is the same.

[0311] (F) The computer implementation of embodiment (A) further includes: receiving a message payload comprising: (i) an indication of a subsequent utterance or event associated with an interaction between a user and an application, and (ii) an application context; identifying an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both is different from an identifier associated with a topic context instance, an executable action, or both; and in response to determining that the identifier is different, creating another thread having an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both, wherein the other thread maintains the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both as a separate context for another distinctly different conversation between the user and the digital assistant; and processing the message payload using the other thread.

[0312] (G) The computer implementation of embodiment (F) further includes: in response to determining that the identifier is different, sending a confirmation communication to the user requesting confirmation of (i) a subsequent statement or interface action, (ii) an application context, or (iii) both; and at the digital assistant, receiving confirmation from the user of (i) a subsequent statement or interface action, (ii) an application context, or (iii) both, wherein in response to determining that the identifier is different and receiving the confirmation statement, the other thread is created.

[0313] (H) A system includes: one or more processors; and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations including: receiving, at a digital assistant, natural language utterances from a user during a session between a user and the digital assistant; obtaining a topic context instance for the natural language utterances; generating, by a generative artificial intelligence (GenAI) model, a list of executable actions based on one or more candidate actions associated with the topic context instance; and performing the executable actions to produce output, wherein the execution includes: determining that no thread is running within the session associated with the topic context instance, the executable actions, or both; in response to determining that no thread is running, creating a thread associated with the topic context instance, the executable actions, or both, wherein the thread maintains the topic context instance, the executable actions, or both as a context separate from and associated with any other conversation for a distinctly different conversation including natural language utterances between the user and the digital assistant; and using the thread to perform the executable actions to obtain output; and sending the output or communication derived from the output to the user.

[0314] (I) The operation of embodiment (H) further includes: at the digital assistant, receiving subsequent natural language utterances from the user during a session between the user and the digital assistant; obtaining a topic context instance for the natural language utterances; generating another list including another executable action by the GenAI model based on one or more candidate actions associated with the topic context instance; executing the other executable action, wherein execution includes: identifying a thread running within the session that is associated with the topic context instance, the executable action, or both; determining, based on an identifier associated with the topic context instance, the executable action, or both, that the other executable action should be executed using the thread; and using the thread to execute the other executable action to obtain another output; and sending the other output or communication derived from the other output to the user.

[0315] (J) The operation of embodiment (H) further includes: at the digital assistant, receiving subsequent natural language utterances from the user during a session between the user and the digital assistant; obtaining a topic context instance for the subsequent natural language utterances; generating, by the GenAI model, another list including another executable action based on one or more candidate actions associated with the topic context instance for the subsequent natural language utterances; and executing the other executable action, wherein execution includes: identifying a thread running within the session that is associated with the topic context instance for the natural language utterances, the executable action, or both; determining that there is no thread within the session associated with the topic context instance for the subsequent natural language utterances, the other executable action, or both. While the associated thread is running, in response to determining that no thread is running within the session associated with a topic context instance for subsequent natural language utterances, another thread is created having an identifier associated with the topic context instance for subsequent natural language utterances, the other executable action, or both, wherein the other thread maintains the topic context instance for subsequent natural language utterances, the other executable action, or both as a separate context for another distinct conversation between the user and the digital assistant, and performs the other executable action using the other thread to obtain another output; and sends the other output or communication derived from the other output to the user.

[0316] (K) The operation of embodiment (J) further includes: in response to identifying that a thread running in the session is associated with a topic context instance for a natural language utterance, an executable action, or both, and determining that no thread is running in the session associated with a topic context instance for a subsequent natural language utterance, sending a confirmation communication to the user requesting confirmation of the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both; and at the digital assistant, receiving a confirmation utterance from the user confirming the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both, wherein in response to determining that no thread is running in the session associated with a topic context instance for the subsequent natural language utterance and receiving the confirmation utterance, the other thread is created.

[0317] (L) The operation of embodiment (H) further includes: receiving a message payload comprising (i) an indication of a subsequent utterance or event associated with an interaction between the user and the application, and (ii) an application context; identifying an identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both is the same identifier associated with a topic context instance, an executable action, or both; and processing the message payload using a thread in response to determining that the identifier is the same.

[0318] (M) The operation of embodiment (H) further includes: receiving a message payload comprising: (i) an indication of a subsequent utterance or event associated with an interaction between the user and the application, and (ii) an application context; identifying an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both is different from the identifier associated with a topic context instance, an executable action, or both; and in response to determining that the identifier is different, creating another thread having an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both, wherein the other thread maintains the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both as a separate context for another distinctly different conversation between the user and the digital assistant; and processing the message payload using the other thread.

[0319] (N) The operation of embodiment (M) further includes: in response to determining that the identifier is different, sending a confirmation communication to the user requesting confirmation of (i) a subsequent utterance or interface action, (ii) an application context, or (iii) both; and at the digital assistant, receiving confirmation from the user of (i) a subsequent utterance or interface action, (ii) an application context, or (iii) both, wherein in response to determining that the identifier is different and receiving the confirmation, the other thread is created.

[0320] (O) A non-transitory computer-readable medium storing one or more instructions, which, when executed by one or more processors, cause the one or more processors to perform operations including: at a digital assistant, receiving natural language utterances from a user during a session between the user and the digital assistant; obtaining a topic context instance for the natural language utterances; generating a list of executable actions by a generative artificial intelligence (GenAI) model based on one or more candidate actions associated with the topic context instance; and executing the executable actions to produce output, wherein the execution includes: determining that no thread is running within the session associated with the topic context instance, the executable actions, or both; in response to determining that no thread is running, creating a thread associated with the topic context instance, the executable actions, or both, wherein the thread maintains the topic context instance, the executable actions, or both as a context separate from and associated with any other conversation for a distinctly different conversation including the natural language utterances between the user and the digital assistant; and using the thread to execute the executable actions to obtain output; and sending the output or communications derived from the output to the user.

[0321] (P) The operation of embodiment (O) further includes: at the digital assistant, receiving subsequent natural language utterances from the user during a session between the user and the digital assistant; obtaining a topic context instance for the natural language utterances; generating another list including another executable action by the GenAI model based on one or more candidate actions associated with the topic context instance; executing the other executable action, wherein execution includes: identifying a thread running within the session that is associated with the topic context instance, the executable action, or both; determining, based on an identifier associated with the topic context instance, the executable action, or both, that the other executable action should be executed using the thread; and executing the other executable action using the thread to obtain another output; and sending the other output or communication derived from the other output to the user.

[0322] (Q) The operation of embodiment (O) further includes: at the digital assistant, receiving subsequent natural language utterances from the user during a session between the user and the digital assistant; obtaining a topic context instance for the subsequent natural language utterances; generating, by the GenAI model, another list including another executable action based on one or more candidate actions associated with the topic context instance for the subsequent natural language utterances; and executing said other executable action, wherein execution includes: identifying a thread running within the session that is associated with the topic context instance for the natural language utterances, the executable action, or both; determining that there is no thread within the session associated with the topic context instance for the subsequent natural language utterances, said other executable action, or both. While the associated thread is running, in response to determining that no thread is running within the session associated with a topic context instance for subsequent natural language utterances, another thread is created having an identifier associated with the topic context instance for subsequent natural language utterances, the other executable action, or both, wherein the other thread maintains the topic context instance for subsequent natural language utterances, the other executable action, or both as a separate context for another distinct conversation between the user and the digital assistant, and performs the other executable action using the other thread to obtain another output; and sends the other output or communication derived from the other output to the user.

[0323] (R) The operation of embodiment (Q) further includes: in response to identifying that a thread running in the session is associated with a topic context instance for a natural language utterance, an executable action, or both, and determining that no thread is running in the session associated with a topic context instance for a subsequent natural language utterance, sending an acknowledgment communication to the user requesting confirmation of the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both; and at the digital assistant, receiving an acknowledgment from the user confirming the subsequent natural language utterance, the topic context instance for the subsequent natural language utterance, or both, wherein, in response to determining that no thread is running in the session associated with a topic context instance for the subsequent natural language utterance and receiving the acknowledgment, the other thread is created.

[0324] (S) The operation of embodiment (O) further includes: receiving a message payload comprising (i) an indication of a subsequent utterance or event associated with an interaction between the user and the application, and (ii) an application context; identifying an identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of the subsequent utterance or interface action, (ii) the application context, or (iii) both is the same identifier associated with a topic context instance, an executable action, or both; and processing the message payload using a thread in response to determining that the identifier is the same.

[0325] (T) The operation of embodiment (O) further includes: receiving a message payload comprising: (i) an indication of a subsequent utterance or event associated with an interaction between the user and the application, and (ii) an application context; identifying an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both; determining that the identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both is different from the identifier associated with a topic context instance, an executable action, or both; and in response to determining that the identifier is different, creating another thread having an identifier associated with the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both, wherein the other thread maintains the indication of (i) a subsequent utterance or interface action, (ii) the application context, or (iii) both as a separate context for another distinct conversation between the user and the digital assistant; and processing the message payload using the other thread.

[0326] (U) The operation of embodiment (T) further includes: in response to determining that the identifier is different, sending a confirmation communication to the user requesting confirmation of (i) a subsequent utterance or interface action, (ii) an application context, or (iii) both; and at the digital assistant, receiving confirmation from the user of (i) a subsequent utterance or interface action, (ii) an application context, or (iii) both, wherein in response to determining that the identifier is different and receiving the confirmation, the other thread is created.

[0327] Explanatory System

[0328] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, and clustering software, etc.). Therefore, because these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0329] In some instances, IaaS customers can access resources and services over a wide area network (WAN) such as the internet, and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0330] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0331] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for disposition (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0332] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0333] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some instances, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0334] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, one or more inbound / outbound traffic group rules may also be provided to define how inbound / outbound traffic to the network and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provided. The infrastructure can evolve incrementally as more and / or more infrastructure elements are expected and added.

[0335] In some instances, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some instances, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0336] Figure 13 This is a block diagram 1300 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service operator 1302 may be communicatively coupled to a secure host lease 1304, which may include a virtual cloud network (VCN) 1306 and a secure host subnet 1308. In some examples, service operator 1302 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software (such as Microsoft Windows Mobile®) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and supporting the Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, client computing devices may be general-purpose personal computers, including, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. Client computing devices can be workstation computers running a variety of commercially available UNIX® or UNIX-like operating systems, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, client computing devices can be any other electronic device, such as thin client computers, internet-enabled gaming systems (e.g., Microsoft Xbox game consoles with or without Kinect® gesture input devices), and / or personal messaging devices capable of communicating over a network that can access VCN 1306 and / or the internet.

[0337] VCN 1306 may include a local peering gateway (LPG) 1310, which may be communicatively coupled to a secure shell (SSH) VCN 1312 via an LPG 1310 contained in an SSH VCN 1312. SSH VCN 1312 may include an SSH subnet 1314, and SSH VCN 1312 may be communicatively coupled to a control plane VCN 1316 via an LPG 1310 contained in a control plane VCN 1316. Furthermore, SSH VCN 1312 may be communicatively coupled to a data plane VCN 1318 via an LPG 1310. Control plane VCN 1316 and data plane VCN 1318 may be contained within a service lease 1319 that may be owned and / or operated by an IaaS provider.

[0338] The control plane VCN 1316 may include a control plane demilitarized zone (DMZ) layer 1320 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 1320 may include one or more load balancer (LB) subnets 1322, a control plane application layer 1324 that may include one or more application subnets 1326, and a control plane data layer 1328 that may include one or more database (DB) subnets 1330 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 1322 contained in the control plane DMZ layer 1320 may be communicatively coupled to one or more application subnets 1326 contained in the control plane application layer 1324 and an Internet gateway 1334 that may be contained in the control plane VCN 1316. The application subnets 1326 may be communicatively coupled to one or more DB subnets 1330 contained in the control plane data layer 1328, as well as a service gateway 1336 and a Network Address Translation (NAT) gateway 1338. The control plane VCN 1316 may include the service gateway 1336 and the NAT gateway 1338.

[0339] The control plane VCN 1316 may include a data plane mirror application layer 1340, which may include one or more application subnets 1326. The one or more application subnets 1326 included in the data plane mirror application layer 1340 may include a virtual network interface controller (VNIC) 1342 capable of executing a compute instance 1344. The compute instance 1344 may communicatively couple the one or more application subnets 1326 of the data plane mirror application layer 1340 to the one or more application subnets 1326 that may be included in the data plane application layer 1346.

[0340] Data plane VCN 1318 may include data plane application layer 1346, data plane DMZ layer 1348, and data plane data layer 1350. Data plane DMZ layer 1348 may include one or more LB subnets 1322 communicatively coupled to one or more application subnets 1326 of data plane application layer 1346 and Internet gateway 1334 of data plane VCN 1318. One or more application subnets 1326 may be communicatively coupled to service gateway 1336 and NAT gateway 1338 of data plane VCN 1318. Data plane data layer 1350 may also include one or more DB subnets 1330 communicatively coupled to one or more application subnets 1326 of data plane application layer 1346.

[0341] Internet gateway 1334 of control plane VCN 1316 and data plane VCN 1318 can be communicatively coupled to metadata management service 1352, and metadata management service 1352 can be communicatively coupled to public internet 1354. Public internet 1354 can be communicatively coupled to NAT gateway 1338 of control plane VCN 1316 and data plane VCN 1318. Service gateway 1336 of control plane VCN 1316 and data plane VCN 1318 can be communicatively coupled to cloud service 1356.

[0342] In some examples, the service gateway 1336, either the control plane VCN 1316 or the data plane VCN 1318, can make application programming interface (API) calls to the cloud service 1356 without traversing the public internet 1354. API calls from the service gateway 1336 to the cloud service 1356 can be unidirectional: the service gateway 1336 can make API calls to the cloud service 1356, and the cloud service 1356 can send requested data to the service gateway 1336. However, the cloud service 1356 may not initiate API calls to the service gateway 1336.

[0343] In some examples, secure host lease 1304 can be directly connected to service lease 1319, which would otherwise be isolated. Secure host subnet 1308 can communicate with SSH subnet 1314 via LPG 1310, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 1308 to SSH subnet 1314 allows secure host subnet 1308 to access other entities within service lease 1319.

[0344] Control plane VCN 1316 may allow users of service lease 1319 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 1316 may be deployed or otherwise used in data plane VCN 1318. In some examples, control plane VCN 1316 may be isolated from data plane VCN 1318, and the data plane mirror application layer 1340 of control plane VCN 1316 may communicate with the data plane application layer 1346 of data plane VCN 1318 via VNIC 1342, which may be included in both the data plane mirror application layer 1340 and the data plane application layer 1346.

[0345] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 1354, which can transmit requests to the metadata management service 1352. The metadata management service 1352 can transmit requests to the control plane VCN 1316 via internet gateway 1334. Requests can be received by one or more LB subnets 1322 contained in the control plane DMZ layer 1320. The LB subnets 1322 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 1326 contained in the control plane application layer 1324. If the request is validated and requires a call to the public internet 1354, the call to the public internet 1354 can be transmitted to a NAT gateway 1338 that can make calls to the public internet 1354. The request may expect stored metadata to be stored in one or more DB subnets 1330.

[0346] In some examples, the data plane mirroring application layer 1340 can facilitate direct communication between the control plane VCN 1316 and the data plane VCN 1318. For example, it may be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 1318. Through VNIC 1342, the control plane VCN 1316 can communicate directly with the resources contained in the data plane VCN 1318, and thus can perform configuration changes, updates, or other appropriate modifications.

[0347] In some embodiments, the control plane VCN 1316 and data plane VCN 1318 may be included in service lease 1319. In this case, the system's users or customers may not own or operate the control plane VCN 1316 or data plane VCN 1318. Alternatively, the IaaS provider may own or operate both the control plane VCN 1316 and data plane VCN 1318, and both planes may be included in service lease 1319. This embodiment can enable the isolation of networks that might prevent users or customers from interacting with resources from other users or customers. Furthermore, this embodiment can allow users or customers of the system to privately store databases without relying on the public Internet 1354, which may not have the desired level of threat prevention for storage.

[0348] In other embodiments, one or more LB subnets 1322 included in the control plane VCN 1316 may be configured to receive signals from the service gateway 1336. In this embodiment, the control plane VCN 1316 and the data plane VCN 1318 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 1354. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on service lease 1319, which may be isolated from the public internet 1354.

[0349] Figure 14 This is a block diagram 1400 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1402 (e.g., Figure 13 The service provider 1302 can communicatively couple to the secure host lease 1404 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1404 may include a virtual cloud network (VCN) 1406 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1408 (e.g., Figure 13 The secure host subnet 1308). VCN 1406 may include a local peering gateway (LPG) 1410 (e.g., Figure 13 The LPG 1310, which can be communicatively coupled to the Secure Shell (SSH) VCN 1412 (e.g., LPG 1310 contained in the SSH VCN 1412) Figure 13 SSH VCN 1312). SSH VCN 1412 can include SSH subnet 1414 (e.g., Figure 13SSH subnet 1314), and SSH VCN 1412 can be communicatively coupled to control plane VCN 1416 via LPG 1410 contained in control plane VCN 1416 (e.g., Figure 13 Control plane VCN 1316). Control plane VCN 1416 may be included in service lease 1419 (e.g., Figure 13 In the service lease 1319), and the data plane VCN 1418 (e.g., Figure 13 The data plane VCN 1318 may be included in a customer lease 1421 that may be owned or operated by a user or customer of the system.

[0350] Control plane VCN 1416 may include control plane DMZ layer 1420 (e.g., Figure 13 The control plane DMZ layer 1320), which may include one or more LB subnets 1422 (e.g., Figure 13 (one or more) LB subnets 1322), may include (one or more) application subnets 1426 (e.g., Figure 13 The control plane application layer 1424 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324 may include one or more database (DB) subnets 1430 (e.g., similar to...). Figure 13 The control plane data layer 1428 of (one or more) DB subnets 1330 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1422 contained in the control plane DMZ layer 1420 can be communicatively coupled to one or more application subnets 1426 contained in the control plane application layer 1424 and an Internet gateway 1434 that can be contained in the control plane VCN 1416 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1426 can communicatively couple to DB subnet(s) 1430 contained in control plane data layer 1428 and service gateway 1436 (e.g., Figure 13 Service gateway 1336) and Network Address Translation (NAT) gateway 1438 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1416 may include the service gateway 1436 and the NAT gateway 1438.

[0351] The control plane VCN 1416 may include a data plane mirror of the application layer 1440, which may include one or more application subnets 1426 (e.g., Figure 13The data plane mirror application layer 1340). One or more application subnets 1426 contained in the data plane mirror application layer 1440 may include computational instances 1444 (e.g., similar to...). Figure 13 The virtual network interface controller (VNIC) 1442 (e.g., the VNIC of 1342) of the computing instance 1344. The computing instance 1444 may facilitate the mirroring of the application subnet(s) 1426 of the application layer 1440 in the data plane and may be included in the application layer 1446 in the data plane (e.g., Figure 13 Communication between one or more application subnets 1426 in the data plane application layer 1346 via VNIC 1442 contained in the data plane mirror application layer 1440 and VNIC 1442 contained in the data plane application layer 1446.

[0352] The Internet gateway 1434, included in the control plane VCN 1416, can be communicatively coupled to the metadata management service 1452 (e.g., Figure 13 Metadata management service 1352), which can communicatively couple to the public Internet 1454 (e.g., Figure 13 The public internet 1454 can communicatively couple to a NAT gateway 1438 contained in a control plane VCN 1416. The service gateway 1436 contained in the control plane VCN 1416 can communicatively couple to a cloud service 1456 (e.g., ...). Figure 13 Cloud services 1356).

[0353] In some examples, data plane VCN 1418 may be included in customer lease 1421. In this case, the IaaS provider may provide control plane VCN 1416 for each customer, and the IaaS provider may set up a unique compute instance 1444 for each customer, included in service lease 1419. Each compute instance 1444 may allow communication between control plane VCN 1416 included in service lease 1419 and data plane VCN 1418 included in customer lease 1421. Compute instance 1444 may allow resources provisioned in control plane VCN 1416 included in service lease 1419 to be deployed or otherwise used in data plane VCN 1418 included in customer lease 1421.

[0354] In other examples, an IaaS provider's customer may have a database residing in customer lease 1421. In this example, control plane VCN 1416 may include data plane mirror application layer 1440, which may include one or more application subnets 1426. Data plane mirror application layer 1440 may reside in data plane VCN 1418, but may not reside in data plane VCN 1418. That is, data plane mirror application layer 1440 may have access to customer lease 1421, but may not reside in data plane VCN 1418 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 1440 may be configured to invoke data plane VCN 1418, but may not be configured to invoke any entity contained in control plane VCN 1416. Customers may expect to deploy or otherwise use resources provisioned in the control plane VCN 1416 in the data plane VCN 1418, and the data plane mirroring application layer 1440 can facilitate the customer's expected deployment or other use of resources.

[0355] In some embodiments, an IaaS provider's customer may apply filters to data plane VCN 1418. In this embodiment, the customer may determine what data plane VCN 1418 can access, and the customer may restrict access from data plane VCN 1418 to the public internet 1454. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 1418's access to any external networks or databases. Applying filters and controls to data plane VCN 1418 contained in customer lease 1421 helps isolate data plane VCN 1418 from other customers and the public internet 1454.

[0356] In some embodiments, cloud service 1456 may be invoked by service gateway 1436 to access services that may not exist on public internet 1454, control plane VCN 1416, or data plane VCN 1418. The connection between cloud service 1456 and control plane VCN 1416 or data plane VCN 1418 may not be real-time or continuous. Cloud service 1456 may reside on different networks owned or operated by an IaaS provider. Cloud service 1456 may be configured to receive calls from service gateway 1436 and may be configured not to receive calls from public internet 1454. Some cloud services 1456 may be isolated from other cloud services 1456, and control plane VCN 1416 may be isolated from cloud services 1456 that may not be in the same region as control plane VCN 1416. For example, control plane VCN 1416 may be located in "Region 1," and cloud service "Deployment 13" may be located in both "Region 1" and "Region 2." If the service gateway 1436, contained in the control plane VCN 1416 located in region 1, makes a call to deployment 13, then that call can be transmitted to deployment 13 in region 1. In this example, the control plane VCN 1416 or deployment 13 in region 1 may not be communicatively coupled to or otherwise communicate with deployment 13 in region 2.

[0357] Figure 15 This is a block diagram 1500 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1502 (e.g., Figure 13 The service provider 1302) can communicatively couple to the secure host lease 1504 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1504 may include a virtual cloud network (VCN) 1506 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1508 (e.g., Figure 13 The secure host subnet 1308). VCN 1506 can include LPG 1510 (e.g., Figure 13 The LPG 1310), which can be communicatively coupled to the SSH VCN 1512 via the LPG 1510 included in the SSH VCN 1512 (e.g., Figure 13 SSH VCN 1312). SSH VCN 1512 can include SSH subnet 1514 (e.g., Figure 13 SSH subnet 1314), and SSH VCN 1512 can be communicatively coupled to control plane VCN 1516 via LPG 1510 included in control plane VCN 1516 (e.g., Figure 13The control plane VCN 1316) and coupled to the data plane VCN 1518 via the LPG 1510 contained in the data plane VCN 1518 (e.g., Figure 13 Data plane 1318). Control plane VCN 1516 and data plane VCN 1518 may be included in service lease 1519 (e.g., Figure 13 In the service rental (1319).

[0358] The control plane VCN 1516 may include one or more load balancer (LB) subnets 1522 (e.g., Figure 13 The control plane DMZ layer 1520 of (one or more) LB subnets 1322) (e.g., Figure 13 The control plane DMZ layer 1320 may include one or more application subnets 1526 (e.g., similar to...). Figure 13 The control plane application layer 1524 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324), and may include (one or more) DB subnets 1530, and the control plane data layer 1528 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1522 contained in the control plane DMZ layer 1520 can be communicatively coupled to one or more application subnets 1526 contained in the control plane application layer 1524 and an Internet gateway 1534 that can be contained in the control plane VCN 1516 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1526 can communicatively couple to DB subnet(s) 1530 contained in control plane data layer 1528 and service gateway 1536 (e.g., Figure 13 The service gateway) and Network Address Translation (NAT) gateway 1538 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1516 may include the service gateway 1536 and the NAT gateway 1538.

[0359] Data plane VCN 1518 may include data plane application layer 1546 (e.g., Figure 13 Data plane application layer 1346), data plane DMZ layer 1548 (e.g., Figure 13 Data plane DMZ layer 1348), and data plane data layer 1550 (e.g., Figure 13The data plane data layer 1350. The data plane DMZ layer 1548 may include one or more trusted application subnets 1560 and one or more untrusted application subnets 1562 that are communicatively coupled to the data plane application layer 1546, and one or more LB subnets 1522 of the Internet gateway 1534 contained in the data plane VCN 1518. The one or more trusted application subnets 1560 may be communicatively coupled to the service gateway 1536, the NAT gateway 1538 contained in the data plane VCN 1518, and one or more DB subnets 1530 contained in the data plane data layer 1550. The one or more untrusted application subnets 1562 may be communicatively coupled to the service gateway 1536 contained in the data plane VCN 1518 and the one or more DB subnets 1530 contained in the data plane data layer 1550. The data plane data layer 1550 may include one or more DB subnets 1530 that can be communicatively coupled to the service gateway 1536 contained in the data plane VCN 1518.

[0360] One or more untrusted application subnets 1562 may include one or more primary VNICs 1564(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1566(1)-(N). Each tenant VM 1566(1)-(N) may be communicatively coupled to a corresponding application subnet 1567(1)-(N) that may be contained in a corresponding container egress VCN 1568(1)-(N), which may be contained in a corresponding customer lease 1570(1)-(N). A corresponding secondary VNIC 1572(1)-(N) may facilitate communication between one or more untrusted application subnets 1562 contained in data plane VCN 1518 and application subnets contained in container egress VCN 1568(1)-(N). Each container exit VCN 1568(1)-(N) may include a NAT gateway 1538, which can communicatively couple to the public Internet 1554 (e.g., Figure 13 The public internet (1354).

[0361] Internet gateway 1534, contained in control plane VCN 1516 and data plane VCN 1518, can be communicatively coupled to metadata management service 1552 (e.g., Figure 13A metadata management system 1552 can be communicatively coupled to the public internet 1554. The public internet 1554 can be communicatively coupled to a NAT gateway 1538 contained in a control plane VCN 1516 and a data plane VCN 1518. A service gateway 1536 contained in a control plane VCN 1516 and a data plane VCN 1518 can be communicatively coupled to a cloud service 1556.

[0362] In some embodiments, the data plane VCN 1518 may be integrated with the customer lease 1570. Such integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support may be expected during code execution. Customers may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider may determine whether to run the code provided by the customer to the IaaS provider.

[0363] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality attached to the data plane application layer 1546. The code running this functionality may execute in VMs 1566(1)-(N) and may not be configured to run anywhere else on the data plane VCN 1518. Each VM 1566(1)-(N) may be connected to a customer lease 1570. The corresponding container 1571(1)-(N) contained in VMs 1566(1)-(N) may be configured to run the code. In this case, dual isolation may exist (e.g., container 1571(1)-(N) runs the code, where container 1571(1)-(N) may be contained in at least one VM 1566(1)-(N) contained in untrusted application subnet 1562), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Containers 1571(1)-(N) may be communicatively coupled to customer lease 1570 and may be configured to transmit or receive data from customer lease 1570. Containers 1571(1)-(N) may not be configured to transmit or receive data from any other entity in data plane VCN 1518. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of containers 1571(1)-(N).

[0364] In some embodiments, one or more trusted application subnets 1560 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 1560 may be communicatively coupled to one or more database subnets 1530 and configured to perform CRUD operations in one or more database subnets 1530. One or more untrusted application subnets 1562 may be communicatively coupled to one or more database subnets 1530, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 1530. Containers 1571(1)-(N) that may be contained in each customer's VM 1566(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 1530.

[0365] In other embodiments, the control plane VCN 1516 and the data plane VCN 1518 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 1516 and the data plane VCN 1518. However, communication can occur indirectly through at least one method. The LPG 1510 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 1516 and the data plane VCN 1518. In another example, either the control plane VCN 1516 or the data plane VCN 1518 may invoke the cloud service 1556 via the service gateway 1536. For example, an invocation of the cloud service 1556 from the control plane VCN 1516 may include a request for a service that can communicate with the data plane VCN 1518.

[0366] Figure 16 This is a block diagram 1600 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1602 (e.g., Figure 13 The service provider 1302) can communicatively couple to the secure host lease 1604 (e.g., Figure 13 Secure hosting lease 1304), the secure hosting lease 1604 may include a virtual cloud network (VCN) 1606 (e.g., Figure 13 VCN 1306) and Secure Host Subnet 1608 (e.g., Figure 13 The secure host subnet 1308). VCN 1606 can include LPG 1610 (e.g., Figure 13 The LPG 1310), the LPG 1610 can be accessed via SSH VCN 1612 (e.g., LPG 1310), Figure 13The LPG 1610 in SSH VCN 1312 is communicatively coupled to SSH VCN 1612. SSH VCN 1612 may include SSH subnet 1614 (e.g., Figure 13 SSH subnet 1314), and SSH VCN 1612 can be communicatively coupled to control plane VCN 1616 via LPG 1610 included in control plane VCN 1616 (e.g., Figure 13 The control plane VCN 1316) and coupled to the data plane VCN 1618 via the LPG 1610 contained in the data plane VCN 1618 (e.g., Figure 13 Data plane 1318). Control plane VCN 1616 and data plane VCN 1618 may be included in service lease 1619 (e.g., Figure 13 In the service rental (1319).

[0367] The control plane VCN 1616 may include (one or more) LB subnets 1622 (e.g., Figure 13 The control plane DMZ layer 1620 of (one or more) LB subnets 1322) (e.g., Figure 13 The control plane DMZ layer 1320 may include one or more application subnets 1626 (e.g., Figure 13 The control plane application layer 1624 of (one or more) application subnets 1326 (e.g., Figure 13 The control plane application layer 1324 may include (one or more) DB subnets 1630 (e.g., Figure 15 The control plane data layer 1628 of (one or more) DB subnets 1530 (e.g., Figure 13 The control plane data layer 1328). One or more LB subnets 1622 contained in the control plane DMZ layer 1620 can be communicatively coupled to one or more application subnets 1626 contained in the control plane application layer 1624 and an Internet gateway 1634 that can be contained in the control plane VCN 1616 (e.g., Figure 13 Internet gateway 1334), and application subnet(s) 1626 can communicatively couple to DB subnet(s) 1630 contained in control plane data layer 1628 and service gateway 1636 (e.g., Figure 13 The service gateway) and Network Address Translation (NAT) gateway 1638 (e.g., Figure 13 (NAT gateway 1338). The control plane VCN 1616 may include the service gateway 1636 and the NAT gateway 1638.

[0368] The data plane VCN 1618 may include the data plane application layer 1646 (e.g., Figure 13 Data plane application layer 1346), data plane DMZ layer 1648 (e.g., Figure 13 Data plane DMZ layer 1348), and data plane data layer 1650 (e.g., Figure 13 The data plane data layer 1350). The data plane DMZ layer 1648 may include one or more trusted application subnets 1660 that can be communicatively coupled to the data plane application layer 1646 (e.g., Figure 15 (one or more) trusted application subnets 1560) and (one or more) untrusted application subnets 1662 (e.g., Figure 15 The data plane includes one or more untrusted application subnets 1562 and one or more LB subnets 1622 of Internet gateway 1634 contained in data plane VCN 1618. One or more trusted application subnets 1660 may communicatively couple to service gateway 1636, NAT gateway 1638, and DB subnets 1630 contained in data plane VCN 1618. One or more untrusted application subnets 1662 may communicatively couple to service gateway 1636 and DB subnets 1630 contained in data plane VCN 1618 and data plane data layer 1650, respectively. Data plane data layer 1650 may include one or more DB subnets 1630 that may communicatively couple to service gateway 1636 contained in data plane VCN 1618.

[0369] One or more untrusted application subnets 1662 may include a primary VNIC 1664(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1666(1)-(N) residing within one or more untrusted application subnets 1662. Each tenant VM 1666(1)-(N) may run code in a corresponding container 1667(1)-(N) and is communicatively coupled to an application subnet 1626 that may be contained in a data plane application layer 1646 contained in a container egress VCN 1668. A corresponding secondary VNIC 1672(1)-(N) may facilitate communication between one or more untrusted application subnets 1662 contained in a data plane VCN 1618 and the application subnets contained in a container egress VCN 1668. The container egress VCN may include a public internet 1654 (e.g., Figure 13 The public internet (1354) uses NAT gateway 1638.

[0370] Internet gateway 1634, contained in control plane VCN 1616 and data plane VCN 1618, can be communicatively coupled to metadata management service 1652 (e.g., Figure 13 The metadata management system 1352, which is communicatively coupled to the public internet 1654, is also communicatively coupled to a NAT gateway 1638 contained in a control plane VCN 1616 and a data plane VCN 1618. A service gateway 1636 contained in both the control plane VCN 1616 and the data plane VCN 1618 is communicatively coupled to a cloud service 1656.

[0371] In some examples, Figure 16 The architecture shown in the block diagram 1600 can be considered as... Figure 15 This is an exception to the pattern shown in the architecture of block diagram 1500, and this pattern may be what the IaaS provider's customers expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 1667(1)-(N) contained in each customer's VM 1666(1)-(N). Container 1667(1)-(N) can be configured to invoke the corresponding auxiliary VNIC 1672(1)-(N) contained in one or more application subnets 1626 of the data plane application layer 1646, which may be contained in the container egress VCN 1668. The auxiliary VNIC 1672(1)-(N) can transmit the invocation to NAT gateway 1638, which can transmit the invocation to the public Internet 1654. In this example, containers 1667(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1616 and from other entities contained in the data plane VCN 1618. Containers 1667(1)-(N) can also be isolated from resources from other clients.

[0372] In other examples, a client can use containers 1667(1)-(N) to invoke cloud service 1656. In this example, the client can run code within containers 1667(1)-(N) requesting a service from cloud service 1656. Container 1667(1)-(N) can then forward the request to a secondary VNIC 1672(1)-(N), which can then forward the request to a NAT gateway, which can forward the request to the public internet 1654. The public internet 1654 can then forward the request via internet gateway 1634 to one or more LB subnets 1622 contained in control plane VCN 1616. In response to determining that the request is valid, one or more LB subnets can forward the request to one or more application subnets 1626, which can then forward the request to cloud service 1656 via service gateway 1636.

[0373] It should be recognized that the IaaS architectures 1300, 1400, 1500, and 1600 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0374] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0375] Figure 17 An example computer system 1700, in which various embodiments can be implemented, is illustrated. System 1700 can be used to implement any of the computer systems described above. As shown, computer system 1700 includes a processing unit 1704 that communicates with a plurality of peripheral subsystems via a bus subsystem 1702. These peripheral subsystems may include a processing acceleration unit 1706, an I / O subsystem 1708, a storage subsystem 1718, and a communication subsystem 1724. Storage subsystem 1718 includes a tangible computer-readable storage medium 1722 and system memory 1710.

[0376] Bus subsystem 1702 provides a mechanism for allowing various components and subsystems of computer system 1700 to communicate with each other as intended. While bus subsystem 1702 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1702 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0377] A processing unit 1704, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1700. One or more processors may be included in the processing unit 1704. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1704 may be implemented as one or more independent processing units 1732 and / or 1734, each including a single-core or multi-core processor. In other embodiments, the processing unit 1704 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0378] In various embodiments, processing unit 1704 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 1704 and / or storage subsystem 1718. With appropriate programming, processor(s) 1704 can provide the various functions described above. Computer system 1700 may additionally include processing acceleration unit 1706, which may include digital signal processor (DSP), dedicated processor, etc.

[0379] I / O subsystem 1708 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, keyboards, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices such as the Microsoft Xbox® 360 game controller via a natural user interface using gestures and voice commands. User interface input devices may also include eye posture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., “blinking” when taking a photo and / or making menu selections) and translates the eye posture into input to an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0380] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0381] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1700 to the user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0382] Computer system 1700 may include a storage subsystem 1718 that provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 1704, provide the aforementioned functionality. Storage subsystem 1718 may also provide a repository for storing data used according to this disclosure.

[0383] As in Figure 17 As illustrated in the example, storage subsystem 1718 may include various components, including system memory 1710, computer-readable storage medium 1722, and computer-readable storage medium reader 1720. System memory 1710 may store program instructions that can be loaded and executed by processing unit 1704. System memory 1710 may also store data used during instruction execution and / or data generated during program instruction execution. Various types of programs may be loaded into system memory 1710, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0384] System memory 1710 may also store operating system 1716. Examples of operating system 1716 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS and Palm® OS. In some implementations of computer system 1700 that execute one or more virtual machines, the virtual machine, along with its guest operating system (GOS), may be loaded into system memory 1710 and executed by one or more processors or cores of processing unit 1704.

[0385] Depending on the type of computer system 1700, system memory 1710 can have different configurations. For example, system memory 1710 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, system memory 1710 may include a basic input / output system (BIOS), which contains basic routines that facilitate, for example, the transfer of information between elements within computer system 1700 during startup.

[0386] Computer-readable storage medium 1722 may represent a remote, local, fixed and / or removable storage device, and a storage medium for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 1704 of computer system 1700) for use by computer system 1700.

[0387] Computer-readable storage medium 1722 may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0388] For example, computer-readable storage medium 1722 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, disk drives that read from or write to removable non-volatile magnetic disks, and optical disc drives that read from or write to removable non-volatile optical discs (such as CD ROMs, DVDs, and Blu-ray® discs or other optical media). Computer-readable storage medium 1722 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital audio tapes, and so on. Computer-readable storage medium 1722 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules and other data for computer system 1700.

[0389] Machine-readable instructions executable by one or more processors or cores of processing unit 1704 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0390] The communication subsystem 1724 provides an interface to other computer systems and networks. The communication subsystem 1724 serves as an interface for receiving data from other systems and sending data from computer system 1700 to other systems. For example, the communication subsystem 1724 enables computer system 1700 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1724 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technology, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 1724 may provide a wired network connection (e.g., Ethernet).

[0391] In some embodiments, the communication subsystem 1724 may also represent one or more users who can use the computer system 1700 to receive input communications in the form of structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc.

[0392] For example, the communication subsystem 1724 can be configured to receive data feeds 1726 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0393] Furthermore, the communication subsystem 1724 can also be configured to receive data in the form of a continuous data stream, which may include event streams 1728 and / or event updates 1730 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quotation machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0394] The communication subsystem 1724 can also be configured to output structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1700.

[0395] The computer system 1700 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing system.

[0396] Due to the constantly evolving nature of computers and networks, the description of the computer system 1700 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0397] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can freely operate within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the series of transactions and steps described. The various features and aspects of the above embodiments can be used individually or in combination.

[0398] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuitry to perform the operations, by programming programmable electronic circuitry (such as a microprocessor), or any combination thereof. Processes can communicate using various technologies, including but not limited to conventional inter-process communication technologies, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0399] Accordingly, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0400] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, are to be interpreted as covering both singular and plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are to be interpreted as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the embodiments and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0401] Exclusion language, such as the phrase “at least one of X, Y, or Z”, unless otherwise explicitly stated, is intended to be understood in the context generally used to represent items, terms, etc., and may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such exclusion language is generally not intended to, and should not, imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0402] This document describes preferred embodiments of the present disclosure, including the best modes known for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to suitably employ such variations and may practice the present disclosure in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Furthermore, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0403] All references cited in this article, including publications, patent applications and patents, are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0404] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more hardware processors, cause to perform operations including: Present the clinical interface within a graphical user interface (GUI); The first group of patient information corresponding to a specific patient is being presented on the clinical interface; In response to the confirmation that the first group of patient information is being presented on the clinical interface: Based on the first set of patient information, a second set of patient information corresponding to the specific patient is selected and displayed on the GUI without receiving user input requesting the second set of patient information; The second group of patients' information is displayed in the clinical interface.

2. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more hardware processors, cause to perform operations including: Present the clinical interface within a graphical user interface (GUI); Monitor interactions with the clinical interface to generate a set of interface interaction data; While monitoring the interaction with the clinical interface, the system also monitors the audio signals corresponding to the clinical discussion to generate a set of clinical discussion data. The combined context is calculated based at least on the set of interface interaction data and the set of clinical discussion data; Select a set of patient information to display on the clinical interface based on the combined context; The patient information is displayed in the clinical interface.

3. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more hardware processors, cause to perform operations including: The current session that presents the clinical interface within the graphical user interface (GUI); Generate a session context dataset based on the current session of the clinical interface; Select candidate clinical information datasets from multiple clinical information datasets; Calculate relevance scores for candidate clinical information datasets based on the session context dataset; Determine if the correlation score meets the threshold; In response to the determination that the relevance score meets the threshold, a candidate clinical information dataset is selected as the first clinical information dataset. The first set of patient information was generated based on the first clinical information dataset; Present the first set of patient information for display in the clinical interface.

Images