Executing execution plan using digital assistant and using large language model

Abstract

Techniques for a digital assistant with generative artificial intelligence (genAI) to execute an execution plan are disclosed herein. The first genAI model may generate a list of executable actions based on utterances provided by a user. An execution plan may be generated to include executable actions. The execution plan may be executed by performing an iterative process on each executable action. The iterative processing may include identifying an action type, invoking one or more states, and performing, by the one or more states, an executable action using the asset to obtain an output. A second prompt may be generated based on an output obtained from performing each of the executable actions. The second genAI model may generate a response to the utterance based on the second cue.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to non-provisional application No. 18 / 825,573, filed September 5, 2024, which claims the benefit and priority of U.S. provisional application No. 63 / 583,028, filed September 15, 2023, pursuant to 35 USC 119(e). The disclosure of each application is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] This disclosure generally relates to digital assistants, and more particularly, but not exclusively, to techniques for using digital assistants and large language models to execute an execution plan for generating utterance responses. Background Technology

[0004] Artificial intelligence (AI) has a wide variety of applications, with a notable evolution in the realm of digital assistants and chatbots. Initially, many users sought immediate responses through instant messaging or chat platforms. Recognizing the potential for engagement, organizations leverage these platforms to interact with entities such as end users in real-time conversations.

[0005] However, maintaining real-time communication channels with entities through human service personnel has proven costly for organizations. To address this challenge, digital assistants or chatbots (also known as bots) have emerged as a solution that simulates conversations with entities, particularly on the internet. These bots enable entities to interact with users through messaging applications they already use or other applications with messaging capabilities.

[0006] Initially, traditional chatbots relied on predefined skill or intent models, requiring entities to communicate within a fixed set of keywords or commands. Unfortunately, this approach limited the bot's ability to interact intelligently and context-awarely in live conversations, hindering its capacity for natural communication. Entities were constrained by the requirement to use specific commands that the bot could understand, often resulting in difficulties in effectively conveying intent.

[0007] The landscape has changed since large language models (LLMs) were integrated into digital assistants or chatbots. LLMs are deep learning algorithms capable of performing a wide range of natural language processing (NLP) tasks. They utilize a neural network architecture called a transformer, which learns from patterns and structures in natural language and engages in more nuanced and context-aware conversations for various domains and purposes. This evolution marks a significant shift from rigid, keyword-based interactions to a more adaptive and intuitive communication experience compared to traditional chatbots, enhancing the overall ability of digital assistants or chatbots to understand and respond to user queries. Summary of the Invention

[0008] In various embodiments, a computer-implemented method can be used to generate responses to utterances using a digital assistant. The method may include generating a list of one or more executable actions by a first generative artificial intelligence model based on a first prompt including a natural language utterance provided by a user. The method may include creating an execution plan including the one or more executable actions. The method may include executing the execution plan. Executing the execution plan may include performing iterative processing on each of the one or more executable actions. The iterative processing may include (i) identifying the action type of the executable action, (ii) invoking one or more states configured to perform the action type, and (iii) using assets by the one or more states to perform the executable action to obtain an output. The method may include generating a second prompt based on the output obtained by performing each of the one or more executable actions. The method may include generating a response to the natural language utterance by a second generative artificial intelligence model based on the second prompt.

[0009] In some embodiments, creating an execution plan may include performing an evaluation of the one or more executable actions. Additionally or alternatively, the evaluation may include evaluating the one or more executable actions based on the execution plans of one or more ongoing conversation paths initiated by the user and any current activities. Additionally or alternatively, creating an execution plan may include (i) incorporating the one or more executable actions into the execution plan of a current activity associated with the ongoing conversation path, the execution plan of the current activity including an ordered list of the one or more executable actions and one or more previous actions, when the evaluation determines that the natural language utterance is not part of an ongoing conversation path; or (ii) creating a new execution plan including an ordered list of the one or more executable actions when the evaluation determines that the natural language utterance is not part of an ongoing conversation path.

[0010] In some embodiments, the iterative process may include (i) determining whether one or more parameters are available for an executable action, (ii) when the one or more parameters are available, invoking the one or more states and performing the executable action based on the one or more parameters; and (iii) when the one or more parameters for the executable action are unavailable, obtaining the unavailable one or more parameters, then invoking the one or more states and performing the executable action based on the one or more parameters.

[0011] In some embodiments, obtaining the one or more parameters may include generating a natural language request to the user to obtain the one or more parameters of the executable action, and receiving a response from the user including the one or more parameters.

[0012] In some embodiments, invoking one or more states configured to perform the action type may include (i) invoking a first state to identify that the executable action has not yet been performed to generate a response, and (ii) invoking a second state to determine whether one or more parameters are available for the executable action. Additionally or alternatively, performing the executable action using an asset to obtain output may include invoking a third state to generate output. Additionally or alternatively, the first, second, and third states may be different from each other.

[0013] In some embodiments, generating the list may include selecting one or more executable actions from a list of candidate agent actions determined using semantic indexing. Additionally or alternatively, creating an execution plan may include (i) identifying one or more executable actions, at least in part, based on metadata associated with the candidate agent actions within the list of candidate agent actions, that provide information or knowledge for generating a response to natural language utterances; and (ii) generating a structured output for the execution plan by creating an ordered list of the one or more executable actions and a set of dependencies between the one or more executable actions.

[0014] In some embodiments, the iterative process may include determining one or more dependencies between the executable action and at least one other executable action among the one or more executable actions, based on a set of dependencies between the one or more executable actions. Additionally or alternatively, the executable actions may be executed sequentially according to the one or more dependencies determined to exist between the executable action and the at least one other executable action.

[0015] In various embodiments, an apparatus is provided that includes components for implementing some or all of the operations and / or methods disclosed herein.

[0016] In various embodiments, a computer program product is provided, the computer program product including computer instructions that, when executed by a processor, implement some or all of the operations and / or methods disclosed herein.

[0017] In various embodiments, a system is provided that includes one or more processors and one or more computer-readable media storing instructions, which, when executed by the one or more processors, cause the system to perform some or all of a variety of operations. The system may generate a list of one or more executable actions by a first generative artificial intelligence model based on a first prompt including a natural language utterance provided by a user. The system may create an execution plan including the one or more executable actions. The system may execute the execution plan, and executing the execution plan may include performing iterative processing on each of the one or more executable actions. The iterative processing may include (i) identifying the action type of the executable action, (ii) invoking one or more states configured to perform the action type, and (iii) using assets by the one or more states to perform the executable action to obtain an output. The system may generate a second prompt based on the output obtained by performing each of the one or more executable actions. The system may generate a response to the natural language utterance by a second generative artificial intelligence model based on the second prompt.

[0018] In various embodiments, one or more non-transitory computer-readable media are provided to store instructions that, when executed by one or more processors, cause the system to perform some or all of a variety of operations. These operations may include generating a list of one or more executable actions by a first generative artificial intelligence model based on a first prompt including a natural language utterance provided by a user. These operations may include creating an execution plan including the one or more executable actions. These operations may include executing the execution plan, and executing the execution plan may include performing iterative processing on each of the one or more executable actions. The iterative processing may include (i) identifying the action type of the executable action, (ii) invoking one or more states configured to perform the action type, and (iii) using assets by the one or more states to perform the executable action to obtain an output. These operations may include generating a second prompt based on the output obtained by performing each of the one or more executable actions. These operations may include generating a response to the natural language utterance by a second generative artificial intelligence model based on the second prompt.

[0019] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided in the diagram below, as described in more detail below. However, the implementations and contexts below are only a few of the many. Attached Figure Description

[0020] Figure 1 This is a simplified block diagram of a distributed environment incorporating a chatbot system according to various embodiments.

[0021] Figure 2 This is an exemplary architecture of an LLM-based digital assistant according to various embodiments.

[0022] Figure 3 This is a simplified block diagram of a computing environment including a digital assistant capable of executing an execution plan in response to a user's speech, according to various embodiments.

[0023] Figure 4 This is a simplified block diagram illustrating the data flow for updating the semantic context and memory store of a digital assistant according to various embodiments, which can execute an execution plan in response to utterances from a user.

[0024] Figure 5 This is a simplified block diagram of examples of data streams for planning responses to user utterances using a digital assistant capable of executing execution plans, according to various embodiments.

[0025] Figure 6 This is a flowchart, according to various embodiments, of a process for executing an execution plan using a digital assistant including generative artificial intelligence.

[0026] Figure 7 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0027] Figure 8 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0028] Figure 9 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0029] Figure 10 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0030] Figure 11 This is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Implementation

[0031] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be clear that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word “exemplary” as used herein means “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other embodiments or designs.

[0032] introduction

[0033] Artificial intelligence (AI) technology has broad applicability. For example, digital assistants are AI-driven interfaces that use natural language to help users complete various tasks. Typically, for each digital assistant, a client can assemble one or more skills focused on specific types of tasks, such as tracking inventory, submitting attendance cards, and creating expense reports. When an end-user interacts with a digital assistant, the assistant evaluates the end-user's input against their intent and routes conversations to and from appropriate skills based on the perceived user intent. However, traditional intent-based skills have some drawbacks, including limited understanding of natural language, inability to handle unknown input, limited ability to maintain off-script natural conversation, and challenges in integrating external knowledge.

[0034] The emergence of Large Language Models (LLMs) like GPT-4 has pushed the field of chatbot design to unprecedented levels of complexity, overcoming these and other shortcomings of traditional intent-based skills. LLMs are neural networks employing a transformer architecture, specifically designed to process and generate sequential data, such as text or words from a conversation. Trained on vast amounts of text data, LLMs gradually hone their ability to generate text that closely mimics human written or spoken language. While LLMs excel at predicting the next word in a sequence, it's important to note that their output is not guaranteed to be perfectly accurate. Their text generation relies on patterns and information learned from the training data, which can be incomplete, erroneous, or outdated, as their knowledge is limited by their training dataset. LLMs do not possess the ability to recall facts from memory; instead, they focus on generating text that appears appropriate in context.

[0035] To address this limitation, LLMs can be enhanced with tools that grant them access to external knowledge sources and train them to understand and respond to user queries in a context-sensitive manner. This enhancement can be achieved through various means, including knowledge graphs, custom knowledge bases, application programming interfaces (APIs), web crawling, and more. Enhanced LLMs are often referred to as "agents." Once configured, agents can be deployed within AI-based systems such as chatbot applications. Users interact with the chatbot, asking questions or making requests, and the agent generates responses based on a combination of its underlying LLM capabilities and access to external knowledge. This combination of powerful language generation and access to real-time information allows chatbots to provide more accurate, relevant, and context-appropriate responses across a wide range of applications and domains.

[0036] For each digital assistant, a user can assemble one or more agents. An agent (which may at least partially include one or more Large Language Models (LLMs)) is an independent bot that provides human-like conversational abilities for various types of tasks, such as tracking inventory, submitting attendance cards, updating accounts, and creating expense reports. Agents are primarily defined using natural language. Users (such as developers) can create functional agents by pointing them to assets such as application programming interfaces (APIs), knowledge-based assets such as documents, URLs, images, etc., data repositories, previous conversations, etc. Assets are imported into the agent, and then, because the agent is LLM-based, the user can again use natural language to customize the agent to provide additional API customizations for dialogue and routing / reasoning. Operations performed by the agent are achieved through the execution of one or more actions. Actions can be explicit actions written (e.g., actions created to generate natural language text or audio responses for natural language prompts written for responses such as queries—"What impact does XYZ have on my 401k contribution limit?"), or implicit actions created when importing assets (e.g., actions created for the "Change Contribution" and "Get Contribution" APIs, which are available through API assets and configured to change a user's 401k contribution).

[0037] When an end user interacts with a digital assistant, the digital assistant evaluates the end user input and routes conversations to and from the appropriate agent. The digital assistant can be made available to the end user through various channels, such as Facebook® Messenger, Skype Mobile® Messenger, or Short Message Service (SMS), and via application interfaces that have been developed to include the digital assistant, for example, using a digital assistant software development kit (SDK). Channels carry back-and-forth chat between the end user and the digital assistant and its various agents. In these back-and-forth exchanges, the selected agent receives processed input in the form of a query and processes the query to generate a response. This is done by the agent's LLM predicting the most relevant and grammatically correct response in the context based on its training data and the inputs it receives (e.g., queries and configuration data). The generated response may undergo post-processing to ensure it adheres to guidelines, policies, and formatting standards. This step helps make the response more coherent and user-friendly. The final response is delivered to the user through the appropriate channel, whether it is a text-based chat interface, a voice-based system, or other media. According to various embodiments, the digital assistant maintains the conversation context, allowing users and the agent to engage in further interaction and dynamic back-and-forth communication, where later interactions can build upon earlier ones.

[0038] Digital assistants, such as those described above, can receive one or more inputs, such as words, from an end user. This input can instruct the end user to expect the digital assistant to perform more than one action—such as two, three, four, or more actions. For example, an end user could input words into the digital assistant instructing them to order pizza and to know about any pizza-related specials. Performing more than one action based on input to the digital assistant can be challenging. For example, determining the set of actions to perform, and the order in which to perform them, can be difficult. Accordingly, different approaches are needed to address these and other challenges.

[0039] Execution plans can be used to address the aforementioned problems. A digital assistant may include, or be otherwise communicatively coupled to, a planning module configured to generate execution plans. An execution plan may include a set of actions to be performed, the order in which those actions are performed, assets (such as APIs, knowledge, etc.) used to perform those actions, etc. Execution plans can be generated by a generative model (such as a large language model) in response to input received by the digital assistant from an end user. The generative model can receive utterances from the input and can generate execution plans based on those utterances. The digital assistant can receive execution plans from the generative model and can perform the actions contained within the execution plans. In some embodiments, using a generative model to generate execution plans can enhance the functionality of the digital assistant or other suitable technological fields by providing a more flexible experience for the end user. For example, it may not be necessary to explicitly program every possible action or combination and sequence of actions into the digital assistant to provide a response to an input query. Additionally or alternatively, using a generative model can facilitate broader access to assets, knowledge, etc., allowing the digital assistant to provide broader and higher-quality responses to input from the end user. This improves the technological field of facilitating digital interaction between users and computing resources, such as chatbots.

[0040] A digital assistant can use an execution plan to perform a set of actions in response to input received from an end user. The end user can input one or more utterances into the digital assistant, which can be configured to generate and transmit responses to those utterances. In some embodiments, responding to one or more utterances may involve the digital assistant performing a set of actions, including one, two, three, four, or more actions. Each action may be associated with a different asset, such as an API, knowledge base, etc. The digital assistant can use generative models, such as large language models, to generate an execution plan for generating and executing that plan.

[0041] A digital assistant or its associated generative model can access a semantic context and a memory repository to receive a set of potential actions that the digital assistant can perform. In some embodiments, the digital assistant can semantically search the semantic context and memory repository to receive the set of potential actions, knowledge or a knowledge base, a set of assets associated with the set of potential actions, etc. The digital assistant can enable the generative model to receive the set of potential actions and input from the end user, and the generative model can be configured to generate an execution plan. In some embodiments, the execution plan may include (i) a set of actions to be performed in response to input from the end user, and / or (ii) the order in which the set of actions in (i) are performed.

[0042] As used herein, when an action is “based on” something, it means that the action is based at least partially on at least a portion of that thing. As used herein, the terms “similarly,” “substantially,” “about,” and “approximately” are defined as substantially, but not necessarily fully specified (and include fully specified), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms “similarly,” “substantially,” “approximately,” or “approximately” may be replaced with “[percentage]” of the specified content, where the percentage includes 0.1%, 1%, 5%, and 10%.

[0043] Digital assistants and knowledge dialogue

[0044] A bot (also known as an intelligent agent, chatbot, chattterbot, or talkbot) is a computer program that can converse with an end user. Bots typically respond to natural language messages (e.g., asking questions or commenting) through messaging applications that use natural language messaging. Businesses can use one or more bot systems to communicate with end users via messaging applications. The messaging application (which may be referred to as a channel) can be one that the end user already has installed and is familiar with, and is the end user's preferred messaging application. Therefore, the end user does not need to download and install a new application to chat with the bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), mobile and web application extensions that leverage chat capabilities to extend native or hybrid / responsive mobile or web applications, or voice-based input (such as devices or applications with interfaces that allow interaction using Siri, Cortana, Google Voice, or other voice inputs).

[0045] In some examples, a robot system may be associated with a Uniform Resource Identifier (URI). A URI can be used as a string to identify the robot system. A URI can be used as a webhook for one or more messaging applications. A URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The robot system may be designed to receive messages from messaging applications (e.g., Hypertext Transfer Protocol (HTTP) POST request messages). HTTP POST request messages can be directed from the messaging application to the URI. In some embodiments, the message may be different from an HTTP POST request message. For example, the robot system may receive messages from a Short Message Service (SMS). While the discussion herein may refer to communication received by the robot system as a message, it should be understood that a message can be an HTTP POST request message, an SMS message, or any other type of communication between the two systems.

[0046] End users can interact with robotic systems through conversational interactions (sometimes called conversational user interface (UI)), much like human-to-human interactions. In some cases, the interaction may involve the end user saying "hello" to the robot, and the robot responding with "hi" and asking how it can help the end user. In other cases, the interaction may be transactional, such as with a banking robot, like transferring funds from one account to another; informational, such as with an HR robot, like checking holiday balances; or with, for example, a retail robot, like discussing returning purchased goods or seeking technical support.

[0047] In some embodiments, a robot system can intelligently handle end-user interactions without interacting with the robot system's administrator or developers. For example, an end-user can send one or more messages to the robot system to achieve a desired goal. Messages may include content such as text, emojis, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., a Representational State Transition (REST) ​​or API call for an enterprise service with the correct parameters) and generate a natural language response. The robot system may also prompt the end-user for additional parameters or request other additional information. In some embodiments, the robot system may also initiate communication with the end-user rather than passively responding to the end-user's utterances. This document describes various techniques for identifying explicit calls to a robot system and determining the input of the called robot system. In some embodiments, explicit call analysis is performed by the master robot based on the call name detected in the utterance. In response to the detection of the call name, the utterance can be refined or preprocessed for input to the robot identified as associated with that call name and / or communication.

[0048] Figure 1 This is a simplified block diagram of an environment 100 incorporating a digital assistant system according to certain embodiments. Environment 100 includes a Digital Assistant Builder Platform (DABP) 105, which enables a user 110 to create and deploy the digital assistant system 115. For the purposes of this disclosure, a digital assistant is an entity that assists a user in performing various tasks through natural language conversation. The DABP and digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. In some instances, environment 100 is part of an Infrastructure as a Service (IaaS) cloud service (described in detail below), and the DABP and digital assistant can be implemented as part of the IaaS by leveraging scalable computing resources and storage capabilities provided by the IaaS provider to handle and manage large amounts of data and complex computations. This setup allows the DABP and digital assistant to provide real-time responsive interaction while ensuring high availability, security, and performance scalability to meet changing levels of demand. Digital assistants can be implemented or realized in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Accordingly, for the purposes of this disclosure, the terms digital assistant and chatbot system are used interchangeably.

[0049] The DABP 105 can be used to create one or more digital assistant (or DA) systems. For example, such as Figure 1 As shown, user 110 of a specific business can use DABP 105 to create and deploy digital assistants 115A for that specific business's users. For example, a bank can use DABP 105 to create one or more digital assistants for its customers to use, for example, to change 401k payments. Multiple businesses can use the same DABP 105 platform to create digital assistants. As another example, a restaurant owner, such as a pizzeria, can use DABP 105 to create and deploy digital assistants 115B that enable the restaurant's customers to order food (e.g., order pizza).

[0050] To create one or more digital assistant systems 115, the DABP 105 is equipped with a set of tools 120 that enable the acquisition of LLMs, the creation of agents, the identification of assets, and the communication via computing platforms (such as those related to...). Figure 7-11 A detailed description of the cloud computing platform) in the service architecture for users (as described in this article) Figure 3(Detailed Description) An embedded digital assistant system. In some instances, tool 120 can be used to access pre-trained and / or fine-tuned LLMs from a data repository or computing system. Pre-trained LLMs serve as foundational elements, possessing extensive language understanding derived from massive datasets. This capability enables models to generate coherent responses across a wide range of topics, facilitating transfer learning. Pre-trained models offer cost-effectiveness and flexibility, allowing for scalable improvements and continuous pre-training with new data, typically establishing benchmarks in natural language processing (NLP) tasks. Conversely, fine-tuned models are trained specifically for a task or industry (e.g., leveraging the contextual learning capabilities of LLMs for plan creation, knowledge or information retrieval on behalf of an agent, generating responses for human-like conversations, etc.), enhancing their performance on specific applications and enabling efficient learning from smaller, specialized datasets. Fine-tuning offers advantages such as task specialization, data efficiency, faster training times, model customization, and resource efficiency. In some embodiments, fine-tuning can be particularly advantageous for niche applications and continuous enhancement.

[0051] In other instances, tool 120 can be used for pre-training and / or fine-tuning LLMs. Tool 120, or any subset thereof, can be a standalone machine learning operational framework, which includes hardware components such as processors (e.g., CPUs, GPUs, TPUs, FPGAs, or any combination thereof), memory, and storage devices. This framework operates software or computer program instructions (e.g., TensorFlow, PyTorch, Keras, etc.) to execute arithmetic, logical, and input / output commands for training, validating, and deploying machine learning models in production environments. In some instances, tool 120 uses cloud platforms such as Oracle Cloud Infrastructure (OCI) for model training, validation, and deployment. Leveraging cloud platforms makes machine learning more accessible, flexible, and cost-effective, facilitating faster model development and deployment by developers.

[0052] Tool 120 also includes a prompt-based agent composition unit for creating agents and their associated actions (e.g., prompts such as "Tell me a joke," implicit "Change payment," and "Get payment" API calls), which can ultimately be invoked by an end user. Agents (e.g., a 401k "Change payment" agent) can be primarily defined as an assembly of agent artifacts using natural language within the prompt-based agent composition unit. User 110 can quickly create functional agents by providing agent artifact information, parameters, and configuration, and by pointing to assets. Assets can be or include resources that can be used by the agent to perform actions, such as APIs for interfacing with applications, files and / or documents for retrieving knowledge, data repositories for interacting with data, etc. Assets are imported, and user 110 can then again provide additional API customizations for dialogue and routing / reasoning using natural language. Much of what the agent performs can involve performing actions. Actions can be explicit actions written in natural language (similar to creating agent artifacts—for example, the "What impact does XYZ have on my 401k contribution limit?" action in the "401k Contribution Agent" diagram below), or implicit actions created when importing assets (automatically imported based on metadata and / or specifications associated with the asset when pointing to a given asset—for example, actions created for the "Change Contribution" and "Get Contribution" APIs in the "401k Contribution Agent" diagram below). Users can easily create explicit actions at design time. For example, when a user learns that a new FAQ needs to be added (because it is not currently in the knowledge document (asset) referenced by the agent (and therefore not implicitly added as an action)), the user can select a "Rich Text" action type (see the list of exemplary action types in Table 1) and create the artifact named "What impact does XYZ have on my 401k contribution limit?".

[0053] Table 1:

[0054]

[0055] There are various ways to associate or add agents and assets to a digital assistant 115. In some instances, agents can be developed by an enterprise and then added to the digital assistant using DABP 105. In other instances, agents can be developed and created using DABP 105 and then added to a digital assistant created using DABP 105. In still other instances, DABP 105 provides an online digital store (called the "Agent Store") that offers a variety of pre-created agents for a wide range of tasks and actions. Agents offered through the Agent Store can also expose various cloud services. To add an agent to a digital assistant created using DABP 105, a DABP 105 user 110 can access assets via tool 120, select specific assets for the agent, initiate several simulated chat conversations with the agent, and instruct the user to add the agent to the digital assistant created using DABP 105.

[0056] Once deployed in a production environment (such as regarding...) Figure 2 In the described architecture, a digital assistant (such as digital assistant 115A built using DABP105) can be used to perform various tasks via natural language-based conversation between digital assistant 115A and its user 125. As described above, this enables... Figure 1 The digital assistant 115A shown is available for use or access by its user 125 through a variety of different channels, such as, but not limited to, certain applications, social media platforms, various messaging services and applications, and other applications or channels. A single digital assistant may have several channels configured for it, allowing it to run on and be accessed by different services simultaneously.

[0057] As part of the conversation, user 125 may provide one or more user inputs 130 to digital assistant 115A and receive a response 135 from digital assistant 115A. The conversation may include one or more of user inputs 130 and responses 135. Through these conversations, user 125 may request one or more tasks to be performed by digital assistant 115A, and in response, digital assistant 115A is configured to perform the user-requested task and respond to user 125 with an appropriate response using one or more LLMs 140.

[0058] User input 130 is typically in natural language form and is referred to as a speech, which may also be called a prompt, query, request, etc. User input 130 can be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides it as input to the digital assistant 115A. In some embodiments, user input 130 can be in audio input or speech form, such as when a user speaks or utters something provided as input to the digital assistant 115A. User input 130 is typically in the language spoken by user 125. For example, user input 130 can be English or some other language. When user input 130 is in speech form, the speech input is converted into text form of user input 130 in that particular language, and then the text speech is processed by the digital assistant 115A. Various speech-to-text processing techniques can be used to convert speech or audio input into text speech, which is then processed by the digital assistant 115A. In some embodiments, the speech-to-text conversion can be performed by the digital assistant 115A itself. For the purposes of this disclosure, it is assumed that user input 130 is either text speech directly provided by user 125 of digital assistant 115A, or the result of inputting voice speech converted into text form. However, this is not intended to limit or constrain in any way.

[0059] User input 130 can be used by digital assistant 115A to determine a list of candidate agents 145A-N. The list of candidate agents (e.g., 145A-N) includes agents configured to perform one or more actions that may potentially facilitate a response 135 to user input 130. This list can be determined by running searches (such as semantic searches) on a context and a memory repository containing one or more indexes of metadata for all agents 145 available to digital assistant 115A. The metadata of candidate agents 145A-N in the list of candidate agents is then combined with the user input to construct an input prompt for one or more LLMs 140.

[0060] The digital assistant 115A is configured to apply NLP techniques to text and / or speech using one or more LLMs 140 to understand input prompts and to apply Natural Language Understanding (NLU) (including syntactic and semantic analysis of text and / or speech) to determine the meaning of user input 130. Determining the meaning of utterances may involve identifying the user's goal, one or more of the user's intentions, the context surrounding various words or phrases or sentences, one or more entities corresponding to the utterance, etc. NLU processing may include parsing the received user input 130 to understand the structure and meaning of the utterance, refining and reshaping the utterance to develop a more understandable form (e.g., logical form) or structure. The NLU processing performed may include various NLP-related processes such as sentence parsing (e.g., tokenizing, lemmatizing, identifying part-of-speech tags of sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing individual clauses, parsing pronouns, performing chunking, etc.). In some instances, NLU processing, or any part thereof, is performed by the LLM 140 itself. In other instances, the LLM 140 uses additional resources to perform parts of the NLU processing. For example, the syntax and structure of the input utterance sentence can be identified by processing the sentence using parsers, part-of-speech taggers, named entity recognition models, pre-trained language models (such as BERT), etc.

[0061] After understanding the meaning of the utterance, one or more LLMs 140 generate an execution plan that identifies one or more agents (e.g., agent 145A) from a list of candidate agents to perform and conduct one or more actions or operations in response to the understood user meaning or goal. The one or more actions or operations are then executed by the digital assistant 115A on one or more assets (e.g., asset 150A—knowledge, API, SQL operations, etc.) and / or a context and memory repository. The execution of the one or more actions or operations generates output data from the one or more assets and / or generates relevant context and memory information from a context and memory repository that includes the context of the current conversation with the digital assistant 115A. The output data and relevant context and memory information are then combined with user input 130 to construct an output prompt for one or more LLMs 140. The LLM 140 synthesizes a response 135 to user input 130 based on the output data, relevant context and memory information, and user input 130. The response 135 is then sent to user 125 either as a separate response or as part of a conversation with user 125.

[0062] For example, user input 130 can request to order pizza by providing a utterance such as “I want to order pizza.” Upon receiving such a utterance, digital assistant 115A is configured to understand the meaning or goal of the utterance and take an appropriate action. The appropriate action may involve, for example, providing the user with a response 135 that asks the user to input the type of pizza the user wishes to order, the size of the pizza, any toppings, etc. The user's request for the question can be generated by performing an action on knowledge assets (e.g., a pizza restaurant's menu) via an agent (e.g., agent 145A) to retrieve information related to ordering pizza (e.g., to order pizza, the user must provide the type, size, toppings, etc.). The response 135 provided by digital assistant 115A can also be in natural language form and is typically the same language as user input 130. As part of generating these responses 135, digital assistant 115A may perform natural language generation (NLG) using one or more LLMs 140. For a user ordering pizza, the digital assistant 115A can guide the user to provide all the necessary information for the pizza order through a conversation between the user and the digital assistant 115A, and then the pizza is ordered at the end of the conversation. The order can be executed by an agent (e.g., agent 145A) performing an action on an API asset (e.g., an API for ordering pizza) to upload or provide the pizza order to the restaurant's ordering system. The digital assistant 115A can end the conversation by generating a final response 135, providing the user 125 with information indicating that the pizza has been ordered.

[0063] While the various examples provided in this disclosure describe and / or illustrate utterances in the English language, these are merely examples. In some embodiments, the digital assistant 115 is also capable of processing utterances in languages ​​other than English. The digital assistant 115 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes NLU processing flexible and scalable for each language, including allowing different processing sequences. Language packs may be provided for each language, wherein the language packs may register a list of subsystems that can be served from the NLU core server.

[0064] Although Figure 1 The embodiments illustrated herein include a digital assistant 115A comprising one or more LLMs 140 and one or more agents 145A-N, but this is not intended to be limiting. The digital assistant may include various other components that provide the functionality of a digital assistant (e.g., as per [reference to...]). Figure 2Other systems and subsystems (more detailed description of other systems and subsystems). The Digital Assistant 115A and its systems and subsystems may be implemented in software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), in hardware only, or in an implementation using a combination of software and hardware.

[0065] Figure 2 This is an example of the architecture of a computing environment 200 using generative artificial intelligence, according to various embodiments. For example... Figure 2 As shown, the infrastructure, along with various services and features, can be used to enable users to interact with digital assistants (e.g., about) based at least in part on a series of prompts (such as conversation). Figure 1 The interaction is described in the digital assistant 115A. The following is a detailed walkthrough of the conversation flow, and the roles and responsibilities of the components, services, and models of the computing environment 200 within the conversation flow. In this walkthrough, assume that user "David" is interested in making changes to his 401k contribution, and in discourse 202, David provides the following input to the digital assistant: Hi, how are you? I want to change my 401k contribution.

[0066] A utterance 202 can be passed to a digital assistant (e.g., via a text dialog box or microphone) and provided as input to an input pipeline 208. The input pipeline 208 is used by the digital assistant to create an execution plan 210 that identifies one or more agents for resolving the request in the utterance 202, and one or more actions for those agents to perform in response to the request. The execution plan 210 can be generated via a two-step process through the input pipeline 208. First, a search 212 can be performed to identify a list of candidate agents. The search 212 involves running queries based on an index 213 of the utterance 202 against a context and a memory repository 214. In some instances, the search 212 is a semantic search performed using words from the utterance 202. The semantic search uses NLP and optional machine learning techniques to understand the meaning of the utterance 202 and retrieve relevant information from the context and memory repository 214. Compared to traditional keyword-based searches that rely on exact matches between words in the query and data in the context and memory repository 214, semantic search considers relationships between words, the context of the query, synonyms, and other linguistic nuances. This allows digital assistants to provide more accurate and context-sensitive results, making them more effective at understanding user intent in discourse 202.

[0067] The context and memory repository 214 is implemented using a data framework that connects external data to the LLM 216, allowing users to easily plug in custom data sources. The data framework provides a rich and efficient retrieval mechanism on top of data from various sources, such as files, documents, data repositories, APIs, etc. Data can be external (e.g., enterprise assets) and / or internal (e.g., user preferences, memories, digital assistant and agent metadata, etc.). In some instances, the data includes metadata extracted from artifacts 217 associated with the digital assistant and its agents 218 (e.g., 218a and 218b). The digital assistant's artifacts 217 include information about the general capabilities of the digital assistant, as well as specific information about the capabilities (e.g., actions) of each agent 218 (e.g., agent artifacts) available to the digital assistant. Additionally or alternatively, artifacts 217 may contain parameters or information associated with artifacts 217 and that can be used to define agents 218, wherein the parameters or information associated with artifacts 217 may include a name, description, one or more actions, one or more assets, one or more customizations, etc. In some instances, the data also includes metadata extracted from assets 219 associated with the digital assistant and its agents 218 (e.g., 218a and 218b). Assets 219 can be resources that agents 218 can use to perform actions (e.g., actions 225a, 225b, and 225c), such as APIs 220, files and / or documents 222, data repositories 223, etc. The data is indexed in a context and memory repository 214 as an index 213, a data structure that provides a fast and efficient way to find and retrieve specific data records within the data. Therefore, the context and memory repository 214 provides a searchable and comprehensive record of all the capabilities available to the digital assistant to respond to requests from agents and associated assets.

[0068] The results of search 212 include a list of candidate agents that can not only be used by the digital assistant to respond to requests, but also potentially facilitate the generation of a response to utterance 202. The list of candidate agents contains metadata associated with each candidate agent from the context and memory repository 214 (e.g., metadata extracted from artifacts 217 and assets 219). This list may be limited to a predetermined number of candidate agents that satisfy the query (e.g., the top 10), or it may include all agents that satisfy the query. The list of candidate agents with associated metadata is appended to utterance 202 to construct input prompts 227 for LLM 216. In some instances, context 229 about utterance 202 is additionally appended to the list of candidate agents and utterance 202. Context 229 can be retrieved from the context and memory repository 214 and includes user session information, dialogue state, conversation or context history, user information, or any combination thereof. Search 212 is important for the digital assistant because it filters out agents that are unlikely to facilitate the generation of a response to utterance 202. This filtering ensures that the number of lexical units (e.g., word lexical units) generated from input prompt 227 remains within the maximum lexical unit limit or contextual limit set for LLM 216. The lexical unit limit represents the maximum amount of text that can be input into the LLM. This limit is technical in nature and arises from computational constraints such as memory and processing resources, and thus ensures that the LLM can accept input prompts as input.

[0069] The second step of the two-step approach is for the LLM 216 to generate an execution plan 210 based on the input cue 227. The LLM 216 employs a deep generative model architecture (e.g., with an invertible or autoregressive architecture) for generating the execution plan 210. In some instances, the LLM 216 has over 100 billion parameters and uses autoregressive language modeling within a transformer architecture to generate the execution plan 210, allowing the LLM 216 to capture complex patterns and dependencies in the input cue 227. The LLM 216's ability to generate the execution plan 210 is a result of its training on diverse and extensive textual data, enabling the LLM to understand human language in various contexts. During training, the LLM 216 learns to predict the next word in the sequence by considering the context of the preceding words. This process involves adjusting the model's parameters (weights and biases) based on the error between its prediction and the actual next word in the training data. When the LLM 216 receives input such as the input cue 227, it breaks the text into smaller units, such as words or sub-words. Each word is then represented as a vector in a multidimensional space. The LLM 216 processes the input sequence word by word, maintaining an internal representation of the context. The LLM 216's attention mechanism allows it to measure the importance of different words in the context of generating the next word. For each word in the vocabulary, the LLM 216 computes a probability distribution based on its learned parameters. This probability distribution represents the likelihood of each word being the next word, considering the context. To generate execution plan 210, the LLM 216 samples words from the computed probability distribution. The sampled words become the next word in the generated sequence. This process is repeated iteratively, where each newly generated word affects the context for generating subsequent words. The LLM 216 can continue generating words until a predefined length or stopping condition is reached.

[0070] In some instances, such as Figure 2As shown, LLM 216 may not be able to generate a complete execution plan 210 because it lacks information such as whether more information is needed to determine the appropriate agent for the response, to perform one or more actions, etc. In this particular instance, LLM 216 has determined that in order to change a 401k contribution at the user's request, it is necessary to know whether the user is willing to change the contribution by a percentage or a specific monetary amount. To obtain this information, LLM 216 (or another LLM such as LLM 236) generates an end-user response 235 to input prompt 227 (I'm fine. Do you want to change your contribution by a percentage or an amount? [Percentage] [Amount]), which provides the missing information, allowing LLM 216 to generate a complete execution plan 210. In some instances, the response may be presented within a dialog box in a GUI with one or more GUI elements, making it easier for the user to respond. In other instances, the response may be presented within a dialog box in a GUI, allowing the user to respond using that dialog box (or an alternative, such as a microphone). In this particular instance, the user responds with an additional query 238 (What is my current 401k contribution? And can you tell me my contribution limit?) to gather additional information so that the user can reply with response 235. The subsequent response—additional query 238—is fed into input pipeline 208 and undergoes the same processing described above regarding utterance 202, but this time with the context of the previous utterance / response (e.g., utterance 202 and response 235) from the user's conversation with the digital assistant. This time, as... Figure 2 As shown, LLM 216 is able to generate a complete execution plan 210 because it has all the information it needs.

[0071] Execution plan 210 includes an ordered list of agents and / or actions that can be used and / or executed to adequately respond to requests (such as additional query 238). For example, and as... Figure 2As shown, execution plan 210 may be an ordered list including a first agent 242a capable of performing a first action 244a via an associated asset and a second agent 242b capable of performing a second action 244b via the associated asset. The agents and the actions extended therefrom may be ordered such that the first agent 242a performs the first action 244a before the second agent 242b performs the second action 244b. In some instances, execution plan 210 may be ordered based on dependencies indicated by the agents and / or actions included in it. For example, if the execution of the second agent 242b depends on or otherwise requires the output generated by the first agent 242a performing the first action 244a, then execution plan 210 may command the first agent 242a and the second agent 242b to comply with that dependency. It should be understood that other examples of dependencies are also possible.

[0072] The execution plan 210 is then transmitted to the execution engine 250 for implementation. The execution engine 250 includes several engines, including a natural language to programming language translator 252, a knowledge engine 254, an API engine 256, a prompting engine 258, etc., for executing the agent's actions and implementing the execution plan 210. For example, the natural language to programming language translator 252 (such as a conversation to Oracle Meaning Representation Language (C2OMRL) model) can be used by the agent to translate natural language into intermediate logic (e.g., OMRL), convert that intermediate logic into a system programming language (e.g., SQL), and execute that system programming language (e.g., execute an SQL query) on asset 219 (such as a data repository 223) to perform actions and / or obtain data or information. The knowledge engine 254 can be used by the agent to obtain data or information from context and memory repositories 214 or asset 219 (such as files / documents 222). The API engine 256 can be used by the agent to call API 220 and interface with applications (such as retirement account management applications) to perform actions and / or obtain data or information. The prompting engine 258 can be used by an agent to construct prompts for input into an LLM (such as an LLM in a context and memory repository 214 or an asset 219) to perform actions and / or obtain data or information.

[0073] Execution engine 250 implements execution plan 210 by using appropriate (one or more) engines to run each agent and execute each action sequentially based on an ordered list of agents and / or actions. To facilitate this implementation, execution engine 250 is communicatively connected (e.g., via public and / or private networks) to agents (e.g., 242a, 242b, etc.), context and memory repository 214, and assets 219. For example, as... Figure 2As shown, when execution engine 250 implements execution plan 210, it first uses API engine 256 to execute agent 242a and action 244a to call API 220 and interface with the retirement account management application to retrieve the user's current 401k contributions. Subsequently, execution engine 250 can use knowledge engine 254 to execute agent 242b and action 244b to retrieve knowledge about 401k contribution limits. In some instances, knowledge engine 254 retrieves knowledge from asset 219 (e.g., file / document 222). In other instances (as in this particular instance), knowledge is retrieved by knowledge engine 254 from context and memory repository 214. Various techniques, including internal task mapping and / or machine learning models such as attached LLM models, can be used to implement knowledge retrieval and action execution using context and memory repository 214. For example, a query for “What is the 401k contribution limit?” and the associated agent can be mapped to a “semantic search” knowledge task type to search for a response to a given query in index 213 within the context and memory repository 214. As another example, a request such as “Can you summarize the key points related to 401k contributions?” can be or include a “summarize” knowledge task type, which can be mapped to different indexes within the context and memory repository 214 that have LLMs trained to create natural language responses (e.g., summaries of key points related to 401k contributions) for a given query. Over time, a library of general end-user task or action types (e.g., semantic search, summarization, comparison / contrast, heterogeneous data synthesis, etc.) can be built to ensure that the indexes and models within the context and memory repository 214 are optimized for various task or action types.

[0074] The result of executing execution plan 210 is output data 269 (e.g., the outcome of an action, data, information, etc.), which is transmitted to output pipeline 270 to generate end-user response 272. For example, output data 269 from assets 219 (knowledge, APIs, conversation history, etc.) and relevant information from context and memory repository 214 can be transmitted to output pipeline 270. Output data 269 is appended to utterance 202 to construct output prompt 274 for input to LLM 236. In some instances, context 229 about utterance 202 is additionally appended to output data 269 and utterance 202. Context 229 can be retrieved from context and memory repository 214 and includes user session information, conversation state, conversation or context history, user information, or any combination thereof. LLM 236 generates response 272 based on output prompt 274. In some instances, LLM 236 is the same or similar model to LLM 216. In other instances, LLM 236 differs from LLM 216 (e.g., trained on different datasets, with different architectures, trained for one or more different tasks, etc.). In any instance, LLM 236 has a deep generative model architecture for generating responses 272 using similar training and generative processing as described with respect to LLM 216 (e.g., with an invertible or autoregressive architecture). In some instances, LLM 236 has over 100 billion parameters and uses autoregressive language modeling to generate responses 272 within a transformer architecture, allowing LLM 236 to capture complex patterns and dependencies in the output cues 274.

[0075] In some instances, the end-user response 272 can be formatted using the Conversational Message Model (CMM) and output as a rich, multimodal response. The CMM defines the various message types that a digital assistant can send to a user (outbound) and that a user can send to a digital assistant (inbound). In some instances, the CMM recognizes the following message types:

[0076] -Text: Basic text message

[0077] - Cards: Card representations that include a title and optional descriptions, images, and links.

[0078] - Attachment: Message with media URL (file, image, video, or audio)

[0079] -Location: A message with geographic location coordinates

[0080] - Postback: A message with a postback payload

[0081] In CMM, messages are channel-independent and can be created using CMM syntax. Channel-specific connectors transform CMM messages into the format required for a specific channel, allowing users to run digital assistants on multiple channels without having to create separate message formats for each channel.

[0082] Finally, output pipeline 270 transmits response 272 to the end user (e.g., via user device or interface). In some instances, response 272 is presented within a dialog box in the GUI, allowing the user to view and respond using the dialog box (or alternative means, such as a microphone). In other instances, response 272 is presented within a GUI dialog box with one or more GUI elements, allowing for easier user responses. In this particular instance, the initial response 272 to the additional query 238 (What is my current 401k contribution? Also, could you tell me my contribution limit?) is presented within a dialog box in the GUI. Furthermore, to follow up on information still needed to obtain the initial statement 202, LLM 236 generates another response 272 prompting the user for missing information (Would you like to change your contribution by percentage or amount? [Percentage] [Amount]).

[0083] Although Figure 2 The embodiments of computing environment 200 illustrate the interaction of a digital assistant in a specific conversation flow, but this is not intended to be restrictive, but merely to facilitate a better understanding of the role and responsibilities of the components, services, models, etc. of computing environment 200 within the conversation flow.

[0084] Block diagram of a computing environment including digital assistants

[0085] Figure 3 This is a simplified block diagram of a computing environment including a digital assistant 300 capable of executing an execution plan in response to speech from a user, according to various embodiments. In some embodiments, speech may be provided to the digital assistant 300 from a user via input 302. Input 302 may be or include natural language speech, which may include text input, voice input, image input, or any other suitable input for the digital assistant 300. For example, input 302 may include text input provided by the user via a keyboard or touchscreen of a computing device used by the user. In other examples, input 302 may include spoken words provided by the user via a microphone of a computing device. In other examples, input 302 may include image data, video data, or other media provided by the user via a computing device. Additionally or alternatively, input 302 may include instructions for actions to be performed by the digital assistant 300 on behalf of the user. For example, input 302 may include instructions for the user to order pizza, instructions for the user to renew retirement account contributions, or other suitable instructions.

[0086] Input 302 can be provided to planner 304 of digital assistant 300. Planner 304 can generate an execution plan based on input 302 and the context provided to planner 304. Planner 304 can receive input 302 and can invoke semantic context and memory repository 306 to retrieve the context. In some embodiments, semantic context and memory repository 306 includes one or more assets 308, which may be similar to or identical to asset 219. Planner 304 can provide at least a portion of input 302 to semantic context and memory repository 306, which can perform a semantic search on asset 308 and / or other knowledge contained in semantic context and memory repository 306. The semantic search can generate a list of candidate actions that can be used to resolve input 302 or any subset thereof from all actions that can be performed via one or more assets 308. In some embodiments, candidate actions can be generated based solely on contextual information. For example, input 302 can be compared with metadata of actions to generate candidate actions.

[0087] Planner 304 can use candidate actions to form input cues for a generative AI model. This generative AI model may be or is included in generative AI model 310, which may include one or more large language models (LLMs). Planner 304 may be communicatively coupled to generative AI model 310 via a common language model interface layer (CLMI layer 312). CLMI layer 312 may be an adapter layer that allows planner 304 to invoke various different generative AI models that may be included in generative AI model 310. For example, planner 304 may generate input cues and provide them to CLMI layer 312, which may convert the input cues into model-specific input cues for input into a specific generative AI model. Planner 304 may receive output from this specific generative AI model that can be used to generate an execution plan. This output may be or includes an execution plan. In other embodiments, the output may be used as input by planner 304 to allow planner 304 to generate an execution plan. Based on the utterance contained in input 302, the output may include a list containing one or more executable actions. In some embodiments, the execution plan may include an ordered list of actions to be performed to resolve input 302.

[0088] Planner 304 can transmit an execution plan to execution engine 314 for execution. Execution engine 314 can perform iterative processing on each executable action contained in the execution plan. For example, for each executable action, execution engine 314 can identify an action type, invoke one or more states for executing that action type, and use assets to execute the executable action to obtain output. Execution engine 314 can be communicatively coupled to action executor 316, which can be configured to perform at least a portion of the iterative processing. For example, action executor 316 can identify one or more action types for each executable action contained in the execution plan. In a particular example, action executor 316 can identify a first action type 318a for a first executable action of the execution plan. First action type 318a can be or include semantic actions, such as summarizing text or other suitable semantic actions. Additionally or alternatively, action executor 316 can identify a second action type 318b for a second executable action of the execution plan. Second action type 318b can involve calling an API, such as an API for adjusting an account or other suitable API. Additionally or alternatively, the action executor 316 may identify a third action type 318c for a third executable action of the execution plan. The third action type 318c may be or include knowledge actions, such as providing an answer to a technical question or other suitable knowledge actions. In some embodiments, the third action type 318c may involve invoking at least one generative artificial intelligence model in the generative artificial intelligence model 310 to retrieve specific knowledge or a specific answer. In other embodiments, the third action type 318c may involve invoking semantic context and memory repository 306 or other knowledge documents.

[0089] Action executor 316 can continue iterative processing based on the action type indicated by the executable actions contained in the execution plan. Once action executor 316 identifies the action type, it can identify and / or invoke one or more states for each executable action based on the action type. The state of an action can involve an indication of whether the action can be executed or has already been executed. For example, the state of a particular executable action can include "preparing," "ready," "executing," "successful," "failed," or any other suitable state. Action executor 316 can determine whether an executable action is ready to be executed based on the state of the invoked executable action, and if the executable action is not yet ready to be executed, action executor 316 can identify any missing information or assets required to continue executing the executable action. In response to determining that an executable action is ready to be executed, and in response to determining that no dependencies exist for the executable action (or that existing dependencies are satisfied), action executor 316 can execute the executable action to generate output.

[0090] Action executor 316 can execute each executable action or any subset thereof contained in the execution plan to generate a set of outputs. This set of outputs may include knowledge outputs, semantic outputs, API outputs, and other suitable outputs. Action executor 316 can provide this set of outputs to output engine 320. Output engine 320 can be configured to generate a second input cue based on this set of outputs. The second input cue can be provided to at least one generative AI model in generative AI model 310 to generate a response 322 to input 302. Output engine 320 can invoke at least one generative AI model to generate the response 322, which can be provided to the user in response to input 302. In some embodiments, the at least one generative AI model used to generate the response 322 may be similar to or the same as the at least one generative AI model used to generate the outputs to generate the execution plan, or in other cases, the same model.

[0091] Figure 4 This is a simplified block diagram illustrating the data flow for updating the semantic context and memory repository 306 of a digital assistant 300, according to various embodiments, which can execute an execution plan in response to utterances from a user. Figure 4 As shown, entity 402 can provide different types of input for updating the semantic context and memory repository 306. The first data flow 400a illustrates knowledge updates for the semantic context and memory repository 306, while the second data flow 400b illustrates API updates for the semantic context and memory repository 306.

[0092] As shown in the first data flow 400a, entity 402 can provide knowledge input 404 for updating the semantic context and memory repository 306. Entity 402 can provide knowledge input 404 via a computing device configured to provide a UI / API 406. The UI / API 406 can be or include a user interface that can be used to manage APIs about the digital assistant 300 or otherwise manage updates to the semantic context and memory repository 306. Knowledge input 404 can include updates to rules, additional information that can be provided to the user, and any other suitable knowledge input. UI / API 406 can receive knowledge input 404 and can provide knowledge input 404 or a transformed version thereof to ingestion pipeline 408. Ingestion pipeline 408 can be communicatively coupled to one or more LLMs 410, which can be similar to or the same as one or more generative artificial intelligence models included in generative artificial intelligence model 310. Ingestion pipeline 408 can generate input prompts based on knowledge input 404 that can be provided to one or more LLMs 410 for generating outputs. In some embodiments, one or more LLMs 410 can be configured to generate output based on input prompts, wherein the output may be or include content based on knowledge input 404 that can be stored at the semantic context and memory repository 306. The content may contain the essential portion of knowledge input 404 in a concise form and a compatible format for storage at the semantic context and memory repository 306. Additionally or alternatively, one or more LLMs 410 may generate a summary of knowledge input 404 and provide the summary to the UI / API 406.

[0093] Content and summary-based indexes can be stored at a semantic context and memory repository 306. The semantic context and memory repository 306 may include a document repository 412, a metadata index 414, and any other suitable data store and / or index. Content generated by one or more LLMs 410 can be transferred to the document repository 412 for storage via the ingestion pipeline 408, and the UI / API 406 can transfer the index to the metadata index 414 for storage. Content may be accessible to the digital assistant 300, such as via a search index, in response to future inputs related to knowledge input 404. Additionally or alternatively, the UI / API 406 may transfer summaries to an ATP 416. The ATP 416 may be or include a data store that can store descriptions of assets and knowledge stored at the semantic context and memory repository 306.

[0094] As shown in the second data flow 400b, entity 402 can provide API input 418 for updating the semantic context and memory repository 306. Entity 402 can provide API input 418 via a computing device configured to provide UI / API 406. UI / API 406 can be or includes a user interface that can be used to manage APIs related to the digital assistant 300 or otherwise manage updates to the semantic context and memory repository 306. API input 418 can include additional assets related to the API, or can otherwise include updates to APIs that can be invoked by the digital assistant 300. For example, API input 418 can include instructions to allow the digital assistant 300 to make new API calls involving new assets. In a particular example, API input 418 can indicate a new API for updating a new type of account through the digital assistant 300. UI / API 406 can store the semantic object model or artifact associated with API input 418 at ATP 416. Additionally or alternatively, UI / API 406 may generate or identify metadata based on API input 418, and UI / API 406 may transfer an index involving that metadata to the metadata index 414 of the semantic context and memory repository 306.

[0095] Figure 5 This is a simplified block diagram illustrating examples of data streams for planning responses to user utterances using a digital assistant 300 capable of executing execution plans, according to various embodiments. Figure 5 As shown, input 502 can be received, for example, from a user of the digital assistant 300. In some embodiments, input 502 can be or include natural language, such as natural language text, natural language audio, or other suitable forms of natural language. Input 502 can be received by an action planner 504, such as via a generative AI dialogue manager 506. The generative AI dialogue manager 506 can be or includes an LLM-based dialogue manager that can be an entry point for input from the user and can detect existing actions, rewrite queries, and execute actions. For example, if the generative AI dialogue manager 506 determines that no action is currently being performed or scheduled to be performed, then the generative AI dialogue manager 506 can provide input 502 or any subset or variation thereof to a candidate action generator 508.

[0096] Candidate action generator 508 may perform or cause to perform a semantic search based on input 502 or any subset or variant thereof. For example, candidate action generator 508 may generate a query and transmit it to semantic context and memory repository 306, causing semantic context and memory repository 306 to parse one or more indexes to identify candidate actions 509 based on input 502, etc. The query may involve parsing and / or searching through action and metadata index 510 to identify candidate actions 509. In some embodiments, semantic search may involve searching within asset 512 to identify candidate actions 509. For example, the query may include a task indicated by input 502, and may cause semantic context and memory repository 306 to compare the indicated task with metadata about asset 512 to identify candidate actions 509 using only the context (such as metadata about asset 512). In a particular example, the query may include a task such as updating an account balance, and semantic search may involve searching within asset 512 for a specific asset (such as an API asset) that has metadata indicating that the specific asset can update the account balance. In such an example, the results of a semantic search may include candidate actions 509, which include specific actions that can be performed by that particular asset.

[0097] Candidate action 509 may also be influenced by data stored in short-term memory 514 and / or long-term memory 516. For example, historical access data may be retrieved by candidate action generator 508 to determine candidate action 509. Historical access data may include historical data indicating actions previously selected by other users in response to other inputs provided by other users. For example, if a particular action has been selected in most historical instances in response to similar inputs, then candidate action generator 508 may include that particular action in candidate action 509 regardless of whether the metadata associated with the particular action (or the asset capable of performing the particular action) is similar to the query provided by input 502 or candidate action generator 508.

[0098] Candidate actions 509 (which include actions selected by candidate action generator 508 based on historical access data and the similarity between the actions and queries provided to initiate semantic search) can be provided to generative AI planner 518. Generative AI planner 518 can receive candidate actions 509 and can generate execution plan 520 based on the actions included in candidate actions 509. For example, generative AI planner 518 can determine whether each action or any subset thereof of candidate actions 509 is available and can generate an ordered list of available actions as execution plan 520. In some embodiments, generative AI planner 518 can identify any dependencies between the actions included in candidate actions 509 and can include such dependencies in execution plan 520. In some embodiments, for each executable action included in candidate actions 509, generative AI planner 518 can create an artifact representing that executable action, and the artifact can include indications of any dependencies, indications of whether the executable action is available or ready to be executed, indications of what additional information (if any) is needed to transition the state of the executable action to ready to be executed, and / or any other suitable indications.

[0099] Execution plan 520 can be provided to an execution engine, such as execution engine 314, which can execute the actions included in execution plan 520. In some embodiments, the execution engine can execute the actions included in execution plan 520 that are indicated to be ready for execution in sequence. That is, the execution engine can execute actions included in execution plan 520 that have been invoked to the ready-to-execute state, have no dependencies (or all dependencies have been satisfied), etc. Action tracker 522 can track the progress of execution plan 520. For example, action tracker 522 can determine whether an action has been executed, whether the executed action was successful or failed, etc. The status of the actions included in execution plan 520 can be saved and continuously updated or persisted in short-term memory 514 for future or iterative use by generative AI planner 518.

[0100] Flowchart of the execution plan

[0101] Figure 6 This is a flowchart of a process 600 for executing an execution plan using a digital assistant including generative artificial intelligence, according to various embodiments. Figure 6 The processing described herein can be implemented on software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores), hardware, or a combination thereof of the corresponding system. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 6The processes presented and described below are intended to be illustrative and non-limiting. Although Figure 6 The illustrations depict various processing steps that occur in a specific order or sequence, but this is not intended to be limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed at least partially in parallel. In some embodiments, Figure 6 The processing described in the text can be determined by... Figure 1-5 One or more of the components, computing devices, services, etc. illustrated and described are used to perform actions such as digital assistants, first and / or second generative artificial intelligence models (LLMs), etc.

[0102] At 602, a list of one or more executable actions is generated by a first generative AI model. The first generative AI model may generate the list of one or more executable actions based on a first prompt, which includes natural language utterances provided by the user of the digital assistant. In some examples, the first prompt may include natural language utterances enhanced with separate prompts, such that the first generative AI model outputs the list of one or more executable actions. The list of one or more executable actions may include one or more executable actions, and each executable action may be associated with an asset that the digital assistant can access or invoke. Executable actions may include actions that can be performed (such as execution engine 314) to perform a task indicated by the natural language utterances. In a particular example, the task may include providing information requested by the user, updating an account based on a user request to update an account, etc. In some embodiments, planner 304 may generate the first prompt and may transmit the first prompt to the first generative AI model, such that the first generative AI model outputs the list of one or more executable actions. In some embodiments, generating a list of one or more executable actions may include selecting one or more executable actions from a list of candidate actions, which are determined via a semantic search of a semantic index, which may be contained in a semantic context and a memory repository 306.

[0103] At 604, an execution plan is created, and this execution plan includes the one or more executable actions. The execution plan (which may be similar to or identical to execution plan 520) may be or include an ordered list of one or more executable actions. In some embodiments, creating an execution plan may involve evaluating one or more executable actions. The evaluation may include evaluating one or more executable actions based on one or more ongoing conversation paths initiated by the user (if any). The evaluation may also include evaluating one or more executable actions based on the execution plan of any current activity. Evaluating one or more executable actions may involve determining whether a similar action to the one or more executable actions is scheduled to be performed in an ongoing conversation path or in the execution plan of the current activity, or whether a similar action has been previously performed.

[0104] In some embodiments, in response to an evaluation determining that a natural language utterance is part of an ongoing conversation path, creating an execution plan may additionally include incorporating the one or more executable actions into the execution plan of a current activity associated with the ongoing conversation path. After incorporating the one or more executable actions, the execution plan of the current activity may be or include an ordered list of the one or more executable actions and one or more previous actions. In some embodiments, in response to an evaluation determining that a natural language utterance is not part of an ongoing conversation path, creating an execution plan may additionally include creating a new execution plan, which may be or include an ordered list of the one or more executable actions.

[0105] In some embodiments, creating an execution plan may additionally include identifying one or more executable actions, at least in part, based on metadata associated with candidate agent actions within a list of candidate agent actions, that provide information or knowledge for generating a response to natural language utterances. Additionally or alternatively, creating an execution plan may additionally include generating a structured output for the execution plan by creating an ordered list of the one or more executable actions and a set of dependencies between the one or more executable actions.

[0106] At 606, an iterative process is used to execute an execution plan for each of one or more executable actions. In some embodiments, the iterative process may include identifying the action type of the executable action, invoking one or more states configured to execute that action type, and having the one or more states execute the executable action using assets to obtain output. The action type may indicate a workflow for the corresponding executable action, or a sequence or set of states to be invoked. For example, if the corresponding executable action has a first action type, the digital assistant may use the first set of states to invoke a workflow for executing that executable action, and if the corresponding executable action has a second action type, the digital assistant may use the second set of states to invoke a workflow for executing the corresponding executable action, wherein the first set of states and the second set of states may be different from each other.

[0107] One or more states may include indications about whether a particular action is ready to be performed, whether more information or additional assets are needed to perform it, whether it has been performed (e.g., successfully or unsuccessfully), and whether it is currently being performed. For example, one or more states may be invoked to perform a particular action type. A first state may be invoked to identify whether an executable action of a particular action type has been performed to generate a response. If, in response to invoking the first state, it is determined that the executable action has been performed and a response has been generated, then iterative processing can continue. If, in response to invoking the first state, it is determined that the executable action has not been performed or a response has not been generated, then a second state may be invoked to determine whether one or more parameters are available for the executable action. If the one or more parameters are unavailable, then the digital assistant may generate a response requesting the one or more parameters from the user. In other embodiments, if the one or more parameters are unavailable, then the digital assistant may generate a prompt to enable a generative artificial intelligence model to recognize or generate the one or more parameters.

[0108] One or more states can be used to perform executable actions using an asset to obtain output. For example, a third state, which may differ from the first and / or second states described above, can be invoked to generate output. The third state can be an execution state that causes the digital assistant to invoke the asset or otherwise initiate an operation to use the asset to generate output. In some embodiments, the output can be populated into a set of outputs provided to an output engine that can be used to generate a response. This set of outputs may include outputs generated by executing each executable action contained in the execution plan.

[0109] In some embodiments, the iterative process may additionally include determining whether one or more parameters are available for an executable action. A specific state may be invoked to identify the one or more parameters or to determine that the one or more parameters are unavailable. In embodiments where the one or more parameters are available, the iterative process may additionally include invoking one or more states as described above and performing the executable action based on the one or more parameters. In examples where the one or more parameters are unavailable, the iterative process may additionally include obtaining the unavailable one or more parameters, then invoking one or more states and performing the executable action based on the one or more parameters. In some embodiments, obtaining the one or more parameters may include generating a natural language request to a user to obtain the one or more parameters of the executable action, and receiving a response from the user, wherein the response may include the one or more parameters. In some embodiments, the iterative process may additionally include determining that there are one or more dependencies between an executable action and at least one other executable action among the one or more executable actions, based on a set of dependencies between the one or more executable actions. The executable actions may be executed sequentially according to the determined one or more dependencies between the executable action and at least one other executable action.

[0110] At point 608, a second prompt is generated based on the output obtained by performing each of one or more executable actions. The second prompt may be generated by an output engine, and the output engine may generate the second prompt based on this set of outputs. The second prompt may include each output in the set of outputs and may include enhanced natural language or other inputs to enable the generative AI model to generate the desired output.

[0111] At point 610, a second generative AI model generates a response to the natural language utterance based on a second prompt. In some embodiments, the second generative AI model may be similar to or the same as the first generative AI model. In other embodiments, the second generative AI model may be different from the first generative AI model. The second prompt may be provided to the second generative AI model so that the second generative AI model generates the response. In some embodiments, the response may be or include natural language text, fields, links, or other suitable components for the response. Natural language text may be or include words, phrases, sentences, etc., that respond to the natural language utterance. In an example where the digital assistant may request additional information from the user, the response may include, along with the natural language text, fields for allowing the user to input information, links to predefined responses or digital locations to find answers, etc. The digital assistant may transmit the response to a computing device associated with the user to present the response to the user, request additional information from the user, etc.

[0112] Examples of cloud infrastructure architectures

[0113] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also offer various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, security software, load balancing software, and clustering software, etc.). Therefore, because these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0114] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0115] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0116] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0117] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0118] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0119] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, one or more inbound / outbound traffic group rules may also be provisioned to define how inbound and / or outbound traffic to the network and one or more virtual machines (VMs) are configured. Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure can evolve incrementally as more and / or additional infrastructure elements are expected and / or added.

[0120] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, it may be necessary to first set up the infrastructure on which the code will be deployed. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0121] Figure 7This is a block diagram 700 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service operator 702 may be communicatively coupled to a secure host lease 704, which may include a virtual cloud network (VCN) 706 and a secure host subnet 708. In some examples, service operator 702 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software (such as Microsoft Windows Mobile®) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and supporting the Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, client computing devices may be general-purpose personal computers, including, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. The client computing device can be a workstation computer running a variety of commercially available UNIX® or UNIX-like operating systems, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, the client computing device can be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without Kinect® gesture input), and / or a personal messaging device capable of communicating over a network that can access VCN 706 and / or the internet.

[0122] VCN 706 may include a local peering gateway (LPG) 710, which may be communicatively coupled to a secure shell (SSH) VCN 712 via the LPG 710 included in SSH VCN 712. SSH VCN 712 may include an SSH subnet 714, and SSH VCN 712 may be communicatively coupled to a control plane VCN 716 via the LPG 710 included in control plane VCN 716. Furthermore, SSH VCN 712 may be communicatively coupled to a data plane VCN 718 via the LPG 710. Control plane VCN 716 and data plane VCN 718 may be contained within a service lease 719 that may be owned and / or operated by an IaaS provider.

[0123] The control plane VCN 716 may include a control plane demilitarized zone (DMZ) layer 720 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 720 may include one or more load balancer (LB) subnets 722, a control plane application layer 724 that may include one or more application subnets 726, and a control plane data layer 728 that may include one or more database (DB) subnets 730 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 722 contained in the control plane DMZ layer 720 may be communicatively coupled to one or more application subnets 726 contained in the control plane application layer 724 and an Internet gateway 734 that may be contained in the control plane VCN 716. The application subnets 726 may be communicatively coupled to one or more DB subnets 730 contained in the control plane data layer 728, as well as a service gateway 736 and a Network Address Translation (NAT) gateway 738. The control plane VCN 716 may include the service gateway 736 and the NAT gateway 738.

[0124] The control plane VCN 716 may include a data plane mirror application layer 740, which may include one or more application subnets 726. The one or more application subnets 726 included in the data plane mirror application layer 740 may include a virtual network interface controller (VNIC) 742 capable of executing a compute instance 744. The compute instance 744 may communicatively couple the one or more application subnets 726 of the data plane mirror application layer 740 to the one or more application subnets 726 that may be included in the data plane application layer 746.

[0125] Data plane VCN 718 may include data plane application layer 746, data plane DMZ layer 748, and data plane data layer 750. Data plane DMZ layer 748 may include one or more LB subnets 722 communicatively coupled to one or more application subnets 726 of data plane application layer 746 and Internet gateway 734 of data plane VCN 718. One or more application subnets 726 may be communicatively coupled to service gateway 736 and NAT gateway 738 of data plane VCN 718. Data plane data layer 750 may also include one or more DB subnets 730 communicatively coupled to one or more application subnets 726 of data plane application layer 746.

[0126] The Internet gateway 734 of the control plane VCN 716 and data plane VCN 718 can be communicatively coupled to the metadata management service 752, which can be communicatively coupled to the public Internet 754. The public Internet 754 can be communicatively coupled to the NAT gateway 738 of the control plane VCN 716 and data plane VCN 718. The service gateway 736 of the control plane VCN 716 and data plane VCN 718 can be communicatively coupled to the cloud service 756.

[0127] In some examples, the service gateway 736 of the control plane VCN 716 or data plane VCN 718 can make application programming interface (API) calls to the cloud service 756 without traversing the public internet 754. API calls from the service gateway 736 to the cloud service 756 can be unidirectional: the service gateway 736 can make API calls to the cloud service 756, and the cloud service 756 can send requested data to the service gateway 736. However, the cloud service 756 may not initiate API calls to the service gateway 736.

[0128] In some examples, secure host lease 704 can be directly connected to service lease 719, which would otherwise be isolated. Secure host subnet 708 can communicate with SSH subnet 714 via LPG 710, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 708 to SSH subnet 714 allows secure host subnet 708 to access other entities within service lease 719.

[0129] Control plane VCN 716 allows users of service lease 719 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 716 can be deployed or otherwise used in data plane VCN 718. In some examples, control plane VCN 716 can be isolated from data plane VCN 718, and the data plane mirror application layer 740 of control plane VCN 716 can communicate with the data plane application layer 746 of data plane VCN 718 via VNIC 742, which can be included in both the data plane mirror application layer 740 and the data plane application layer 746.

[0130] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 754, which can transmit requests to the metadata management service 752. The metadata management service 752 can transmit the request to the control plane VCN 716 via internet gateway 734. The request can be received by one or more LB subnets 722 contained in the control plane DMZ layer 720. The one or more LB subnets 722 can determine that the request is valid, and in response to this determination, the one or more LB subnets 722 can transmit the request to one or more application subnets 726 contained in the control plane application layer 724. If the request is validated and requires a call to the public internet 754, the call to the public internet 754 can be transmitted to a NAT gateway 738 that can make calls to the public internet 754. The request may expect the storage to be located in one or more DB subnets 730.

[0131] In some examples, the data plane mirroring application layer 740 can facilitate direct communication between the control plane VCN 716 and the data plane VCN 718. For example, it might be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 718. Through VNIC 742, the control plane VCN 716 can communicate directly with the resources contained in the data plane VCN 718, and thus can perform configuration changes, updates, or other appropriate modifications.

[0132] In some embodiments, the control plane VCN 716 and data plane VCN 718 may be included in a service lease 719. In this case, the system's users or customers may not own or operate the control plane VCN 716 or the data plane VCN 718. Alternatively, the IaaS provider may own or operate both the control plane VCN 716 and the data plane VCN 718, and both planes may be included in the service lease 719. This embodiment can enable the isolation of networks that might prevent users or customers from interacting with resources of other users or customers. Furthermore, this embodiment can allow users or customers of the system to privately store databases without relying on the public Internet 754, which may not have the desired level of security for storage.

[0133] In other embodiments, one or more LB subnets 722 included in the control plane VCN 716 may be configured to receive signals from the service gateway 736. In this embodiment, the control plane VCN 716 and the data plane VCN 718 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 754. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 719, which may be isolated from the public internet 754.

[0134] Figure 8 This is a block diagram 800 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 802 (e.g., Figure 7 The service provider (702) can communicatively couple to the secure host lease (804) (e.g., Figure 7 Secure hosting lease 704), the secure hosting lease 804 may include a Virtual Cloud Network (VCN) 806 (e.g., Figure 7 VCN 706) and Secure Host Subnet 808 (e.g., Figure 7 The secure host subnet 708). VCN 806 may include a local peering gateway (LPG) 810 (e.g., Figure 7 The LPG 710), which can be communicatively coupled to the Secure Shell (SSH) VCN 812 (e.g., via the LPG 710 contained in the SSH VCN 812) Figure 7 SSH VCN 712). SSH VCN 812 can include SSH subnet 814 (e.g., Figure 7 SSH subnet 714), and SSH VCN 812 can be communicatively coupled to control plane VCN 816 via LPG 810 included in control plane VCN 816 (e.g., Figure 7 Control plane VCN 716). Control plane VCN 816 may be included in service lease 819 (e.g., Figure 7 In the service lease 719), and the data plane VCN 818 (e.g., Figure 7 The data plane (VCN718) may be included in a customer lease 821 that may be owned or operated by the system's users or customers.

[0135] Control plane VCN 816 may include control plane DMZ layer 820 (e.g., Figure 7 The control plane DMZ layer 720), which may include one or more LB subnets 822 (e.g., Figure 7(one or more) LB subnets 722), may include (one or more) application subnets 826 (e.g., Figure 7 The control plane application layer 824 of (one or more) application subnets 726 (e.g., Figure 7 The control plane application layer 724) may include one or more database (DB) subnets 830 (e.g., similar to...). Figure 7 The control plane data layer 828 of (one or more) DB subnets 730 (e.g., Figure 7 The control plane data layer 728). One or more LB subnets 822 contained in the control plane DMZ layer 820 can be communicatively coupled to one or more application subnets 826 contained in the control plane application layer 824 and an Internet gateway 834 that can be contained in the control plane VCN 816 (e.g., Figure 7 Internet gateway 734), and application subnet(s) 826 can communicatively couple to DB subnet(s) 830 contained in control plane data layer 828 and service gateway 836 (e.g., Figure 7 The service gateway) and the Network Address Translation (NAT) gateway 838 (e.g., Figure 7 (NAT gateway 738). The control plane VCN 816 may include the service gateway 836 and the NAT gateway 838.

[0136] The control plane VCN 816 may include a data plane mirror application layer 840 that may contain one or more application subnets 826 (e.g., Figure 7 The data plane mirror application layer 740). One or more application subnets 826 contained in the data plane mirror application layer 840 may include compute instances 844 capable of performing computations (e.g., similar to...). Figure 7 The virtual network interface controller (VNIC) 842 (e.g., the VNIC of 742) of the computing instance 744. The computing instance 844 may facilitate the mirroring of the application subnet(s) 826 of the application layer 840 in the data plane and may be included in the application layer 846 in the data plane (e.g., Figure 7 Communication between one or more application subnets 826 in the data plane application layer 746 via VNIC 842 contained in the data plane mirror application layer 840 and VNIC 842 contained in the data plane application layer 846.

[0137] The Internet gateway 834 included in the control plane VCN 816 can be communicatively coupled to the metadata management service 852 (e.g., Figure 7 Metadata management service 752), which can communicatively couple to the public Internet 854 (e.g., Figure 7The public internet 854 can communicatively couple to a NAT gateway 838 contained in the control plane VCN 816. The service gateway 836 contained in the control plane VCN 816 can communicatively couple to a cloud service 856 (e.g., ...). Figure 7 Cloud services 756).

[0138] In some examples, data plane VCN 818 may be included in customer lease 821. In this case, the IaaS provider may provide control plane VCN 816 for each customer, and the IaaS provider may set up a unique compute instance 844 for each customer, included in service lease 819. Each compute instance 844 may allow communication between control plane VCN 816 included in service lease 819 and data plane VCN 818 included in customer lease 821. Compute instance 844 may allow resources provisioned in control plane VCN 816 included in service lease 819 to be deployed or otherwise used in data plane VCN 818 included in customer lease 821.

[0139] In other examples, an IaaS provider's customer may have a database residing in customer lease 821. In this example, control plane VCN 816 may include data plane mirror application layer 840, which may include one or more application subnets 826. Data plane mirror application layer 840 may reside in data plane VCN 818, but may not reside in data plane VCN 818. That is, data plane mirror application layer 840 may have access to customer lease 821, but may not reside in data plane VCN 818 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 840 may be configured to invoke data plane VCN 818, but may not be configured to invoke any entity contained in control plane VCN 816. Customers may expect to deploy or otherwise use resources provided in the control plane VCN 816 in the data plane VCN 818, and the data plane mirroring application layer 840 can facilitate customers' expected deployments or other uses of resources.

[0140] In some embodiments, an IaaS provider's customer can apply filters to data plane VCN 818. In this embodiment, the customer can determine what data plane VCN 818 can access, and the customer can restrict access from data plane VCN 818 to the public Internet 854. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 818's access to any external networks or databases. Applying filters and controls to data plane VCN 818 contained in customer lease 821 can help isolate data plane VCN 818 from other customers and the public Internet 854.

[0141] In some embodiments, cloud service 856 may be invoked by service gateway 836 to access services that may not exist on public internet 854, control plane VCN 816, or data plane VCN 818. The connection between cloud service 856 and control plane VCN 816 or data plane VCN 818 may not be real-time or continuous. Cloud service 856 may reside on different networks owned or operated by an IaaS provider. Cloud service 856 may be configured to receive calls from service gateway 836 and may be configured not to receive calls from public internet 854. Some cloud services 856 may be isolated from other cloud services 856, and control plane VCN 816 may be isolated from cloud services 856 that may not be in the same region as control plane VCN 816. For example, control plane VCN 816 may be located in "Region 1," and cloud service "Deployment 5" may be located in both "Region 1" and "Region 2." If the service gateway 836, contained in the control plane VCN 816 located in region 1, makes a call to deployment 5, then that call can be transmitted to deployment 5 in region 1. In this example, the control plane VCN 816 or deployment 5 in region 1 may not be communicatively coupled to or otherwise communicate with deployment 5 in region 2.

[0142] Figure 9 This is a block diagram 900 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 902 (e.g., Figure 7 The service provider (702) can communicatively couple to the secure host lease (904) (e.g., Figure 7 Secure hosting lease 704), the secure hosting lease 904 may include a Virtual Cloud Network (VCN) 906 (e.g., Figure 7 VCN 706) and Secure Host Subnet 908 (e.g., Figure 7 The secure host subnet 708). VCN 906 can include LPG 910 (e.g., Figure 7The LPG 710), which can be communicatively coupled to the SSH VCN 912 via the LPG 910 included in the SSH VCN 912 (e.g., Figure 7 SSH VCN 712). SSH VCN 912 can include SSH subnet 914 (e.g., Figure 7 SSH subnet 714), and SSH VCN 912 can be communicatively coupled to control plane VCN 916 via LPG 910 contained in control plane VCN 916 (e.g., Figure 7 The control plane VCN 716) and coupled to the data plane VCN 918 via the LPG 910 contained in the data plane VCN 918 (e.g., Figure 7 Data plane 718). Control plane VCN 916 and data plane VCN 918 may be included in service lease 919 (e.g., Figure 7 (Service rental 719).

[0143] The control plane VCN 916 may include a subnet 922 that may contain one or more load balancer (LB) subnets (e.g., Figure 7 The control plane DMZ layer 920 of (one or more) LB subnets 722) (e.g., Figure 7 The control plane DMZ layer 720 may include one or more application subnets 926 (e.g., similar to...). Figure 7 The control plane application layer 924 of (one or more) application subnets 726 (e.g., Figure 7 The control plane application layer 724), may include (one or more) DB subnets 930, and the control plane data layer 928 (e.g., Figure 7 The control plane data layer 728). One or more LB subnets 922 contained in the control plane DMZ layer 920 can be communicatively coupled to one or more application subnets 926 contained in the control plane application layer 924 and an Internet gateway 934 that can be contained in the control plane VCN 916 (e.g., Figure 7 Internet gateway 734), and application subnet 926(e.g.) can communicatively couple to DB subnet 930(e.g., contained in control plane data layer 928) and service gateway 936(e.g., Figure 7 The service gateway) and Network Address Translation (NAT) gateway 938 (e.g., Figure 7 (NAT gateway 738). The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0144] The data plane VCN 918 may include the data plane application layer 946 (e.g., Figure 7Data plane application layer 746), data plane DMZ layer 948 (e.g., Figure 7 Data plane DMZ layer 748), and data plane data layer 950 (e.g., Figure 7 The data plane data layer 750. The data plane DMZ layer 948 may include one or more trusted application subnets 960 and one or more untrusted application subnets 962 communicatively coupled to the data plane application layer 946, and one or more LB subnets 922 of the Internet gateway 934 contained in the data plane VCN 918. One or more trusted application subnets 960 may be communicatively coupled to the service gateway 936 contained in the data plane VCN 918, the NAT gateway 938 contained in the data plane VCN 918, and one or more DB subnets 930 contained in the data plane data layer 950. One or more untrusted application subnets 962 may be communicatively coupled to the service gateway 936 contained in the data plane VCN 918 and one or more DB subnets 930 contained in the data plane data layer 950. The data plane data layer 950 may include one or more DB subnets 930 communicatively coupled to the service gateway 936 contained in the data plane VCN 918.

[0145] One or more untrusted application subnets 962 may include one or more primary VNICs 964(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 966(1)-(N). Each tenant VM 966(1)-(N) may be communicatively coupled to a corresponding application subnet 967(1)-(N) that may be contained in a corresponding container egress VCN 968(1)-(N), which may be contained in a corresponding customer lease 970(1)-(N). A corresponding secondary VNIC 972(1)-(N) may facilitate communication between one or more untrusted application subnets 962 contained in a data plane VCN 918 and the application subnets contained in the container egress VCN 968(1)-(N). Each container egress VCN 968(1)-(N) may include a NAT gateway 938 that can be communicatively coupled to the public Internet 954 (e.g., Figure 7 The public internet (754).

[0146] An Internet gateway 934, contained in the control plane VCN 916 and the data plane VCN 918, can be communicatively coupled to a metadata management service 952 (e.g., Figure 7A metadata management system 752 is provided, which can communicatively couple to the public internet 954. The public internet 954 can communicatively couple to a NAT gateway 938 contained in a control plane VCN 916 and a data plane VCN 918. A service gateway 936 contained in a control plane VCN 916 and a data plane VCN 918 can communicatively couple to a cloud service 956.

[0147] In some embodiments, the data plane VCN 918 can be integrated with the customer lease 970. Such integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support might be expected during code execution. Customers may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider can determine whether to run the code provided by the customer.

[0148] In some examples, an IaaS provider's customer can grant temporary network access to the IaaS provider and request functionality attached to the data plane application layer 946. The code running this functionality can execute in VMs 966(1)-(N), and this code may not be configured to run anywhere else on the data plane VCN 918. Each VM 966(1)-(N) can be connected to a customer lease 970. The corresponding container 971(1)-(N) contained in VM 966(1)-(N) can be configured to run the code. In this case, there can be dual isolation (e.g., container 971(1)-(N) runs the code, where container 971(1)-(N) may be contained in at least one or more untrusted application subnets 962 containing VM 966(1)-(N)), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Container 971(1)-(N) may be communicatively coupled to customer lease 970 and may be configured to transmit or receive data from customer lease 970. Container 971(1)-(N) may not be configured to transmit or receive data from any other entity in data plane VCN 918. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of container 971(1)-(N).

[0149] In some embodiments, one or more trusted application subnets 960 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 960 may be communicatively coupled to one or more database subnets 930 and configured to perform CRUD operations in one or more database subnets 930. One or more untrusted application subnets 962 may be communicatively coupled to one or more database subnets 930, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 930. Containers 971(1)-(N) that may be contained in each customer's VM 966(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 930.

[0150] In other embodiments, the control plane VCN 916 and the data plane VCN 918 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 916 and the data plane VCN 918. However, communication can occur indirectly through at least one method. The LPG 910 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 916 and the data plane VCN 918. In another example, either the control plane VCN 916 or the data plane VCN 918 may invoke the cloud service 956 via the service gateway 936. For example, an invocation of the cloud service 956 from the control plane VCN 916 may include a request for a service that can communicate with the data plane VCN 918.

[0151] Figure 10 This is a block diagram 1000 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1002 (e.g., Figure 7 The service provider 702 can communicatively couple to the secure host lease 1004 (e.g., Figure 7 Secure hosting lease 704), the secure hosting lease 1004 may include a virtual cloud network (VCN) 1006 (e.g., Figure 7 VCN 706) and Secure Host Subnet 1008 (e.g., Figure 7 The secure host subnet 708). VCN 1006 may include LPG1010 (e.g., Figure 7 The LPG 710), the LPG 1010 can be accessed via SSH VCN 1012 (e.g., LPG 710), Figure 7 The LPG 1010 in SSH VCN 1012 is communicatively coupled to SSH VCN 1012. SSH VCN 1012 may include SSH subnet 1014 (e.g., Figure 7 SSH subnet 714), and SSH VCN 1012 can be communicatively coupled to control plane VCN 1016 via LPG1010 included in control plane VCN 1016 (e.g., Figure 7 The control plane VCN 716) and coupled to the data plane VCN 1018 via the LPG 1010 contained in the data plane VCN 1018 (e.g., Figure 7 Data plane 718). Control plane VCN 1016 and data plane VCN 1018 may be included in service lease 1019 (e.g., Figure 7 (Service rental 719).

[0152] The control plane VCN 1016 may include one or more LB subnets 1022 (e.g., Figure 7 The control plane DMZ layer 1020 of (one or more) LB subnets 722) (e.g., Figure 7 The control plane DMZ layer 720 may include one or more application subnets 1026 (e.g., Figure 7 The control plane application layer 1024 of (one or more) application subnets 726 (e.g., Figure 7 The control plane application layer 724) may include (one or more) DB subnets 1030 (e.g., Figure 9 The control plane data layer 1028 of (one or more) DB subnets 930 (e.g., Figure 7 The control plane data layer 728). One or more LB subnets 1022 contained in the control plane DMZ layer 1020 can be communicatively coupled to one or more application subnets 1026 contained in the control plane application layer 1024 and an Internet gateway 1034 that can be contained in the control plane VCN 1016 (e.g., Figure 7 Internet gateway 734), and application subnet(s) 1026 can communicatively couple to DB subnet(s) 1030 contained in control plane data layer 1028 and service gateway 1036 (e.g., Figure 7 The service gateway) and Network Address Translation (NAT) gateway 1038 (e.g., Figure 7 (NAT gateway 738). The control plane VCN 1016 may include the service gateway 1036 and the NAT gateway 1038.

[0153] Data plane VCN 1018 may include data plane application layer 1046 (e.g., Figure 7 Data plane application layer 746), data plane DMZ layer 1048 (e.g., Figure 7Data plane DMZ layer 748), and data plane data layer 1050 (e.g., Figure 7 The data plane data layer 750). The data plane DMZ layer 1048 may include one or more trusted application subnets 1060 that can be communicatively coupled to the data plane application layer 1046 (e.g., Figure 9 (one or more) trusted application subnets 960 and (one or more) untrusted application subnets 1062 (e.g., Figure 9 The data plane VCN 1018 includes one or more untrusted application subnets 962 and one or more LB subnets 1022 of Internet gateway 1034. One or more trusted application subnets 1060 may communicatively couple to service gateway 1036, NAT gateway 1038, and DB subnets 1030 in data plane VCN 1050. One or more untrusted application subnets 1062 may communicatively couple to service gateway 1036 and DB subnets 1030 in data plane VCN 1018. Data plane VCN 1050 may include one or more DB subnets 1030 that may communicatively couple to service gateway 1036 in data plane VCN 1018.

[0154] One or more untrusted application subnets 1062 may include a primary VNIC 1064(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1066(1)-(N) residing within one or more untrusted application subnets 1062. Each tenant VM 1066(1)-(N) may run code in a corresponding container 1067(1)-(N) and is communicatively coupled to an application subnet 1026 that may be contained in a data plane application layer 1046 contained in a container egress VCN 1068. A corresponding secondary VNIC 1072(1)-(N) may facilitate communication between one or more untrusted application subnets 1062 contained in a data plane VCN 1018 and the application subnet contained in a container egress VCN 1068. The container egress VCN may include a public internet 1054 (e.g., Figure 7 The public internet (754) uses NAT gateway 1038.

[0155] Internet gateway 1034, contained in control plane VCN 1016 and data plane VCN 1018, can be communicatively coupled to metadata management service 1052 (e.g., Figure 7A metadata management system 752 is provided, which can communicatively couple to the public internet 1054. The public internet 1054 can communicatively couple to a NAT gateway 1038 contained in a control plane VCN 1016 and a data plane VCN 1018. A service gateway 1036 contained in a control plane VCN 1016 and a data plane VCN 1018 can communicatively couple to a cloud service 1056.

[0156] In some examples, Figure 10 The architecture shown in block diagram 1000 can be considered as... Figure 9 This is an exception to the pattern shown in block diagram 900, and this pattern may be what the IaaS provider's customers expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 1067(1)-(N) contained in each customer's VM 1066(1)-(N). Container 1067(1)-(N) can be configured to invoke a corresponding auxiliary VNIC 1072(1)-(N) contained in one or more application subnets 1026 of the data plane application layer 1046, which may be contained in a container egress VCN 1068. The auxiliary VNIC 1072(1)-(N) can transmit the invocation to a NAT gateway 1038, which can then transmit the invocation to the public internet 1054. In this example, containers 1067(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1016 and from other entities contained in the data plane VCN 1018. Containers 1067(1)-(N) can also be isolated from resources from other clients.

[0157] In other examples, a client can use containers 1067(1)-(N) to invoke cloud service 1056. In this example, the client can run code within containers 1067(1)-(N) requesting a service from cloud service 1056. Container 1067(1)-(N) can then forward the request to a secondary VNIC 1072(1)-(N), which can then forward the request to a NAT gateway, which can forward the request to the public internet 1054. The public internet 1054 can then forward the request via internet gateway 1034 to one or more LB subnets 1022 contained in control plane VCN 1016. In response to determining that the request is valid, one or more LB subnets can forward the request to one or more application subnets 1026, which can then forward the request to cloud service 1056 via service gateway 1036.

[0158] It should be recognized that the IaaS architectures 700, 800, 900, and 1000 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0159] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0160] Examples of computer systems or devices

[0161] Figure 11 An example computer system 1100 in which various embodiments can be implemented is illustrated. System 1100 can be used to implement any of the computer systems or processing systems described above. As shown, computer system 1100 includes a processing unit 1104 that communicates with a plurality of peripheral subsystems via a bus subsystem 1102. These peripheral subsystems may include a processing acceleration unit 1106, an I / O subsystem 1108, a storage subsystem 1118, and a communication subsystem 1124. Storage subsystem 1118 includes a tangible computer-readable storage medium 1122 and system memory 1110.

[0162] Bus subsystem 1102 provides a mechanism for allowing various components and subsystems of computer system 1100 to communicate with each other as intended. While bus subsystem 1102 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1102 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures available. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0163] A processing unit 1104, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1100. One or more processors may be included in the processing unit 1104. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1104 may be implemented as one or more independent processing units 1132 and / or 1134, wherein each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 1104 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0164] In various embodiments, processing unit 1104 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 1104 and / or storage subsystem 1118. With appropriate programming, processor(s) 1104 can provide the various functions described above. Computer system 1100 may additionally include processing acceleration unit 1106, which may include digital signal processor (DSP), dedicated processor, etc.

[0165] I / O subsystem 1108 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, keyboards, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices such as the Microsoft Xbox® 360 game controller via a natural user interface using gestures and voice commands. User interface input devices may also include eye posture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., “blinking” when taking a photo and / or making menu selections) and translates the eye posture into input in an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0166] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0167] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1100 to a user or other computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0168] Computer system 1100 may include storage subsystem 1118 that provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 1104, provide the aforementioned functionality. Storage subsystem 1118 may also provide a repository for storing data used according to this disclosure.

[0169] As in Figure 11 As illustrated in the example, storage subsystem 1118 may include various components, including system memory 1110, computer-readable storage medium 1122, and computer-readable storage medium reader 1120. System memory 1110 may store program instructions that can be loaded and executed by processing unit 1104. System memory 1110 may also store data used during instruction execution and / or data generated during program instruction execution. Various types of programs may be loaded into system memory 1110, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0170] System memory 1110 may also store operating system 1116. Examples of operating system 1116 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS and Palm® OS. In some implementations where computer system 1100 executes one or more virtual machines, the virtual machine, along with its guest operating system (GOS), may be loaded into system memory 1110 and executed by one or more processors or cores of processing unit 1104.

[0171] Depending on the type of computer system 1100, system memory 1110 can have different configurations. For example, system memory 1110 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, system memory 1110 may include a basic input / output system (BIOS), which contains basic routines that facilitate, for example, the transfer of information between elements within computer system 1100 during startup.

[0172] Computer-readable storage medium 1122 may represent a remote, local, fixed and / or removable storage device, as well as a storage medium for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 1104 of computer system 1100) for use by computer system 1100.

[0173] Computer-readable storage medium 1122 may include any suitable medium known or used in the art, including storage and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0174] For example, computer-readable storage medium 1122 may include a hard disk drive that reads or writes to a non-removable non-volatile magnetic medium, a disk drive that reads or writes to a removable non-volatile magnetic disk, and an optical disc drive that reads or writes to a removable non-volatile optical disc (such as a CD-ROM, DVD, and Blu-ray® disc or other optical media). Computer-readable storage medium 1122 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital audio tapes, and so on. Computer-readable storage medium 1122 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules and other data for computer system 1100.

[0175] Machine-readable instructions executable by one or more processors or cores of processing unit 1104 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0176] The communication subsystem 1124 provides an interface to other computer systems and networks. The communication subsystem 1124 serves as an interface for receiving data from other systems and sending data from computer system 1100 to other systems. For example, the communication subsystem 1124 enables computer system 1100 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1124 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technologies, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), Wi-Fi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 1124 may provide a wired network connection (e.g., Ethernet).

[0177] In some embodiments, the communication subsystem 1124 may also represent one or more users who can use the computer system 1100 to receive input communications in the form of structured and / or unstructured data feeds 1126, event streams 1128, event updates 1130, etc.

[0178] As an example, the communication subsystem 1124 may be configured to receive data feeds 1126 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0179] Furthermore, the communication subsystem 1124 can also be configured to receive data in the form of a continuous data stream, which may include an event stream 1128 and / or event updates 1130 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0180] The communication subsystem 1124 can also be configured to output structured and / or unstructured data feeds 1126, event streams 1128, event updates 1130, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1100.

[0181] The computer system 1100 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing systems.

[0182] Due to the ever-evolving nature of computers and networks, the description of the computer system 1100 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or combinations thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0183] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalent forms are possible. The embodiments are not limited to operation within certain specific data processing environments but can operate freely within multiple data processing environments. Furthermore, although certain embodiments have been described using specific series of processes and steps, this is not intended to be restrictive. While some flowcharts describe operations as sequential processes, many operations can be executed in parallel or concurrently. Moreover, the order of operations can be rearranged. Processes may have additional steps not included in the figures. Various features and aspects of the above embodiments can be used individually or in combination.

[0184] Furthermore, while specific combinations of hardware and software have been described using examples, it should be recognized that other combinations of hardware and software are also possible. Some embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors.

[0185] When a device, system, component, or module is described as being configured to perform certain operations or functions, such configuration can be achieved, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations (such as by executing computer instructions or code), or by being programmed to execute a processor or core of code or instructions stored on a non-transitory storage medium, or any combination thereof. Processes can communicate using various technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes can use different technologies, or the same pair of processes can use different technologies at different times.

[0186] The embodiments may be implemented using a computer program product comprising a computer program / instructions that, when executed by a processor, cause the processor to perform any of the methods described in the disclosure.

[0187] Specific details are set forth in this disclosure to provide a thorough understanding of the embodiments. However, embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the embodiments. This specification provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the foregoing description of the embodiments provides an enabling description for implementing various embodiments. Various changes can be made to the functionality and arrangement of the elements.

[0188] Accordingly, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific embodiments have been described, they are not intended to be restrictive. Various modifications and equivalents are within the scope of the following claims.

Claims

1. A computer-implemented method, comprising: The first generative artificial intelligence model generates a list of one or more executable actions based on a first prompt, including natural language utterances provided by the user; Create an execution plan that includes the one or more executable actions; Executing the execution plan, wherein executing the execution plan includes performing iterative processing on each of the one or more executable actions, and wherein the iterative processing includes: Identify the action type for executable actions. The call is configured to execute one or more states of the action type, and The executable action is performed using the assets based on the one or more states to obtain output; A second prompt is generated based on the output obtained by executing each of the one or more executable actions; and The second generative artificial intelligence model generates a response to the natural language utterance based on the second prompt.

2. The computer-implemented method as described in claim 1, wherein: Creating the execution plan includes evaluating the one or more executable actions; The evaluation includes assessing the one or more executable actions based on one or more ongoing conversation paths initiated by the user and the execution plan of any current activities; and Creating the execution plan also includes: (i) When the assessment determines that the natural language utterance is part of an ongoing conversation path, the one or more executable actions are incorporated into an execution plan for a current activity associated with the ongoing conversation path, the execution plan for the current activity comprising an ordered list of the one or more executable actions and one or more prior actions, or (ii) When the assessment determines that the natural language utterance is not part of an ongoing conversation path, a new execution plan is created that includes an ordered list of the one or more executable actions.

3. The computer-implemented method of claim 1, wherein the iterative process further comprises: Determine whether one or more parameters are available for the executable action; When one or more parameters are available, invoke one or more states and execute the executable action based on one or more parameters; and When one or more parameters for the executable action are unavailable, obtain the unavailable one or more parameters, then invoke the one or more states and execute the executable action based on the one or more parameters.

4. The computer-implemented method of claim 3, wherein obtaining the one or more parameters comprises generating a natural language request to the user to obtain the one or more parameters of the executable action, and receiving a response from the user including the one or more parameters.

5. The computer-implemented method according to any one of claims 1 to 4, wherein: Invoking one or more states configured to perform the action type includes: Invoke the first state to identify that the executable action has not yet been executed in order to generate a response, and Invoke the second state to determine whether one or more parameters are available for the executable action; Using the asset to perform the executable action to obtain output includes invoking a third state to generate output; and The first state, the second state, and the third state are different from each other.

6. The computer-implemented method of any one of claims 1 to 4, wherein generating the list comprises selecting the one or more executable actions from a list of candidate agent actions determined by using semantic indexing, and wherein creating the execution plan further comprises: The one or more executable actions that provide information or knowledge for generating a response to the natural language utterance are identified, based at least in part on metadata associated with candidate agent actions within the list of candidate agent actions. as well as By creating an ordered list of the one or more executable actions and a set of dependencies between the one or more executable actions, a structured output is generated for the execution plan.

7. The computer-implemented method of claim 6, wherein the iterative process further comprises determining, based on the set of dependencies between the one or more executable actions, that there exists one or more dependencies between the executable action and at least one other executable action among the one or more executable actions, and wherein the executable actions are executed sequentially according to the one or more dependencies determined to exist between the executable action and the at least one other executable action.

8. A system comprising: One or more processors; as well as A computer-readable medium storing one or more instructions, which, when executed by the one or more processors, cause the system to perform operations, including: The first generative artificial intelligence model generates a list of one or more executable actions based on a first prompt, including natural language utterances provided by the user; Create an execution plan that includes the one or more executable actions; Executing the execution plan, wherein executing the execution plan includes performing iterative processing on each of the one or more executable actions, and wherein the iterative processing includes: Identify the action type for executable actions. The call is configured to execute one or more states of the action type, and The executable action is performed using the assets based on the one or more states to obtain output; A second prompt is generated based on the output obtained by executing each of the one or more executable actions; and The second generative artificial intelligence model generates a response to the natural language utterance based on the second prompt.

9. The system of claim 8, wherein: Creating the execution plan includes evaluating the one or more executable actions; The evaluation includes assessing the one or more executable actions based on one or more ongoing conversation paths initiated by the user and the execution plan of any current activities; and The operation of creating the execution plan also includes: (i) When the assessment determines that the natural language utterance is part of an ongoing conversation path, the one or more executable actions are incorporated into an execution plan for a current activity associated with the ongoing conversation path, the execution plan for the current activity comprising an ordered list of the one or more executable actions and one or more prior actions, or (ii) When the assessment determines that the natural language utterance is not part of an ongoing conversation path, a new execution plan is created that includes an ordered list of the one or more executable actions.

10. The system of claim 8, wherein the iterative process further comprises: Determine whether one or more parameters are available for the executable action; When one or more parameters are available, invoke one or more states and execute the executable action based on one or more parameters; and When one or more parameters for the executable action are unavailable, obtain the unavailable one or more parameters, then invoke the one or more states and execute the executable action based on the one or more parameters.

11. The system of claim 10, wherein obtaining the one or more parameters comprises generating a natural language request to the user to obtain the one or more parameters of the executable action, and receiving a response from the user including the one or more parameters.

12. The system according to any one of claims 8 to 11, wherein: Invoking an operation that is configured to perform one or more states of the action type includes: Invoke the first state to identify that the executable action has not yet been executed in order to generate a response, and Invoke the second state to determine whether one or more parameters are available for the executable action; The operation of using the asset to perform the executable action to obtain output includes invoking a third state to generate output; and The first state, the second state, and the third state are different from each other.

13. The system of any one of claims 8 to 11, wherein generating the list includes selecting the one or more executable actions from a list of candidate agent actions determined by using semantic indexing, and wherein creating the execution plan further includes: The one or more executable actions that provide information or knowledge for generating a response to the natural language utterance are identified, based at least in part on metadata associated with candidate agent actions within the list of candidate agent actions. as well as By creating an ordered list of the one or more executable actions and a set of dependencies between the one or more executable actions, a structured output is generated for the execution plan.

14. The system of claim 13, wherein the iterative processing further comprises determining, based on the set of dependencies between the one or more executable actions, that there is one or more dependencies between the executable action and at least one other executable action among the one or more executable actions, and wherein the executable actions are executed sequentially according to the one or more dependencies determined to exist between the executable action and the at least one other executable action.

15. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, including: The first generative artificial intelligence model generates a list of one or more executable actions based on a first prompt, including natural language utterances provided by the user; Create an execution plan that includes the one or more executable actions; Executing the execution plan, wherein executing the execution plan includes performing iterative processing on each of the one or more executable actions, and wherein the iterative processing includes: Identify the action type for executable actions. The call is configured to execute one or more states of the action type, and The executable action is performed using the assets based on the one or more states to obtain output; A second prompt is generated based on the output obtained by executing each of the one or more executable actions; and The second generative artificial intelligence model generates a response to the natural language utterance based on the second prompt.

16. One or more non-transitory computer-readable media as described in claim 15, wherein: Creating the execution plan includes evaluating the one or more executable actions; The evaluation includes assessing the one or more executable actions based on one or more ongoing conversation paths initiated by the user and the execution plan of any current activities; and The operation of creating the execution plan also includes: (i) When the assessment determines that the natural language utterance is part of an ongoing conversation path, the one or more executable actions are incorporated into an execution plan for a current activity associated with the ongoing conversation path, the execution plan for the current activity comprising an ordered list of the one or more executable actions and one or more prior actions, or (ii) When the assessment determines that the natural language utterance is not part of an ongoing conversation path, a new execution plan is created that includes an ordered list of the one or more executable actions.

17. One or more non-transitory computer-readable media as claimed in claim 15, wherein the iterative process further comprises: Determine whether one or more parameters are available for the executable action; When one or more parameters are available, invoke one or more states and execute the executable action based on one or more parameters; and When one or more parameters for the executable action are unavailable, the unavailable one or more parameters are obtained, and then the one or more states are invoked and the executable action is executed based on the one or more parameters, wherein obtaining the one or more parameters includes generating a natural language request to the user to obtain the one or more parameters for the executable action, and receiving a response from the user including the one or more parameters.

18. One or more non-transitory computer-readable media as described in any one of claims 15-17, wherein: Invoking an operation that is configured to perform one or more states of the action type includes: Invoke the first state to identify that the executable action has not yet been executed in order to generate a response, and Invoke the second state to determine whether one or more parameters are available for the executable action; The operation of using the asset to perform the executable action to obtain output includes invoking a third state to generate output; and The first state, the second state, and the third state are different from each other.

19. One or more non-transitory computer-readable media as claimed in any one of claims 15-17, wherein generating the list includes selecting the one or more executable actions from a list of candidate agent actions determined by using semantic indexing, and wherein creating the execution plan further includes: The one or more executable actions that provide information or knowledge for generating a response to the natural language utterance are identified, based at least in part on metadata associated with candidate agent actions within the list of candidate agent actions. as well as By creating an ordered list of the one or more executable actions and a set of dependencies between the one or more executable actions, a structured output is generated for the execution plan.

20. The one or more non-transitory computer-readable media of claim 19, wherein the iterative processing further comprises determining, based on the set of dependencies between the one or more executable actions, that there exists one or more dependencies between the executable action and at least one other executable action among the one or more executable actions, and wherein the executable actions are executed sequentially according to the one or more dependencies determined to exist between the executable action and the at least one other executable action.

21. An apparatus comprising components for implementing the computer-implemented method as described in any one of claims 1-7.

22. A computer program product comprising computer instructions that, when executed by a processor, implement the operation of the computer-implemented method as described in any one of claims 1-7.

Images