A centralized multi-agent platform for accurate and low-latency processing of user requests.
The centralized multi-agent platform addresses the limitations of general-purpose AI models by using specialized agents and orchestrator-driven query processing, enhancing accuracy and reducing latency in complex user request handling.
Patent Information
- Application Number
- JP2026079066
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2026-04-28
- Filing Date
- 2026-05-08
- Publication Date
- 2026-11-19
AI Technical Summary
Existing automated query systems lack flexibility and accuracy in handling complex user requests due to reliance on general-purpose AI models, leading to increased latency and computational costs, and are prone to hallucinations and context limitations.
A centralized multi-agent platform that coordinates specialized AI agents, using an orchestrator to generate response deployment plans and synthetic queries, reducing reliance on large models and minimizing communication loops to enhance accuracy and reduce latency.
The system achieves improved factual accuracy from 87% to 97% and response latency from 8 seconds to 3 seconds by leveraging specialized AI agents and maintaining isolated contexts, thus providing efficient and flexible query processing.
Smart Images

Figure 2026182855000001_ABST
Abstract
Description
TECHNICAL FIELD
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 803,443, filed May 9, 2025, entitled "Centralized Multi-Agent Platform for Accurate and Low-Latency Handing of User Requests", the entire disclosure of which is incorporated herein by reference.
[0002] The present disclosure relates to query processing technology, and more specifically, to a technology for providing accurate user request responses with low latency using a centralized architecture of artificial intelligence agents. BACKGROUND ART
[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. The achievements of the inventor(s) cited in this specification, to the extent described in this Background Art section, as well as aspects of the specification that may not be recognized as prior art at the time of filing, are not explicitly or implicitly admitted as prior art to the present disclosure.
[0004] In some fields or use cases, users of computer / software-based platforms may have needs (questions, concerns, interests, etc.) that can vary widely in focus, scope, and generality. In the field of digital advertising, for example, a user / entity with a digital advertising account might want to know how their ads are performing, how they can improve the performance of their ad campaigns (or specific aspects of a particular ad campaign), or why a particular ad was not approved by the platform provider. However, users may need to go through a variety of cumbersome procedures to address these needs. For example, an advertising entity might query analytics tools to review past ad performance, configure various parameters, communicate with consultants / experts about techniques to optimize ad campaign performance, or communicate with the platform provider to inquire about why a particular ad was not approved.
[0005] Automated query systems (e.g., user "help" tools) can provide a relatively quick, efficient, and low-cost way to address some of the user problems that arise with a particular application, tool, or platform. However, such systems typically lack the flexibility / ability to provide support for a wide range of issues, especially for complex platforms with numerous components, functions, and other operational aspects. While recent improvements in Large Language Models (LLMs) have made automated query systems more flexible, their applicability remains generally limited due to factors such as vulnerability to hallucinations, limitations in context window size, and the general impossibility of training models of a reasonable size to competently handle a large number of different domains / aspects. Very large models and / or very complex systems can address some of these problems, but this usually comes at the cost of increased processing resource usage and increased latency / delay in providing responses to user queries. In particular, in chat-style platforms, even a moderate amount of "turn-by-turn" latency may be unacceptable. [Overview of the Initiative]
[0006] The technology of this disclosure can solve the aforementioned shortcomings and other technical problems by providing a systematic, efficient, and automated query processing framework that uses an orchestrator to coordinate the actions of multiple specialized AI agents (e.g., reporting agents, diagnostic agents, optimization agents, billing agents, policy agents, etc.). Depending on the embodiment, the orchestrator provides planning and response functions by using separate planner and response models or by using a single model. In response to receiving a user request (e.g., a query or command), the orchestrator generates a response deployment plan by determining, at least in part, which specialized AI agents are relevant to the user request, and generates a synthetic query (e.g., creates a prompt) to the relevant AI agents according to the response deployment plan. As used herein, unless explicitly indicated otherwise by the context of use, the terms “request” or “query” may refer, for example, a single request or query transmission / message, or a set of request or query transmissions / messages (e.g., in the course of a multi-turn “conversation”). Based on the feedback received in response to the synthetic query, the orchestrator generates a response to present to the user. By separating responsibilities between the planner and the response stage or component in this way, faster updates or optimizations can be enabled, which helps ensure that the updates or optimizations yield more predictable results.
[0007] Furthermore, by using specialized or skill-specific AI agents rather than relying on a single general-purpose agent or model, the disclosed technology can provide more detailed expertise and / or context specificity in a particular domain without requiring a single large model that is, for example, computationally intensive, slow, and / or difficult to train, or impractical due to context windows and / or other limitations. In addition, by routing queries to specialized AI agents, the disclosed technology avoids reliance on a single large model that can be hallucinatory, and thus reduces the risk of inaccurate, irrelevant, or false information. Each specialized AI agent is specifically trained / configured for its respective domain, thus mitigating the inaccuracies and misinformation that commonly arise from general-purpose models.
[0008] Therefore, the disclosed technology strikes a balance by providing the ability to accurately and flexibly handle a wide range of complex user requests without requiring computationally expensive, slow, and / or overly complex general-purpose models. Furthermore, the ability to add or remove specific specialized AI agents as needed makes the disclosed architecture highly flexible and scalable, and the centralized architecture of the orchestrator and specialized AI agents enables more efficient resolution of conflicts and enforcement of desired policies, among other things. In addition, the disclosed technology can reduce latency and processing resource usage by resolving inconsistencies, conflicts, lack of conversational fluidity, etc., in a more efficient and centralized manner.
[0009] The disclosed technology further helps ensure that the final response is more relevant to the user who entered the request and / or to other entities associated with the request, such as the advertiser on which the request was entered, by leveraging contextual signals to generate response deployment plans and / or synthetic queries for specialized AI agents. For example, an orchestrator may create a response deployment plan based in part on information about the advertiser associated with the user request, or information about the advertiser's account or business. In some embodiments, the disclosed technology provides even greater contextual relevance by using one or more signals related to the user interface on which the user request was entered. For example, an orchestrator may create a response deployment plan by in part analyzing information about the current state of the user interface (e.g., capturing a screenshot of which page or part of a page the user is currently viewing) to more accurately determine the specialized AI agent associated with the user request.
[0010] In some embodiments, further technical advantages can be obtained by adapting the behavior of the specialized AI agent, the response feedback that the specialized AI agent provides to the orchestrator, and / or how the orchestrator uses the response feedback from the specialized AI agent. For example, when the specialized AI agent receives a query from the orchestrator (according to a response deployment plan), the specialized AI agent may respond by sending to the orchestrator grounding data that the specialized AI agent uses to generate the intermediate inference and which will be deployed to support the response to the query (e.g., to support the response typically generated by the response layer model of the specialized AI agent), and not necessarily generating and / or sending the higher-layer query response itself (or, in some embodiments, the orchestrator receives the higher-layer query response but ignores it). In this way, the response generation task of the specialized agent is effectively delegated to the orchestrator, which generates a response to the user request by using the intermediate inference and grounding data (and possibly other information from the specialized AI agent, such as instructions on how to read the grounding data). By receiving underlying inference and grounding data, the orchestrator obtains enough information to prevent the need to query the specialized AI agent again (at least in some cases). By eliminating loops / rounds (or multiple loops / rounds) in orchestrator / agent communication, such embodiments can reduce the total latency of responses provided to the user while reducing the consumption of processing power. Furthermore, (given the more advanced domain expertise of the specialized AI agent) counterintuitively, hallucinations can be reduced by eliminating the layer in which the specialized AI agent generates its response to the orchestrator query, because the conclusions or inferences derived by the specialized AI agent's response model can be a significant source of hallucinations.
[0011] In some embodiments, further technical advantages can be obtained by storing / maintaining context for at least some of the specialized AI agents. For example, for the purpose of generating response deployment plans, an orchestrator may maintain a first agent context / history showing past communications (e.g., queries / feedback) between the orchestrator and a first specialized AI agent, and separately maintain a second agent context / history showing past communications between the orchestrator and a second specialized AI agent. Centralizing the maintenance of isolated agent-specific contexts / hisories effectively provides a much wider context window (or equivalently, the specialized AI agents themselves do not need to maintain a larger context), while at the same time reducing the possibility of hallucinations, as each specialized AI agent can focus more on its own domain without cross-context contamination (e.g., the specialized AI agent does not need to analyze / consider irrelevant data / history in synthetic queries).
[0012] Experiments have shown that in a system where the orchestrator generates responses to user queries based on grounding data and intermediate inference (rather than higher-level responses) from specialized AI agents, and the orchestrator maintains separate / isolated contexts for different specialized AI agents, factual accuracy (illusions not based on fact) can improve from approximately 87% to approximately 97%, and response latency to the user can improve from approximately 8 seconds to approximately 3 seconds. For example, such improvements are possible because the system avoids illusions that tend to occur in the response layer of the specialized AI agents, and there are fewer turns / rounds of communication between the orchestrator and the specialized AI agents.
[0013] In some embodiments and / or scenarios, the orchestrator responds to an initial user request by performing or triggering an action, at least in part. For example, the orchestrator may identify one or more actions to be performed via a user interface, and the orchestrator (or another component or device / system) may then remotely gain control of the input functionality of a client device to perform the identified action(s) (e.g., observing the corresponding movement of a mouse pointer as the client device user interacts with the user interface).
[0014] Other advantages will become apparent to those skilled in the art by reading this disclosure and examining the corresponding drawings. [Brief explanation of the drawing]
[0015] [Figure 1] This is a block diagram of an exemplary system in which the user request processing technology of this disclosure may be implemented. [Figure 2] Figure 1 shows an exemplary architecture that can be implemented by the computational system. [Figure 3] Figure 2 shows an exemplary set of specialized artificial intelligence (AI) agents and tools that may be used by the computing system in Figure 1 within the architecture of Figure 2. [Figure 4] In exemplary scenarios and embodiments, the messaging diagram represents a sequence of messages that may be exchanged within the computing system of Figure 1 and / or the architecture of Figure 2. [Figure 5] In an alternative embodiment, a messaging diagram is provided that represents other sequences of messages that may be exchanged within the computing system of Figure 1 and / or the architecture of Figure 2. [Figure 6] This illustrates an exemplary process that can be implemented by one or more specialized AI agents. [Figure 7A]In various exemplary scenarios and embodiments, the UI components in Figure 1 illustrate user interfaces (UIs) that can be presented to the user. [Figure 7B] In various exemplary scenarios and embodiments, the UI components in Figure 1 illustrate user interfaces (UIs) that can be presented to the user. [Figure 7C] In various exemplary scenarios and embodiments, the UI components in Figure 1 illustrate user interfaces (UIs) that can be presented to the user. [Figure 7D] In various exemplary scenarios and embodiments, the UI components in Figure 1 illustrate user interfaces (UIs) that can be presented to the user. [Figure 8] This is a flowchart illustrating an exemplary method for processing user requests accurately and with low latency. [Modes for carrying out the invention]
[0016] Figure 1 is a block diagram of an exemplary system 100 in which the technology of the present disclosure for processing user requests may be implemented. The exemplary system 100 includes a computing system 102, a client device 104, one or more servers 106, and a network 110. The computing system 102 may be remote from the client device 104 and / or server(s) 106 and may be communicatively coupled to the client device 104 and server(s) 106 via the network 110.
[0017] Network 110 may be a single communication network (e.g., the Internet), and in some embodiments, it may include one or more additional networks. As just one example, network 110 may include a cellular network, the Internet, and a server-side local area network (LAN). While Figure 1 shows only a single client device 104, it will be understood that the computing system 102 can communicate with many other client devices (e.g., thousands) that are generally similar to client device 104.
[0018] Generally, the computing system 102 can perform user request processing services for entities (e.g., as services / functions within a more comprehensive set of services), and the client device 104 is generally configured to receive and provide information via one or more user interfaces (e.g., a web page and / or a mobile application or other application user interface) that enable interaction with the computing system 102. For example, the entity associated with the computing system 102 may be a provider of digital advertising services, and the entity associated with the client device 104 may be a specific advertiser with an account with the provider. In such a case, a user of the client device 104 can configure and / or monitor one or more advertising campaigns associated with the advertising account (e.g., by creating a new campaign, placing ad groups, creating keywords, monitoring ad performance, setting bids, etc.) via the user interface of a website or application hosted or provided by the computing system 102. As will be further described below, the user interface(s) supported by the computing system 102 and displayed on the client device 104 may include functions and / or tools such as an area of the user interface dedicated to chat interaction, which allows the user of the client device 104 to enter a request for assistance (e.g., a query or command) and view the response generated and sent by the computing system 102.
[0019] The computing system 102 includes a network interface 120, a processor 122, and a memory 124. The network interface 120 includes hardware, firmware, and / or software configured to enable the computing system 102 to exchange electronic data with the client device 104 (and optionally other similar client devices) and the server 106 (which may be plural) via the network 110. For example, the network interface 120 may include a wired or wireless router, and a modem. The processor 122 may be a single processor (e.g., a central processing unit (CPU)), or may include a plurality of processors (e.g., a plurality of CPUs, or one or more CPUs and one or more graphics processing units (GPUs)). The computing system 102 may be a single computing device (e.g., a server) located at a single location, or may include a plurality of cooperative computing devices (e.g., a plurality of servers) that are disposed at the same location or distributed remotely.
[0020] Memory 124 is a computer-readable, non-temporary storage medium, unit, or device, or a collection of such media / units / devices, which may include persistent and / or non-persistent memory components. Memory 124 stores executable instructions for performing various operations by the processor 122, which include instructions for various software applications, as well as data generated and / or used by such applications. In the exemplary system 100 of Figure 1, memory 124 stores user interface (UI) components 130, an orchestrator 132, several specialized AI agents 134, a model 136, and a tool 138. All of these elements may reside on a single server of the computing system 102, or they may be distributed among two or more servers (e.g., servers that are remote to each other). The operation of these components under various embodiments and / or scenarios will be described in more detail below with reference to Figures 2-8. However, generally, the orchestrator 132 is configured to centrally process user requests received from devices such as client devices 104 by selectively identifying and using / invoking specific AI agents of the specialized AI agents 134, each of which is specifically configured to analyze queries in different domains (e.g., trained / or fine-tuned and / or associated with different document databases). The tool 138 may include one or more specialized components, modules, models, etc., which can be invoked as needed (e.g., by the specialized AI agent 134 and / or directly by the planner component 140) to respond to specific requests / queries. Each tool 138 may expose, for example, its own application programming interface (API) for accessing that tool. Specific examples of the specialized AI agents 134 and tools 138 are described below with reference to, for example, Figure 3.
[0021] The UI component 130 communicates with client device applications (e.g., application 170 if running on client device 104) via the network 110 to generate user interfaces and receive information (including requests) entered by the user through those user interfaces. To process user requests entered through the user interfaces, the orchestrator 132 includes a planner component 140 and a response component 142, which respond to requests using “predict” and “create” approaches, respectively. The planner component 140 and the response component 142 may represent separate models, separate functions of a single model, or a model(s) and other associated components. The planner component 140 generates a response deployment plan for a given user request, at least in part, by determining which specialized AI agent 134 is relevant to the user request, and then queries the relevant(s) specialized AI agent 134 according to the response deployment plan. For this purpose, the planner component 140 may send synthetic queries to different specialized AI agents 134 in parallel and / or serially (for example, by using feedback from a first specialized AI agent to generate a synthetic query for a second specialized AI agent). In some embodiments and / or scenarios (for example, for a particular response deployment plan), the planner component 140 may engage in multi-round conversations with one or more of the specialized AI agents 134 (for example, as described below with respect to Figure 4), and / or one or more of the specialized AI agents 134 may be adapted to provide additional and / or different information to the orchestrator 132 as feedback (for example, as described below with respect to Figure 5).The response component 142 generates a response to the user based directly or indirectly on feedback from the queried one or more of the specialized AI agents 134 (e.g., after directly receiving the feedback, or after the planner component 140 or other components or functions of the orchestrator 132 summarize, integrate, filter, and / or otherwise process the feedback).
[0022] In some embodiments, the planner component 140 and / or the response component 142 include respective AI agents, and / or the orchestrator 132 includes an AI agent that communicates with the planner component 140 and / or the response component 142. For example, the top-level AI agent of the orchestrator 132 can analyze a user request and generate a synthetic query to the AI agent of the planner component 140 in response thereto, and the synthetic query causes the AI agent of the planner component 140 to create a response deployment plan and generate an appropriate synthetic query. In addition to (or as part of) the AI agent, the planner component 140 may include an execution engine that invokes / prompts the relevant specialized AI agent 134 using the synthetic query generated by the planner component 140.
[0023] Each AI agent in system 100 (e.g., the specialized AI agent 134, the AI agent of the planner component 140, the AI agent of the response component 142, the top-level AI agent of the orchestrator 132, etc.) may use one or more types of machine learning models included in model 136, such as a large-scale language model (LLM) or a multimodal LLM (MLLM), or, in some cases, for specific specialized AI agents 134, a classification model such as a neural network, support vector machine, clustering model, regression model, decision tree, etc. Some or all of model 136, and each of at least some of the models used by specialized AI agents 134, may be specifically trained (e.g., initially trained and / or fine-tuned) (e.g., using supervised learning techniques) to provide responses in those particular domains, and / or may be configured or arranged in a different way to provide responses in those particular domains (e.g., by providing access to a database of documents particularly relevant to a particular domain, in the case of the Search-Augmented-Generation (RAG) based AI agent described later). In some embodiments, the planner component 140 includes an AI agent that uses a model (of model 136) trained with supervised learning techniques, with manually created (or manually reviewed) queries / responses used as ground truth (for example, the queries / responses indicate that a particular skill / topic / specialization AI agent is relevant to a particular user request).
[0024] In some embodiments, one or more of the specialized AI agents 134 may utilize internal and / or external information sources when addressing synthetic queries from the planner component 140, using RAG or other appropriate techniques. In the exemplary system 100, the account database 180 stores information about the current state of an advertiser or other entity's account (e.g., bids, campaign entries / settings such as keywords, and / or creatives / ads associated with the advertiser's campaigns), and the knowledge corpus 182 may contain other information that may be useful to a particular of the specialized AI agents 134 (e.g., analytical data, policy guidelines, general business information). Although shown as two separate local data stores 180 and 182, it is understood that the account database 180 and / or the knowledge corpus 182 may be stored locally (e.g., in memory 124), remotely, and / or in a distributed manner. In some embodiments, each of one or more specialized AI agents 134 is configured to access only a subset of the knowledge corpus 182 (and / or only a subset of the account database 180), so that the AI agent can retrieve only documents that are more likely to be relevant to its domain (e.g., using RAG). This specialization of the available document database can reduce hallucinations and / or improve efficiency by separating the AI agents from at least some information that is likely irrelevant to their domain.
[0025] Where used herein, “AI agent” may, in some embodiments, include one or more AI sub-agents (and / or peer AI agents) having finer-grained functions, and “model” may, in some embodiments, include one or more sub-models (and / or peer models). For example, the optimization agent of the specialized AI agent 134 may include a first sub-agent specialized in hypothesizing optimization strategies and a second sub-agent specialized in collecting campaign and / or other data related to those optimization strategies.
[0026] In some embodiments, as shown in the exemplary system 100 of Figure 1, the orchestrator 132 maintains configuration data 146 (for example, stored in memory 124 or in memory of one or more remote locations). The configuration data 146 may include data (such as parameter settings) that a user (for example, an administrator using a client device that is part of the computing system 102) inputs to configure the orchestrator 132 for a specific use case or a specific product / service. For example, the configuration data 146 may indicate that a first subset of the specialized AI agent 134 and / or tools 138 is available for selection / use by the planner component 140 when a user of a client device (for example, client device 104) inputs a user request related to an advertising campaign management product, and that a second subset (which may overlap with the first subset) of the specialized AI agent 134 and / or tools 138 is available for selection / use by the planner component 140 when a user of the client device instead inputs a user request related to an advertising analytics product.
[0027] In some embodiments, as shown in the exemplary system 100 of Figure 1, the orchestrator 132 maintains user context data 150 and agent context data 152 (for example, stored in memory 124 or in memory of one or more remote locations). User context data 150 generally provides information that facilitates the planner component 140 (and possibly the response component 142) in generating a response deployment plan (and thus the final user response) that is more relevant and / or more appropriate to the context of the user request. For example, user context data 150 may include chat history across a single or multiple sessions of the user who entered the request (e.g., past user requests from client device 104 and corresponding responses from response component 142), and / or data associated with a particular entity (e.g., characteristics of advertisers or other entities associated with the user request, information associated with the business of such entities, information associated with the account of such entities, etc.). In some embodiments, user context data 150 may also include, or may instead include, information about the current state of the user interface presented to the user on client device 104, as described in more detail below, for example. As just one example, if user context data 150 indicates a chat history where a user on client device 104 has entered text indicating that the advertiser does not want to increase their budget (or, for example, if the user repeatedly ignores suggestions to increase the budget from response component 142), the planner component 140 may generate a synthetic query instructing one of the specialized AI agents 134 (e.g., the optimization agent) to avoid recommendations involving an increase in the advertiser's budget.As another example, if user context data 150 indicates a chat history in which a user on client device 104 entered text expressing dissatisfaction with frequent ad rejections, the planner component 140 may generate a response deployment plan that calls the policy agent of the specialized AI agent 134, even if the content of the corresponding user request does not trigger a call to the policy agent.
[0028] Agent context data 152 includes data maintained (e.g., created and updated) by the orchestrator 132 (e.g., planner component 140) for one, some, or all of the specialized AI agents 134. In particular, agent context data 152 may include history reflecting past communications with the specialized AI agent(s) (e.g., synthetic queries and corresponding responses or other feedback from the specialized AI agents), and the planner component 140 may analyze the relevant history as needed to facilitate the creation of a response deployment plan. For example, if the planner component 140 determines that a policy agent is related to a particular user request, and the context / history maintained by the planner component 140 indicates that the policy agent has recently and frequently returned synthetic query responses indicating that policy guidelines regarding sensitive content are relevant, the planner component 140 may decide to include a call to the sensitive content agent in the response deployment plan the next time it receives a user request related to some policy issue. In some embodiments, the planner component 140 maintains separate / isolated agent contexts / history for some or all different specialized AI agents 134, using the context of a particular AI agent when generating synthetic queries for that AI agent, but not when generating synthetic queries for other AI agents. Centralizing the maintenance of isolated agent-specific contexts / history effectively provides a much wider context window (or, equivalently, the specialized AI agents 134 themselves do not need to maintain a larger context), while simultaneously allowing each specialized AI agent 134 to focus more on its own domain without cross-context contamination (for example, in synthetic queries, there is no need to analyze / consider irrelevant data / history from other specialized AI agents 134), thus reducing the likelihood of hallucinations.
[0029] In some embodiments, the orchestrator 132 (e.g., the planner component 140) is configured (e.g., trained) to resolve conflicts between different specialized AI agents 134, thereby avoiding inefficiencies and further reducing hallucinations. As just one example, if the planner component 140 identifies conflicts in feedback from two specialized AI agents, the planner component 140 may ignore feedback (or relevant portions of feedback) from the AI agent that the planner component 140 has determined to be less relevant / important to the response.
[0030] The client device 104 may be any stationary, mobile, or portable computing device with wired and / or wireless communication capabilities (e.g., a smartphone, tablet computer, laptop computer, desktop computer, smart wearable device such as smart glasses or a smartwatch, or a vehicle head unit computer), or may include such devices. In the exemplary embodiment of Figure 1, the client device 104 includes a network interface 160, a processor 162, memory 164, and a display 166. The processor 162 may be a single processor or may include multiple processors. The memory 164 may include one or more computer-readable non-temporary storage media, units, or devices, which may include persistent memory components and / or non-persistent memory components. The memory 164 stores executable instructions for the processor 162 to perform various operations, including instructions for various software applications, as well as data generated and / or used by such applications.
[0031] In the exemplary system 100 of Figure 1, memory 164 stores at least application 170. For example, application 170 may be a web browser application used to access a website hosted by computing system 102, or a dedicated application downloaded from computing system 102 (e.g., a mobile application), or an application associated with an entity related to computing system 102. Generally, application 170 is executed by processor 162 to present one or more user interfaces via display 166, and the user interface(s) allow the user to input user requests analyzed by planner component 140, view responses to user requests generated by response component 142, and potentially facilitate various other tasks or actions (e.g., user modification of campaign settings, user monitoring of campaign performance, etc.). Exemplary user interfaces that may be supported by UI component 130 (and more generally by orchestrator 132) are described below in relation to Figures 7A-7D.
[0032] The display 166 includes hardware, firmware, and / or software configured to allow the user to view visual output from the client device 104, and may use any suitable display technology (e.g., LED, OLED, LCD, etc.). In some embodiments, the display 166 is incorporated into a touchscreen having both display and manual input functions. Although not shown in Figure 1, the client device 104 may also include one or more audio output devices or components, such as one or more speakers (for example, to present responses to user requests in audio format).
[0033] The network interface 160 includes hardware, firmware, and / or software configured to enable a client device 104 to exchange electronic data with a computing system 102 via the network 110. For example, the network interface 160 may include a cellular communication transceiver, a Wi-Fi transceiver, and / or transceivers for one or more other wired and / or wireless communication technologies.
[0034] Server(s) 106 may include any external (e.g., third-party) servers(s) with which the computing system 102 communicates to obtain data useful for generating responses to user requests. For example, in an embodiment in which the specialized AI agent 134 includes an analysis agent, the analysis agent may request relevant analysis data from the analysis server(s) 106 or invoke a tool (of the tools 138) to obtain relevant analysis data from the analysis server(s). In other embodiments, the system 100 omits Server(s) 106.
[0035] Figure 2 shows an exemplary architecture 200 that the computing system 102 of Figure 1 may implement to process user requests. In Figure 2, arrows indicate the direction of communication between layers of the architecture 200.
[0036] An exemplary architecture 200 includes a UI layer 210 implemented by a UI component 130, on top of an orchestrator layer 220 implemented by an orchestrator 132. The orchestrator layer 220 includes a planner layer 222 implemented by a planner component 140, on top of a response layer 224 implemented by a response component 142. Alternatively, the planner layer 222 and the response layer 224 may reside at the same level in the stack of architecture 200. The orchestrator layer 220 resides on top of a specialized AI agent layer 230 implemented by a specialized AI agent 134, and the specialized AI agent layer 230 resides on top of a tool layer 240 implemented by a tool 138. Alternatively, the specialized AI agent layer 230 and the tool layer 240 may reside at the same level in the stack of architecture 200 (or the specialized AI agent layer 230 may be partially on top of the tool layer 240, or partially at the same level as the tool layer 240, etc.).
[0037] In the exemplary architecture, the UI layer 210 communicates user requests (and potentially other information such as screenshots of the UI or other current state) to the planner layer 222, which then generates a response deployment plan and queries the relevant(s) specialized AI agents 134 according to that plan. As indicated by the bidirectional arrows on the far left of Figure 2, the specialized AI agent layer 230 (i.e., the relevant(s) specialized AI agents 134) may provide feedback (e.g., responses) to the planner layer 222, potentially involving multiple loops / rounds of synthetic queries and responses. The planner layer 222 may then provide feedback (or synthetic queries generated by the planner component 140 based on the feedback) to the response layer 224. Alternatively or additionally (as indicated by the dashed lines in Figure 2), the specialized AI agent layer 230 may provide feedback directly to the response layer 224. Similarly, the specialized AI agent layer 230 may invoke one or more of the tools 138 by communicating with the tool layer 240, possibly involving multiple loops / rounds of tool calls and responses.
[0038] After the orchestrator layer 220 (planner layer 222 or response layer 224) has received all responses from the specialized AI agent layer 230 that are expected in the response deployment plan (perhaps excluding responses that timed out due to unresponsive AI agents), the orchestrator 132 or planner component 140 may generate a synthetic query to be communicated to the response layer 224, the response component 142 then generates a response to the user request and communicates that response to the UI layer 210. The UI layer 210 can then have the user interface of the application 170 on the client device 104 present the response to the user via the display 166 (and / or as an audio output, etc.).
[0039] It is understood that other embodiments may support more or fewer interlayer communications than shown in Figure 2, and / or more or fewer directional interlayer communications than shown in Figure 2. Furthermore, in some embodiments, the architecture 200 may include one or more additional layers (e.g., a model layer implemented by model 136 between layers 230 and 240), and / or omit one or more layers (e.g., a tool layer 240).
[0040] Figure 3 shows an exemplary set 300 of specialized AI agents 302 and tools 304 that may be used within the computing system 102 of Figure 1 and / or the architecture 200 of Figure 2. For example, the specialized AI agents 302 and tools 304 may be the specialized AI agents 134 and tools 138 of Figure 1, respectively.
[0041] In the exemplary set 300 shown in Figure 3, the specialized AI agent 302 includes a reporting agent, a billing agent, an optimization agent, a diagnostic agent, a policy agent, a sensitive content agent, a creative agent, a search engine agent, and a web crawl agent. For example, the illustrated specialized AI agent 302 may be specifically configured to perform the following actions (e.g., being trained, fine-tuned, and provided with access to a specific database for document retrieval): Reporting Agent: In response to a synthetic query from Planner Component 140, (1) locally plans the data / information to be collected (e.g., advertising performance / analysis data from Knowledge Corpus 182), (2) collects the desired data / information using one or more of the tools 304, (3) infers based on the collected data / information and synthetic query using the LLM or MLLM of Model 136, and / or (4) uses the Reporting Agent's response layer model to create a response to Planner Component 140 (e.g., a concise presentation, summary, etc. of the relevant parts of the collected information), and / or provides other feedback (e.g., grounding data) as described below. Billing Agent: In response to a synthetic query from Planner Component 140, (1) locally plans the data / information to be collected (e.g., historical budget / cost data from Knowledge Corpus 182), (2) collects the desired data / information using one or more of the tools 304, (3) infers based on the collected data / information and synthetic query using the LLM or MLLM of Model 136, and / or (4) uses the Billing Agent's response layer model to create a response to Planner Component 140 (e.g., a concise presentation, summary, etc. of the relevant parts of the collected information), and / or provides other feedback (e.g., grounding data) as described below. Optimization Agent: In response to a synthetic query from Planner Component 140, (1) locally plans the data / information to be collected (e.g., campaign configuration information, ad performance / analysis data in Account Database 180), (2) collects the desired data / information using one or more of the Tools 304, (3) infers based on the collected data / information and synthetic query using the LLM or MLLM of Model 136, and / or (4) uses the Optimization Agent's response layer model to create a response to Planner Component 140 (e.g., suggested methods / methods to improve past performance indicated by the collected information), and / or provides other feedback (e.g., grounding data) as described below. Diagnostic Agent: In response to a synthetic query from Planner Component 140, (1) locally plans the data / information to be collected (e.g., notifications of errors / problems for accounts in Account Database 180), (2) collects the desired data / information using one or more of the Tools 304, (3) infers based on the collected data / information and synthetic queries using the LLM or MLLM of Model 136, and / or (4) uses the Diagnostic Agent's response layer model to create a response to Planner Component 140 (e.g., suggested methods / methods for correcting errors / problems indicated by the collected information), and / or provides other feedback (e.g., grounding data) as described below. Policy Agent: In response to a synthetic query from Planner Component 140, (1) locally plan the data / information to be collected (e.g., policy rule notifications from Knowledge Corpus 182, policy-based rejections of ads in advertiser campaigns as indicated in Account Database 180 or Knowledge Corpus 182), (2) collect the desired data / information using one or more of Tools 304, (3) infer based on the collected data / information and synthetic queries using Model 136's LLM or MLLM, and / or (4) create a response to Planner Component 140 (e.g., a brief explanation of why the ad was disapproved) and / or provide other feedback (e.g., grounding data) as described below. Confidential Content Agent: In response to a synthetic query from Planner Component 140, it (1) locally plans the data / information to be collected (e.g., notifications of specific policy rules regarding confidential content from Knowledge Corpus 182, ad rejection based on relevant policies in an advertiser's campaign as indicated in Account Database 180 or Knowledge Corpus 182), (2) collects the desired data / information using one or more of Tools 304, (3) infers based on the collected data / information and synthetic queries using Model 136's LLM or MLLM, and / or (4) uses the Confidential Content Agent's response layer model to produce a response to Planner Component 140 (e.g., a brief explanation of why the ad was rejected due to confidential content, or why a question or instruction in a user request might cause a potential confidential content issue), and / or provides other feedback (e.g., grounding data) as described below. The Confidential Content Agent may be similar to the Policy Agent, but may be more highly specialized, for example. Creative Agent: In response to a synthetic query from Planner Component 140, (1) locally plan the data / information to collect (e.g., a specific creative for an advertiser in Account Database 180), (2) collect the desired data / information using one or more of Tools 304, (3) infer based on the collected data / information and synthetic query using Model 136's LLM or MLLM, and / or (4) use the Creative Agent's response layer model to create a response to Planner Component 140 (e.g., a brief explanation of why the ad was disapproved), and / or provide other feedback (e.g., grounding data) as described below. Search Engine Agent: In response to a synthetic query from Planner Component 140, (1) creates one or more search queries, (2) executes one or more queries in the search engine (e.g., directly or using either Tool 138), (3) infers based on the search results and synthetic queries using the LLM or MLLM of Model 136, and / or (4) uses the Search Engine Agent's response layer model to create a response to Planner Component 140 (or provide the top N search results directly, etc.), and / or provides other feedback (e.g., grounding data) as described below. Web Crawl Agent: In response to a synthetic query from Planner Component 140, (1) creates web crawl parameters (e.g., a list of URLs to crawl, and / or keywords indicating the type of information desired), (2) crawls the web based on the web crawl parameters (e.g., directly or using one of the tools 138), (3) infers based on the web crawl results and synthetic queries using the LLM or MLLM of Model 136, and / or (4) creates a response to Planner Component 140 using the response layer model of the web crawl agent (or provides the results from the web crawl directly, etc.), and / or provides other feedback (e.g., grounding data) as described below.
[0042] As described above, in some embodiments and / or scenarios, the planner component 140 may create a response deployment plan in which the planner component 140 queries all relevant specialized AI agents 134 in parallel, while in other embodiments and / or scenarios, the planner component 140 may specify a serial order or other more complex order (e.g., fan out parallel synthetic queries to the reporting agent and the diagnostic agent, and then query the optimization agent with additional synthetic queries based on the responses from the reporting agent and the diagnostic agent). Furthermore, in some embodiments and / or scenarios, the first specialized AI agent 134 may invoke a subagent, which may be, for example, a different second specialized AI agent 134. For example, the diagnostic agent may invoke a policy agent, and / or the policy agent may invoke a sensitive content agent.
[0043] Furthermore, in the exemplary set 300 shown in Figure 3, tool 304 includes a campaign information tool, a notification tool, and a website information tool. For example, tool 304 shown may be configured to perform the following actions when called (e.g., using its respective API): Campaign Information Tool: Retrieves information about a specific campaign (e.g., from account database 180), including campaign entries / settings (keywords, bids, etc.) and costs. Notification tool: Retrieves information about current notifications associated with a specific account and / or campaign, such as ad approval change notifications and budget notifications (e.g., from account database 180). Website information tools: These tools obtain information about a specific website (e.g., a URL or a set of URLs) by scraping information (such as product descriptions) from landing pages associated with a particular advertisement. Screen capture tool: Captures screenshots of the user interface and / or other user interface state information, indicating which screen or screen portion is currently being viewed (for example, by a user on client device 104).
[0044] One or more of the specialized AI agents 302 (e.g., the planning layer / model of the AI agent) may call / use one or more of the tools 304 as appropriate in response to a specific query. In some embodiments, the planner component 140 itself may also, or instead, directly call / use some or all of the tools 304. It is understood that the specialized AI agents 302 may include more, fewer, and / or different AI agents than those shown in Figure 3, and the tools 304 may include more, fewer, and / or different tools than those shown in Figure 3.
[0045] Figure 4 is a messaging diagram representing a sequence of messages 400 that may be exchanged within the computing system 102 of Figure 1 and / or the architecture 200 of Figure 2 in exemplary scenarios and embodiments. In Figure 4, the orchestrator 402 may represent the orchestrator 132 or the orchestrator layer 220, and the specialized AI agent 404 may represent, for example, one of the specialized AI agents 134 or 302 or the specialized AI agent layer 230.
[0046] In the exemplary embodiments and scenarios of Figure 4, the orchestrator 402 (e.g., the planner component 140) generates a first synthetic query (based on a user request) and sends it to the specialized AI agent 404 (410). The first synthetic query may be designed to directly extract a desired final response from the specialized AI agent 404 (e.g., if the response deployment plan assumes that a usable response will be obtained in a minimum of one conversation turn / round), or it may be designed to collect information useful for generating a subsequent second synthetic query to the same specialized AI agent 404 (e.g., if the response deployment plan assumes that multiple conversation turns / rounds will be required). In either case, in the scenario shown in Figure 1, the orchestrator 402 (e.g., the planner component 140) determines that an additional query is needed after receiving the first response from the specialized AI agent 404 (412), and therefore generates a second synthetic query. The first response may be received (412) by the planner component 140 or the response component 142, depending on the embodiment. For example, the planner component 140 may receive a first response and determine that the first response lacks some necessary information, or the response deployment plan may instruct a follow-up query regardless of the content of the first response.
[0047] Therefore, the orchestrator 402 (e.g., the planner component 140) generates a second synthetic query and sends it to the specialized AI agent 404 (414), and receives a corresponding second response from the specialized AI agent 404 (e.g., by the planner component 140 or the response component 142) (416). Depending on the scenario (or embodiment, e.g., if the orchestrator 132 allows up to two communication turns / rounds with the specialized AI agent 404 to suppress latency), the orchestrator 402 may or may not make one or more additional communication turns / rounds with the specialized AI agent 404. In some embodiments, prompts to the planner component 140 (e.g., generated by an AI agent in a higher layer of the orchestrator 132) instruct the planner component 140 to limit communication to a certain number of turns / rounds or to refrain from generating and / or sending other queries when the planner component 140 determines that the most recent response has a certain level of usefulness.
[0048] As shown in Figure 4, the specialized AI agent 404 may include its own response model 406 and one or more lower-level models 408. For example, the response model 406 may be a relatively small / low-capacity LLM, and the lower-level models may be one or more large, more powerful LLMs. The specialized AI agent 404 may use the lower-level models 408 to perform intermediate inference, while using the response model 406 to generate a "final" response based on that inference. For example, the lower-level models 408 may analyze a first synthetic query to generate a first intermediate inference, and the response model 406 may generate a first response based on the first intermediate inference. Similarly, the lower-level models 408 may analyze a second synthetic query to generate a second intermediate inference, and the response model 406 may generate a second response based on the second intermediate inference. The underlying model(s) 408 may generate intermediate inference based on grounding data that includes information from synthetic queries from the orchestrator 402 and / or other information (e.g., one or more documents retrieved by the specialized AI agent 404 using the RAG architecture).
[0049] Figure 5 is a messaging diagram representing other sequences 500 of messages that can be exchanged within the computing system 102 of Figure 1 and / or the architecture 200 of Figure 2 in an alternative embodiment that can further reduce latency and / or hallucinations. In Figure 5, the orchestrator 502 may represent the orchestrator 132 or the orchestrator layer 220, and the specialized AI agent 504 may represent, for example, one of the specialized AI agents 134 or 302 or the specialized AI agent layer 230. Furthermore, the agent-side response model 506 and lower layer model(s) 508 may be the same as the response model 406 and lower layer model(s) 408 in Figure 4.
[0050] In the exemplary embodiments and scenarios of Figure 5, the orchestrator 502 (e.g., the planner component 140) generates a first synthetic query (based on user requests) and sends it to the specialized AI agent 504 (510). However, in the embodiments of Figure 5, the specialized AI agent 504 is adapted to send (512) to the orchestrator 502 (e.g., the response component 142) the intermediate inference generated by the lower-level model(s) 508 based on the synthetic query, and the grounding data on which the lower-level model(s) 508 based the intermediate inference. In some embodiments, the specialized AI agent 504 also sends (512) instructions on how to read / interpret the grounding data, which may be in a format unknown to the orchestrator 502.
[0051] In the embodiment shown in Figure 5, the specialized AI agent 504 does not send any response generated by the response model 506 based on intermediate inference. Thus, the creation of the higher-layer response is effectively delegated to the orchestrator 502 (e.g., response component 142), which generates the response based on intermediate inference and grounding data (and possibly instructions on how to read the grounding data), but not on any response from the response model 506. In some embodiments, the specialized AI agent 504 bypasses the response model 506 because the higher-layer response on the agent side is not required. In other embodiments, the specialized AI agent 504 completely omits the response model 506. In yet another embodiment, the specialized AI agent 504 sends a higher-layer response (512) along with grounding data and intermediate inference (and possibly instructions), but the orchestrator 502 (e.g., response component 142) ignores the higher-layer response for the purpose of creating a response to the user request. In any of these embodiments, the orchestrator 132 may be provided with sufficient background information to eliminate the need to query the specialized AI agent 504 again (in at least some cases), as shown in Figure 4. By eliminating loops / rounds (or multiple loops / rounds) in the communication between the orchestrator 502 and the specialized AI agent 504, such embodiments can reduce the total latency of the response provided to the user and simultaneously reduce the usage of processing power. Furthermore, (given the more advanced domain expertise of the specialized AI agent 504) counterintuitively, hallucinations can be reduced by removing (or ignoring, for example) the layer in which the specialized AI agent 504 generates its response to the synthetic query from the orchestrator 502, because the conclusions or inferences derived by the response model of the specialized AI agent (e.g., response model 506) can be a significant source of hallucinations.
[0052] In some embodiments, all of the specialized AI agents 134 or 302 delegate the creation of higher-level responses to the orchestrator 502 (e.g., response component 142) using the technique of Figure 5. In other embodiments, only a subset (one or more) of the specialized AI agents 134 or 302 use the technique of Figure 5, while others use other techniques (e.g., the technique of Figure 4). Furthermore, in some embodiments, the delegation of higher-level responses is determined on a case-by-case basis (e.g., by the planner component 140, which may indicate where delegation is required in the response development plan and / or the synthetic queries to the specialized AI agents, or by the specialized AI agents themselves).
[0053] Figure 6 shows an exemplary process 600 that may be implemented by one or more specialized AI agents (e.g., specialized AI agents 134 or 302). Figure 6 corresponds to an embodiment in which a specialized AI agent uses an LLM (e.g., Model 136) specific to that AI agent (i.e., specifically trained to process queries in a particular domain and / or otherwise configured).
[0054] In Figure 6, the synthetic query 602 (e.g., generated by the planner component 140) is received as input by the specialized AI agent, which in response constructs an LLM request (e.g., a prompt) in stage 604. In stage 606, the specialized AI agent uses LLM to generate a response to the LLM request, and in stage 608, the specialized AI agent determines whether a tool call exists in the response and / or whether at least a threshold number of LLM calls have already been made for the synthetic query 602. If a tool call exists and the threshold has not yet been reached, the flow proceeds to stage 610. In stage 610, the specialized AI agent uses a tool handler to invoke the appropriate tool 138. If no tool call exists and / or the threshold has been reached, the flow proceeds to stage 612, in which case the specialized AI agent responds to the synthetic query using the information provided by the tool(s). For example, stage 612 may include sending the upper-level response as shown in Figure 4, or sending the underlying grounding data and intermediate inference as shown in Figure 5.
[0055] Figures 7A to 7D illustrate various user interfaces that the UI component 130 of Figure 1 may present to a user (for example, a user of client device 104 via application 177 and display 166) in various exemplary scenarios and embodiments.
[0056] Referring first to Figure 7A, the user interface 700 includes a work area 702 and a chat area 704. The work area 702 can support services or functions of a digital advertising platform provider, and the chat area 704 allows the user to enter requests and view responses from the orchestrator 132. Specifically, the chat area 704 includes a user request field 710 where the user can enter their requests and a response area 712 that displays the responses. In some embodiments, the chat area 704 also provides additional user request fields to allow the user to continue the conversation with the orchestrator 132.
[0057] In the example in Figure 7A, the user asks, "Which of my ads is performing the best, and how can I further improve my ads?" and the orchestrator 132 responds (at least partially) as shown. Specifically, the response provides the identifier of the advertiser's best-performing ad and its performance metrics (click-through rate and clicks). The response in this example also provides at least one suggestion for improving the advertiser's ad campaign, which, when activated by the user, prompts the user to provide additional details (for example, by activating a deep link to a screen / page of a product or service associated with the work area 702) via interactive control 714.
[0058] As an example, referring to Figures 1 and 3, the response in response area 712 may be generated as follows: 1) Based on the user request in field 710, the planner component 140 generates a response deployment plan specifying that the reporting agent and the optimization agent be invoked in parallel. 2) The planner component 140 generates and sends a composite query corresponding to the reporting agent and the optimization agent, respectively. 3) Based on that query, the reporting agent invokes the campaign information tool to obtain performance metrics for all of the advertiser's ads. 4) Based on the performance metrics, the reporting agent identifies the highest-performing ad and provides feedback to the orchestrator 132 that includes at least the ad identifier and the corresponding performance metrics. 5) In parallel with the reporting agent's operation and based on its own query, the optimization agent invokes the campaign information tool to obtain the advertiser's campaign settings and performance metrics. 6) Based on the campaign settings and performance metrics, the optimization agent creates one or more optimization strategies and provides feedback to the orchestrator 132 indicating those optimization strategies. 7) The response component 142 generates the text shown in the response area 712 based on feedback from the policy agent and the optimization agent. In some embodiments, the reporting agent and / or the optimization agent provide grounding data and intermediate inference, as described above in relation to Figure 5.
[0059] Referring to Figure 7B, the user interface 720 includes a work area 722 and a chat area 724. The work area 722 can support services or functions of the digital advertising platform provider, and the chat area 724 allows the user to enter requests and view responses from the orchestrator 132. For example, the chat area 704 and chat area 724 in Figure 7A (and / or the chat areas in Figures 7C and / or 7D) may be the same, i.e., they may be persistent chat areas that remain in their position even when the user moves to a different screen. The chat area 724 includes a user request field 730 where the user can enter their request and a response area 732 that displays the response.
[0060] In the example in Figure 7B, the user instructs, "Please correct this disapproved ad's policy violation," and the orchestrator 132 responds (at least partially) as shown. Specifically, the response indicates the specific reason why the ad displayed in the work area 722 was not approved. The response in this example also provides an interactive control 734, which, when activated by the user, causes the system to compel the ad to comply with the policy guidelines, or at least attempt to compel the ad to comply with the policy guidelines.
[0061] As an example, referring to Figures 1 and 3, the response in response area 732 may be generated as follows: 1) Based on the user request in field 730, the planner component 140 generates a response deployment plan specifying that the policy agent will be called. 2) The policy agent or planner component 140 calls a screen capture tool to take a screenshot of the user interface as seen by the user on client device 104. 3) The planner component 140 generates and sends a synthetic query to the policy agent. 4) Based on that query and the screenshot (or information derived from the screenshot by the planner component 140), the policy agent calls a notification tool to obtain the reason why the advertisement currently displayed in work area 722 was not approved. 5) Based on the reason for disapproval, the policy agent generates an explanation of the reason for disapproval and identifies corrective actions to address the reason. 6) Based on the feedback from the policy agent, the response component 142 creates the text shown in response area 732. User-interactive control 734 is activated by the user, causing the orchestrator 132 (or specialized AI agent 134 or tool 138, etc.) to communicate with other systems or platforms (e.g., via APIs), thereby executing specific changes described in response area 732 (possibly after the user provides further confirmation). In some embodiments, the policy agent provides grounding data and intermediate inference, as described above in relation to Figure 5.
[0062] Referring to Figure 7C, the user interface 740 includes a work area 742 and a chat area 744. The work area 742 can support services or functions of the digital advertising platform provider, and the chat area 744 allows the user to enter requests and view responses from the orchestrator 132. The chat area 744 includes a user request field 750 where the user can enter their requests and a response area 752 which displays the responses.
[0063] In the example in Figure 7C, the user asks, "Which campaigns require my attention?", and the orchestrator 132 responds (at least partially) as shown. Specifically, the response indicates the specific reasons why a particular advertiser campaign may need improvement, along with possible ways to achieve at least some of those improvements. The response in this example also provides an interactive control 754, which, when activated by the user, causes the system to compel the ads to comply with policy guidelines, or at least attempt to compel the ads to comply with policy guidelines.
[0064] As an example, referring to Figures 1 and 3, the response in response area 752 may be generated as follows: 1) Based on the user request in field 750, planner component 140 generates a response deployment plan specifying that it sequentially invokes the reporting agent and the optimization agent. 2) Planner component 140 generates a synthetic query and sends it to the reporting agent. 3) Based on that query, the reporting agent invokes the campaign information tool to obtain performance metrics for all of the advertiser's campaigns (and / or specific ads within them). 4) Based on the performance metrics, the reporting agent generates a description of the advertiser's relatively underperforming campaigns. 5) Based on the description from the reporting agent, planner component 140 generates a synthetic query and sends it to the optimization agent. 6) Based on that query, the optimization agent identifies potential areas for improvement for one or more of the underperforming campaigns. 7) Based on the feedback from the reporting agent and the optimization agent, response component 142 creates the text shown in response area 752. User interaction control 754 may, for example, cause the planner component 140 to generate other synthetic queries to the optimization agent in order to obtain a more detailed optimization plan. Alternatively, user interaction control 754 may trigger a deep link to a screen / page of a product or service associated with the work area 742. In some embodiments, the reporting agent and / or optimization agent provide grounding data and intermediate inference, as described above in relation to Figure 5.
[0065] Referring to Figure 7D, the user interface 760 includes a work area 762 and a chat area 764. The work area 762 can support services or functions of the digital advertising platform provider, and the chat area 764 allows the user to enter requests and view responses from the orchestrator 132. The chat area 764 includes a user request field 770 where the user can enter their requests and a response area 772 which displays the responses.
[0066] In the example shown in Figure 7D, the user asks, "Are there any other sellers of this service near me?", and the orchestrator 132 responds (at least partially) as shown. Specifically, the response lists other companies geographically close to the advertiser's site using one or more web sources.
[0067] As an example, referring to Figures 1 and 3, the response in response area 772 may be generated as follows: 1) Based on the user request in field 770, the planner component 140 generates a response deployment plan specifying that the search engine agent is invoked after the campaign information tool (or other appropriate tool) is invoked. 2) The planner component 140 uses the campaign information tool (or other appropriate tool) to obtain the advertiser's physical address or other location. 3) The planner component 140 generates a synthetic query including the location and sends it to the search engine agent. 4) Based on that query, the search engine agent invokes the search engine's API (e.g., via tool 138 or 304) using other queries indicating the location and receives search results. 5) The search engine agent provides feedback indicating at least a subset of the relevant search results. 6) Based on the feedback from the search engine agent, the response component 142 creates the text shown in response area 772. In some embodiments, the search engine agent provides grounding data and intermediate inference, as described above in relation to Figure 5.
[0068] As described above, in some embodiments, the orchestrator 132 (e.g., the planner component 140) maintains one or more contexts, as reflected, for example, by the user context data 150 and / or agent context data 152 in Figure 1. As an example of user context data (other than the screenshot example in Figure 7B), in the scenario and embodiment of Figure 7A, the planner component 140 may generate a response deployment plan based not only on the user request in field 710 but also on chat history indicating that the user had previously shown interest in performance over the past 30 days (e.g., the planner component 140 generates a synthetic query to the reporting agent that specifically requests the performance of the highest-performing ads over the past 30 days). As an example of agent context data, in the scenario and embodiment of Figure 7B, the planner component 140 may generate a response deployment plan based not only on the user request in field 730 but also on query / response history between the planner component 140 and the policy agent indicating specific ads that the user is likely to be referring to (e.g., in embodiments where UI context / screen capture technology is not used).
[0069] Furthermore, as described above, in some embodiments, the response component 142 responds to the initial user request by performing or triggering an action, at least partially. For example, in the scenario and embodiment of Figure 7B, the response component 142 may include interactive control in the response area 752. User activation of the interactive control may then cause the computing system 102 (or other device / system) to remotely control the input functions of the client device 104, thereby automatically selecting various controls in the work area 742 and inputting information into various fields.
[0070] In some embodiments, the orchestrator 132 may output text indicating its inference stage according to the response deployment plan, and the computing system 102 may send it to the client device 104 for real-time presentation to the user via the application 170 and display 166. In this way, the planning steps may be more transparent to the user, and the user may be able to remain engaged in the process during delays caused by processing latency. Referring to Figure 7A, for example, after the user enters a query in the user request field 710 (during which time the reporting agent invokes the campaign information tool), the user may first be presented with the text "Collecting performance metrics for all ads," which may then be replaced with the text "Searching for the highest performing ads" (during which time the reporting agent identifies the highest performing ads), and so on.
[0071] Figure 8 is a flowchart of an exemplary method 800 for accurate, low-latency user request processing. Method 800 can be implemented, for example, by the computing system 102 of Figure 1 (e.g., an orchestrator 132 run by a processor 122, and possibly a specialized AI agent 134 and / or other components).
[0072] In block 802, a user request is received from a client device (e.g., client device 104) via the user interface (e.g., by the planner component 140). In block 804, the orchestrator (e.g., orchestrator 132) generates a context-specific response to the user request. Block 804 includes blocks 806, 808, 810, and 812.
[0073] In block 806, the orchestrator (e.g., the planner component 140 of orchestrator 132) generates a response deployment plan based on the user request and one or more context signals (e.g., from user context data 150 and / or agent context data 152). The response deployment plan designates one or more specialized AI agents (e.g., specialized AI agents 134 or 302) by determining, at least in part, that one or more specialized AI agents are relevant to the user request. Each specialized AI agent is configured to analyze queries specific to its respective domain.
[0074] In block 808, the orchestrator queries one or more specialized AI agents according to the response deployment plan. In block 810, it receives feedback from one or more specialized AI agents in response to the query execution. The feedback may be a response from a higher layer (e.g., similar to Figure 4) or it may include intermediate inference and grounding data (e.g., similar to Figure 5).
[0075] In block 812, the orchestrator (for example, response component 142 of orchestrator 132) generates a context-specific response based on each feedback received in block 810.
[0076] In block 814, context-specific responses are made to be presented to the client device via the user interface.
[0077] In other embodiments, method 800 may include more or fewer blocks, and / or certain blocks may occur in a different order than shown in Figure 8. For example, block 812 may include identifying an action to be performed in response to a user request, and method 800 may include additional blocks that cause the action to occur (e.g., causing one or more actions via the user interface by remotely acquiring control of the user interface on a client device).
[0078] As is evident from the above description, some of the technologies disclosed herein use artificial intelligence to generate high-quality videos for specific products and audiences. Artificial intelligence (AI) is a segment of computer science that focuses on creating models that can perform tasks with little or no human intervention. Artificial intelligence systems can utilize, for example, machine learning, natural language processing, and computer vision. Its subsets, such as machine learning and deep learning, focus on developing models that can infer outputs from data. Outputs can include, for example, prediction and / or classification. Natural language processing focuses on analyzing and generating human language. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content, such as images, videos, text, audio, and / or other content, in response to input prompts and / or based on other information.
[0079] Exemplary machine learning models include neural networks or other multi-layered nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine learning models can leverage attention mechanisms such as self-attention. For example, some machine learning models can include multi-head self-attention models (e.g., transformer models).
[0080] Models can be trained using a variety of training or learning techniques. Training can include supervised learning, unsupervised learning, reinforcement learning, etc. Training can use techniques such as backpropagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Parameters can be iteratively updated over several training iterations using gradient descent. Several generalization techniques (e.g., weight decay, dropout) can be used to improve the generalization ability of the trained model.
[0081] Prior to domain-specific alignment, the model(s) may be pre-trained. For example, a model may be pre-trained on a general corpus of training data and then fine-tuned to a more targeted corpus of training data. The model may be aligned using prompts designed to elicit domain-specific outputs. The prompts may be designed to include learned prompt values (e.g., soft prompts). The trained model(s) may be validated before use using input data other than the training data, and may be further updated or improved during use based on additional feedback / input.
[0082] In some embodiments, the computing system 102 may use one or more of the machine learning models or techniques described above to perform any one or more of the operations discussed herein in relation to machine learning. For example, one of the models 136 may use one or more such machine learning techniques to help produce a response (and / or intermediate inference, etc.).
[0083] The following list of embodiments reflects the various embodiments expressly intended by this disclosure. [Examples]
[0084] Example 1. A computer-implemented method for accurate and low-latency user request processing, the computer-implemented method comprising: receiving a user request from a client device via a user interface; generating a response deployment plan by an orchestrator communicably coupled to a plurality of specialized artificial intelligence (AI) agents, at least in part, specifying one or more specialized AI agents from the plurality of specialized AI agents based on (i) the user request and (ii) one or more context signals, wherein generating the response deployment plan includes determining that the one or more specialized AI agents are related to the user request, and each of the plurality of specialized AI agents is configured to analyze queries specific to their respective domains; generating a response deployment plan; querying the one or more specialized AI agents in accordance with the response deployment plan; receiving feedback from the one or more specialized AI agents in response to the queries; generating a context-specific response to the user request based on the feedback from the one or more specialized AI agents; and presenting the context-specific response to the client device via the user interface.
[0085] Example 2. The one or more specialized AI agents include a first specialized AI agent and a second specialized AI agent, and the method implemented on the computer further includes the orchestrator maintaining (i) a first agent context based on past communications between the orchestrator and the first specialized AI agent, and (ii) a second agent context based on past communications between the orchestrator and the second specialized AI agent, wherein the orchestrator maintains the first agent context separately from the second agent context, and the method implemented on the computer further includes querying the one or more specialized AI agents, wherein querying the one or more specialized AI agents includes (i) generating a first synthetic query for the first specialized AI agent, partially based on the first agent context, and (ii) generating a second synthetic query for the second specialized AI agent, partially based on the second agent context, the method implemented on the computer according to Example 1.
[0086] Example 3. A computer-implemented method according to Example 1 or 2, wherein querying one or more specialized AI agents comprises sending a first synthetic query to the first specialized AI agent, receiving the respective feedback comprises receiving from the first specialized AI agent (i) intermediate inference supporting the response to the first synthetic query, and (ii) grounding data used by the first specialized AI agent to generate the intermediate inference, and generating the context-specific response comprises generating the context-specific response based on the grounding data and the intermediate inference.
[0087] Example 4. A method implemented on a computer as described in Example 3, wherein the first specialized AI agent includes a first AI agent response model configured to generate responses to synthetic queries based on intermediate inference, and generating the context-specific responses includes not using any responses generated by the first AI agent response model to generate the context-specific responses.
[0088] Example 5. A method implemented in the computer according to Example 3 or 4, wherein receiving each of the feedbacks includes receiving instructions from the first specialized AI agent regarding how to read the grounding data.
[0089] Example 6. A computer-implemented method according to any one of Examples 1 to 5, wherein generating the response deployment plan and generating the context-specific response are performed by a single generative AI model.
[0090] Example 7. A method implemented on a computer according to any one of Examples 1 to 5, wherein the orchestrator includes a planner model for generating the response deployment plan and a response model for generating the context-specific response, the planner model includes a first AI agent and the response model includes a second AI agent.
[0091] Example 8. A method implemented in a computer according to Example 7, wherein the orchestrator includes a third AI agent that receives the user request, generates a prompt based on the user request, and applies the prompt to the first AI agent to cause the first AI agent to generate the response deployment plan.
[0092] Example 9. A computer implementation method according to any one of Examples 1 to 8, wherein the one or more context signals include one or more characteristics of an entity and one or more information associated with the entity's business or information associated with the entity's account.
[0093] Example 10. A computer implementation method according to any one of Examples 1 to 9, wherein the one or more context signals include data indicating the state of the user interface.
[0094] Example 11. A computer-implemented method according to any one of Examples 1 to 10, wherein generating the context-specific response includes identifying an action to be performed in response to the user request, and the computer-implemented method further includes causing the action.
[0095] Example 12. A method implemented in a computer according to Example 11, wherein identifying the action includes determining one or more desired sets of actions via the user interface, and causing the action to occur includes causing the client device to perform the one or more desired sets of actions via the user interface by remotely acquiring control of the input functions of the client device.
[0096] Example 13. A method implemented in a computer according to any one of Examples 1 to 10, wherein the one or more specialized AI agents include a first specialized AI agent and a second specialized AI agent, and generating the context-specific response includes resolving a conflict between (i) a first feedback provided by the first specialized AI agent in response to the query execution and (ii) a second feedback provided by the second specialized AI agent in response to the query execution.
[0097] Example 14. A computer-implemented method according to any one of Examples 1 to 13, wherein the response deployment plan further specifies the order in which the planner model queries the one or more specialized AI agents.
[0098] Example 15. The method implemented on a computer as described in Example 14, wherein the sequence indicates that the planner model queries at least two specialized AI agents in parallel.
[0099] Example 16. The computer-implemented method described in Example 15, wherein the sequence described above is used to generate a synthetic query for querying a second specialized AI agent, using the respective feedback from the first specialized AI agent.
[0100] Example 17. A computer-implemented method according to any one of Examples 1 to 16, further comprising the planner model maintaining historical data based on past interactions associated with (i) the user of the client device, or (ii) entities associated with the user or the client device, wherein either or both of (i) generating the response deployment plan and (ii) querying one or more specialized AI agents are partially based on the historical data.
[0101] Example 18. The method implemented in the computer according to Example 17, wherein the history data includes chat history displayed at least partially on the user interface.
[0102] Example 19. A computer-implemented method according to any one of Examples 1 to 18, further comprising generating the respective feedbacks by one or more specialized AI agents.
[0103] Example 20. A method implemented on a computer according to Example 19, wherein generating each of the aforementioned feedbacks includes a first specialized AI agent among the one or more specialized AI agents querying the other AI agents.
[0104] Example 21. A method implemented on a computer as described in Example 19, wherein generating each of the aforementioned feedbacks involves a first specialized AI agent among the one or more specialized AI agents calling a tool configured to provide one or more types of information via an application programming interface (API).
[0105] Example 22. A method of implementing the one or more specialized AI agents in a computer according to any one of Examples 1 to 21, wherein the one or more specialized AI agents include (i) a web crawling agent and (ii) a search engine agent, or both.
[0106] Example 23. A method of implementing the one or more specialized AI agents in a computer according to any one of Examples 1 to 22, wherein the one or more specialized AI agents include one or more of a reporting agent, an optimization agent, a policy agent, or a diagnostic agent.
[0107] Example 24. A system comprising one or more processors and one or more memories that, when executed by the one or more processors, store instructions causing the one or more processors to perform a method implemented in any one of the computers of Examples 1 to 23.
[0108] Example 24. One or more non-temporary computer-readable media that, when executed by one or more processors, stores instructions causing the one or more processors to perform the method implemented in any one of the computers in Examples 1 to 23.
[0109] While the preceding text describes in detail many different aspects and embodiments of the present invention, it should be understood that the scope of the patent is defined by the claims language set out at the end of this patent. The embodiments for carrying out the invention should be interpreted as illustrative only and do not describe all possible embodiments, for describing all possible embodiments would be impractical, if not impossible. Numerous alternative embodiments can be implemented using the current art or art developed after the filing date of this patent, and these remain within the scope of the claims.
[0110] The following additional considerations apply to the above description and the appended claims. Throughout this specification, multiple examples may implement components, operations, or structures described as single examples. While individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously, and nothing requires the operations to be performed in the order shown. Structures and functions presented as separate components in the examples may be implemented as combined structures or components. Similarly, structures and functions presented as single components may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter of this disclosure.
[0111] Unless otherwise evident from the context of use, where the same set of “one or more processors” (or the same “multiple processors,” etc.) perform multiple operations, embodiments of the operation may include those in which the execution of the operations is divided among the processors in any suitable manner. For example, “generating X by one or more processors and generating Y by one or more processors” may include (1) embodiments in which one or more processors of a first set (e.g., in a first computing device) generate X and one or more processors of a separate, second set (e.g., in a different second computing device) independently generate Y; (2) embodiments in which all processors in the set of one or more processors (e.g., all within the same device or distributed across multiple devices) contribute to the generation of both X and Y; and (3) other variations.
[0112] Unless otherwise specified, any statements in this disclosure using words such as “process,” “calculate,” “determine,” “present,” or “display” may refer to the operation or process of a machine (e.g., a computer) that receives, stores, transmits, or displays information, and manipulates or transforms data that is represented as a physical (e.g., electronic, magnetic, or optical) quantity in one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components.
[0113] Where used in this disclosure, any reference to “one implementation” or “an implementation” means that certain elements, features, structures, or characteristics described in relation to an implementation are included in at least one implementation. The phrase “in one implementation” appearing in various places in the specification does not necessarily refer to the same implementation in all cases.
[0114] When used in this disclosure, “comprises,” “comprising,” “includes,” “including,” “has,” “having,” or any other variation thereof are intended to encompass non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements and may include other elements not expressly enumerated or inherent in such process, method, article, or apparatus. Furthermore, unless expressly stated otherwise, “or” means an inclusive or not an exclusive or. For example, condition A or B is satisfied by any one of the following: A is true (or exists) and B is false (or does not exist); A is false (does not exist) and B is true (or exists); and both A and B are true (or exist).
[0115] Those skilled in the art will understand, by reading this disclosure, further additional alternative structural and functional designs through the principles described herein. Therefore, while specific embodiments and applications have been described and explained, it should be understood that the disclosed embodiments are not limited to the exact structures and components disclosed herein. Various modifications, changes, and variations obvious to those skilled in the art may be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.
Claims
1. A computer-implemented method for accurate and low-latency processing of user requests, wherein the computer-implemented method is: Receiving user requests from client devices via the user interface, The process involves generating context-specific responses to user requests by an orchestrator that is communicatively coupled to multiple specialized artificial intelligence (AI) agents, at least in part, (i) generating a response deployment plan that designates one or more specialized AI agents from among the plurality of specialized AI agents based on the user request and (ii) one or more context signals, wherein generating the response deployment plan includes determining that the one or more specialized AI agents are relevant to the user request, and each of the plurality of specialized AI agents is configured to analyze queries specific to their respective domains. Executing a query to one or more specialized AI agents in accordance with the response deployment plan, In response to the execution of the aforementioned query, the system receives feedback from one or more of the specialized AI agents, Based on the feedback from each of the one or more specialized AI agents, generate a context-specific response. This generates the context-specific response to the user request, To cause the client device to present the context-specific response via the user interface, A method of implementation in a computer, including
2. The one or more specialized AI agents include a first specialized AI agent and a second specialized AI agent, The method implemented in the computer further includes the orchestrator maintaining (i) a first agent context based on past communications between the orchestrator and the first specialized AI agent, and (ii) a second agent context based on past communications between the orchestrator and the second specialized AI agent, wherein the orchestrator maintains the first agent context separately from the second agent context, and the method implemented in the computer further includes, The process includes querying one or more specialized AI agents, wherein the query includes (i) generating a first synthetic query for the first specialized AI agent based in part on the first agent context, and (ii) generating a second synthetic query for the second specialized AI agent based in part on the second agent context. A method implemented in a computer according to claim 1.
3. Querying one or more specialized AI agents includes sending a first synthetic query to the first specialized AI agent. Receiving each of the aforementioned feedbacks includes receiving from the first specialized AI agent (i) an intermediate inference supporting the response to the first synthetic query, and (ii) grounding data used by the first specialized AI agent to generate the intermediate inference. Generating the context-specific response includes generating the context-specific response based on the grounding data and the intermediate inference. A method implemented in a computer according to claim 1.
4. The first specialized AI agent includes a first AI agent response model configured to generate responses to synthetic queries based on intermediate inference, Generating the context-specific response includes not using any response generated by the first AI agent response model to generate the context-specific response. A method implemented in a computer according to claim 3.
5. Receiving each of the aforementioned feedbacks includes receiving instructions from the first specialized AI agent regarding how to read the grounding data. A method implemented in a computer according to claim 3.
6. The method implemented in a computer according to claim 1, wherein generating the response deployment plan and generating the context-specific response are performed by a single generative AI model.
7. The one or more context signals are One or more properties of an entity, Information related to the business of the aforementioned entity, or A computer-implemented method according to claim 1, comprising one or more pieces of information associated with the account of the entity.
8. The method implemented in a computer according to claim 1, wherein the one or more context signals include data indicating the state of the user interface.
9. Generating the context-specific response includes identifying the action to be taken in response to the user request. The method implemented in the computer further includes causing the action, A method implemented in a computer according to claim 1.
10. Identifying the aforementioned action includes determining one or more desired sets of actions via the user interface. Triggering the aforementioned action includes remotely acquiring control of the input function of the client device, thereby causing the client device to perform one or more desired sets of actions via the user interface. A method implemented in a computer according to claim 9.
11. The one or more specialized AI agents include a first specialized AI agent and a second specialized AI agent, Generating the context-specific response includes (i) resolving a conflict between a first feedback provided by the first specialized AI agent in response to the query execution and (ii) a second feedback provided by the second specialized AI agent in response to the query execution. A method implemented in a computer according to claim 1.
12. The computer-implemented method according to claim 1, wherein the response deployment plan further specifies the order in which the orchestrator queries the one or more specialized AI agents.
13. The computer-implemented method according to claim 12, wherein the sequence indicates that the orchestrator generates a synthetic query for querying a second specialized AI agent using the respective feedback from the first specialized AI agent.
14. The orchestrator further includes (i) maintaining historical data based on past interactions associated with the user of the client device, or (ii) an entity associated with the user or the client device. (i) generating the response deployment plan, and (ii) querying one or more specialized AI agents, either or both of these are based in part on the historical data. A method implemented in a computer according to claim 1.
15. The method implemented in a computer according to claim 14, wherein the history data includes chat history displayed at least partially on the user interface.
16. The method of implementing in a computer according to claim 1, wherein the one or more specialized AI agents include (i) a web crawling agent and (ii) a search engine agent, or both.
17. The one or more specialized AI agents described above are: Reporting agent, Optimization agent, Billing agent, Policy agent, or A method implemented in a computer according to claim 1, comprising one or more diagnostic agents.
18. It is a system, One or more processors, Includes one or more memories for storing instructions, and when an instruction is executed by the one or more processors, the one or more processors Receiving user requests from client devices via the user interface, The process involves generating context-specific responses to user requests by an orchestrator that is communicatively coupled to multiple specialized artificial intelligence (AI) agents, at least in part, (i) generating a response deployment plan that designates one or more specialized AI agents from among the plurality of specialized AI agents based on the user request and (ii) one or more context signals, wherein generating the response deployment plan includes determining that the one or more specialized AI agents are relevant to the user request, and each of the plurality of specialized AI agents is configured to analyze queries specific to their respective domains. Execute a query to one or more specialized AI agents in accordance with the response deployment plan. In response to the execution of the query, receive feedback from one or more specialized AI agents, and The process involves generating context-specific responses to user requests by generating context-specific responses based on the feedback from each of the one or more specialized AI agents, To cause the client device to present the context-specific response via the user interface, A system that performs actions including those mentioned above.
19. The one or more specialized AI agents include a first specialized AI agent and a second specialized AI agent, The operation further includes the orchestrator maintaining (i) a first agent context based on past communications between the orchestrator and the first specialized AI agent, and (ii) a second agent context based on past communications between the orchestrator and the second specialized AI agent, wherein the orchestrator maintains the first agent context separately from the second agent context, and the system further includes querying the one or more specialized AI agents, wherein querying the one or more specialized AI agents includes (i) generating a first synthetic query for the first specialized AI agent, partially based on the first agent context, and (ii) generating a second synthetic query for the second specialized AI agent, partially based on the second agent context. The system according to claim 18.
20. Querying one or more specialized AI agents includes sending a first synthetic query to the first specialized AI agent, and receiving the respective feedback includes receiving from the first specialized AI agent (i) an intermediate inference supporting the response to the first synthetic query, and (ii) grounding data used by the first specialized AI agent to generate the intermediate inference. Generating the context-specific response includes generating the context-specific response based on the grounding data and the intermediate inference. The system according to claim 18.
