Autonomous LLM-supported agents that create and execute a graph of tasks

Abstract

In an embodiment, autonomous large language model (LLM)-supported software agents complete a task by instantiating, causing execution, and inspecting the results of multiple other agents organized in a graph of subtasks. Programmatically, a task request can be received via gRPC or other calling or messaging protocols. An agent receiving a task request uses an action generator to automatically generate one or more tasks, such as search requests, data retrieval, data analysis, or calls to LLM APIs. The agent invokes a search handler to manage Internet search tasks by calling an external search engine. The agent invokes a result synthesizer to combine and interpret the results of searches or queries. The agent invokes a code executor to execute a script or other program code, predefined or automatically generated by the agent. The agent can distribute reports, messages, or other task execution results by publishing using an event service such as MQTT.

Description

[0001] ATTORNEY DOCKET NO. 093459.0134

[0002] 1

[0003] AUTONOMOUS LLM-SUPPORTED AGENTS THAT CREATE AND EXECUTE A GRAPH OF TASKS

[0004] CROSS-REFERENCE TO RELATED APPLICATIONS

[0005] This application claims the benefit under 35 U.S.C. 119(e) of provisional application 63 / 706,579, filed 10 / 11 / 2024, the entire contents of which are hereby incorporated by reference for all purposes as if fully set forth herein.

[0006] COPYRIGHT NOTICE

[0007] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright or rights. © 2024 Arta Finance, Inc.

[0008] TECHNICAL FIELD

[0009] One technical field is generative artificial intelligence. Another technical field is designing and implementing autonomous agent-based systems for complex informationgathering and synthesis tasks.

[0010] BACKGROUND

[0011] The approaches described in this section are approaches that could be pursued but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0012] Recent advancements in Large Language Models (LLMs) have revolutionized various fields, including natural language processing, code generation, and text summarization. LLMs possess an impressive ability to understand and process vast amounts of textual information, enabling them to perform tasks previously thought exclusive to human intelligence. However, current LLM-powered systems face significant limitations when dealing with complex information-gathering tasks that require iterative reasoning, strategic navigation through information sources, and dynamic task decomposition.

[0013] Traditional Al approaches often rely on static knowledge bases and predetermined rules, making them inflexible and incapable of handling the evolving nature of real-world problems. For instance, existing Al systems might excel at providing answers to specific ATTORNEY DOCKET NO. 093459.0134

[0014] 2 questions based on pre-trained datasets but struggle to adapt to scenarios where the information required is scattered across various sources, requires in-depth research and analysis, and necessitates a dynamic approach to problem-solving. Moreover, these systems often lack the ability to evaluate the reliability and relevance of information gathered from diverse sources, leading to potential inaccuracies and biased results.

[0015] The need for dynamic and adaptable Al systems has become increasingly apparent in various domains, including finance, research, and software development. In finance, for example, investment professionals are tasked with analyzing vast quantities of information about companies and markets to make informed decisions. This process requires them to navigate through company reports, news articles, financial data, and other sources to extract relevant information, synthesize it into a coherent narrative, and ultimately form a judgment about the investment potential of a specific company. Similarly, in the research domain, scientists must sift through numerous scholarly articles, datasets, and experimental results to draw meaningful conclusions and identify potential research avenues.

[0016] SUMMARY

[0017] The appended claims may serve as a summary of the invention.

[0018] BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the drawings:

[0020] FIG. 1 illustrates a distributed computer system showing the context of use and principal functional elements with which one embodiment could be implemented.

[0021] FIG. 2 illustrates an example process for performing a complex research task using a plurality of agents in a DAG in accordance with an embodiment.

[0022] FIG. 3 illustrates the architecture of an agent in accordance with an embodiment.

[0023] FIG. 4 illustrates a computer system that could be implemented for one embodiment. DETAILED DESCRIPTION

[0024] 1. INTRODUCTION

[0025] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention. ATTORNEY DOCKET NO. 093459.0134

[0026] 3

[0027] The text of this disclosure, in combination with the drawing figures, is intended to state in prose the algorithms that are necessary to program the computer to implement the claimed inventions at the same level of detail that is used by people of skill in the arts to which this disclosure pertains to communicate with one another concerning functions to be programmed, inputs, transformations, outputs and other aspects of programming. That is, the level of detail set forth in this disclosure is the same level of detail that persons of skill in the art normally use to communicate with one another to express algorithms to be programmed or the structure and function of programs to implement the inventions claimed herein.

[0028] This disclosure may describe one or more different inventions, with alternative embodiments to illustrate examples. Other embodiments may be utilized, and structural, logical, software, electrical, and other changes may be made without departing from the scope of the particular inventions. Various modifications and alterations are possible and expected. Some features of one or more of the inventions may be described with reference to one or more particular embodiments or drawing figures, but such features are not limited to usage in the one or more particular embodiments or figures with reference to which they are described. Thus, the present disclosure is neither a literal description of all embodiments of one or more inventions nor a listing of features of one or more inventions that must be present in all embodiments.

[0029] Headings of sections and the title are provided for convenience but are not intended to limit the disclosure in any way or as a basis for interpreting the claims. Devices described as in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise. In addition, devices that communicate with each other may communicate directly or indirectly through one or more intermediaries, logical or physical.

[0030] A description of an embodiment with several components in communication with one another does not imply that all such components are required. Optional components may be described to illustrate a variety of possible embodiments and to illustrate one or more aspects of the inventions fully. Similarly, although process steps, method steps, algorithms, or the like may be described in sequential order, such processes, methods, and algorithms may generally be configured to work in different orders unless specifically stated to the contrary. Any sequence or order of steps described in this disclosure is not a required sequence or order. The ATTORNEY DOCKET NO. 093459.0134

[0031] 4 steps of the described processes may be performed in any order that is practical. Further, some steps may be performed simultaneously. The illustration of a process in a drawing does not exclude variations and modifications, does not imply that the process or any of its steps are necessary to one or more of the invention(s), and does not imply that the illustrated process is preferred. The steps may be described once per embodiment, but need not occur only once. Some steps may be omitted in some embodiments or occurrences, or some steps may be executed more than once in a given embodiment or occurrence. When a single device or article is described, more than one device or article may be used in place of a single device or article. Where more than one device or article is described, a single device or article may be used instead of more than one device or article.

[0032] The functionality or features of a device may be alternatively embodied by one or more other devices that are not explicitly described as having such functionality or features. Thus, other embodiments of one or more inventions need not include the device itself. Techniques and mechanisms described or referenced herein will sometimes be described in singular form for clarity. However, it should be noted that particular embodiments include multiple iterations of a technique or manifestations of a mechanism unless noted otherwise. Process descriptions or blocks in figures should be understood as representing modules, segments, or portions of code, including one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of embodiments of the present invention in which, for example, functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved.

[0033] 2. STRUCTURAL & FUNCTIONAL OVERVIEW

[0034] In an embodiment, the challenges outlined in the Background are addressed with an agent-based system for complex information-gathering tasks. The system comprises a Directed Acyclic Graph (DAG) of autonomous agents. Each agent is programmed to dynamically decompose a given task into smaller, more manageable sub-tasks. Each agent within the DAG is equipped with its own resources and capabilities, allowing it to perform specific actions and gather information relevant to its assigned sub-task.

[0035] The system’s action generation mechanism utilizes LLMs to evaluate the context of each sub-task and determine the optimal action based on available resources. This mechanism ATTORNEY DOCKET NO.

[0036] 093459.0134

[0037] 5 enables the agents to dynamically adapt their strategies based on the information they gather, navigate through various information sources effectively, and even generate code to automate certain aspects of the research process.

[0038] Furthermore, an evaluation framework uses LLMs to assess the quality and relevance of the information collected by the agents. This evaluation process helps ensure the accuracy and reliability of the final synthesized output, reducing the risk of bias and providing users with a comprehensive and trustworthy analysis.

[0039] The agent-based approach of the disclosure represents a significant departure from existing Al systems, offering a more flexible and adaptable solution for complex informationgathering tasks. It enables the creation of truly autonomous Al agents capable of performing in-depth research, analyzing diverse information sources, and synthesizing meaningful insights while ensuring the accuracy and reliability of the final output. The system’s ability to dynamically adapt to new information and modify its approach based on real-time insights makes it a valuable tool for many applications, promising to significantly improve the efficiency and effectiveness of information-gathering and knowledge discovery across various domains.

[0040] 2.1 DISTRIBUTED COMPUTER SYSTEM EXAMPLE

[0041] FIG. 1 illustrates a distributed computer system showing the context of use and principal functional elements with which one embodiment could be implemented. In an embodiment, a computer system 100 comprises components implemented partially by hardware at one or more computing devices, such as one or more hardware processors executing stored program instructions stored in memory for performing the functions described herein. In other words, all functions described herein are intended to indicate operations performed using programming in a special or general-purpose computer in various embodiments. FIG. 1 illustrates only one of many possible arrangements of components configured to execute the programming described herein. Other arrangements may include fewer or different components, and the division of work between the components may vary depending on the arrangement.

[0042] FIG. 1, and the other drawing figures and all of the description and claims in this disclosure, are intended to present, disclose, and claim a technical system and technical methods in which specially programmed computers, using a special-purpose distributed ATTORNEY DOCKET NO. 093459.0134

[0043] 6 computer system design, execute functions that have not been available before to provide a practical application of computing technology to the problem of complex data analysis. In this manner, the disclosure presents a technical solution to a technical problem, and any interpretation of the disclosure or claims to cover any judicial exception to patent eligibility, such as an abstract idea, mental process, method of organizing human activity, or mathematical algorithm, has no support in this disclosure and is erroneous.

[0044] In the example of FIG. 1, computing device 102 is communicatively coupled via network 120 to a server computer 104. The computing device 102 can comprise a desktop computer, laptop computer, tablet computer, workstation, or other end station comprising an input device such as a keyboard or touchscreen, memory, processor, one or more output devices such as a flat-screen display, and one or more application programs such as a browser or editor. Network 120 broadly represents any combination of packet-switched networks, such as a local area network, wide area network, campus network, or combination, communicatively coupled to one or more internetworks.

[0045] Server computer 104 comprises a processor 106 that hosts an operating system 108 and is coupled to memory 112, such as DRAM, SRAM, or NVRAM. The memory 112 stores a control application 110, which executes under the control of the operating system 108. The control application 110 is programmed in part to instantiate and launch a Main Agent 202, which is further described herein in other sections in connection with FIG. 2. The Main Agent 202 interoperates with a hierarchical directed acyclic graph (DAG) structure 210 of sub-agents as further described.

[0046] In an embodiment, an LLM server 130, a web search engine 140, and a database server 150 are communicatively coupled to the network 120. One or more of the server computer 104, LLM server 130, web search engine 140, and database server 150 can be implemented using one or more virtual compute instances and / or virtual storage in a private data center, public data center, or cloud computing facility.

[0047] The LLM server 130 can comprise an application programming interface (API) capable of receiving programmatic calls from the Main Agent 202 and / or one or more of the sub-agents. Examples of LLM APIs that can be accessed and used include OpenAI or ChatGPT, Google Bard, Anthropic Claude, etc. The web search engine 140 can comprise a server with a callable API capable of conducting website searches and returning result sets of URLs and metadata ATTORNEY DOCKET NO. 093459.0134

[0048] 7 about the URLs. Any commercial web search engine, such as Google, Bing, DuckDuckGo, or others, can be used. The database server 150 can comprise an enterprise database server with an API or other callable interface. The database server 150 can be associated with a specific business enterprise or other institution, such as the same enterprise that owns and operates the server computer 104 or a different enterprise.

[0049] In this arrangement, as further described in other sections, the computing device 102 can interoperate with the control application to define a complex research task in electronic digital text or structured stored data and request the Main Agent 202 to execute the task. Task execution can include the Main Agent 202 calling or submitting subtasks to one or more of the LLM Server 130, web search engine 140, and database server 150, and / or instantiating one or more subagents of the DAG 210.

[0050] 2.2 EXAMPLE SOFTWARE ARCHITECTURE

[0051] In an embodiment, autonomous large language model (LLM)-supported software agents are programmed to complete a specified task by instantiating, causing execution, and inspecting the results of multiple other agents organized in a graph of subtasks. Programmatically, a task request can be received via gRPC or other calling or messaging protocols. An agent receiving a task request uses an action generator to automatically generate one or more tasks, such as search requests, data retrieval, data analysis, or calls to LLM APIs. The agent can invoke a search handler to manage Internet search tasks by calling an external search engine. The agent invokes a result synthesizer to combine and interpret the results of searches or queries. The agent may invoke a code executor to execute a script or other program code predefined or automatically generated by the agent. The agent can distribute reports, messages, or other task execution results by publishing using an event service such as MQTT.

[0052] FIG. 2 illustrates the high-level architecture of an embodiment, showing the main components and their relationships. FIG. 2 and each other flow diagram herein are intended as an illustration of the functional level at which skilled persons, in the art to which this disclosure pertains, communicate with one another to describe and implement a computer-implemented method, as described further herein and / or algorithms using programming. The flow diagrams are not intended to illustrate every instruction, method, object, or sub-step that would be needed to program every aspect of a working program, but are provided at the same functional level ATTORNEY DOCKET NO.

[0053] 093459.0134

[0054] 8 of illustration that is normally used at the high level of skill in this art to communicate the basis of developing working programs.

[0055] In an embodiment, a task execution system 200 can comprise a Main Agent 202, the central coordinator for processing a Complex Research Task 201. The Complex Research Task 201 can be defined using unstructured digitally stored text, forming a natural language prompt or task description, or structured storage like JSON. The Main Agent 202 can be implemented as one or more computer programs, methods, functions, or other software elements in memory 112 and programmed to receive input tasks, automatically programmatically instantiate one or more sub-agents 220, and manage the overall workflow of system 200 as described further in FIG. 3. Each of the sub-agents 220 can be implemented as one or more computer programs, methods, functions, or other software elements that are programmed to execute the functions further described herein.

[0056] The system 200 can utilize the DAG structure 210 to organize and manage flows of information and tasks between agents. The DAG structure 210 can provide an efficient task decomposition and parallel processing framework. In some embodiments, the DAG structure 210 can be implemented using a graph data structure, with nodes representing agents and edges representing task assignments and information flow. In some embodiments, open-source software can be integrated into system 200 to create and manage the DAG structure 210; an example is the Trellis framework available on Github.

[0057] The system's output can be a Final Report 270, which can be a comprehensive document containing the synthesized results of the research task. The Final Report 270 can be generated in a structured format like JSON to facilitate further processing or integration with other systems, or constitute unstructured natural language electronic digital text corresponding to the structured data. The Final Report 270 can include relevant information, analysis, and source links to provide a transparent and traceable account of the research process. Some embodiments can further include a Fact Validation 280 component programmed for evaluating the factual veracity of claims made in the Final Report 270 in view of the results of individual Sub Agents 220. The Fact Validation 280 can be implemented as one or more computer programs, methods, functions, or other software elements in memory 112 ATTORNEY DOCKET NO.

[0058] 093459.0134

[0059] 9

[0060] 2.2.1 SUB-AGENTS

[0061] FIG. 3 depicts an example of software architecture and data flows of a Sub-Agent in one embodiment. In some embodiments, the Main Agent 202 can be structured like a Sub- Agent 220, sharing a common architecture and / or using common programmatic classes or methods. System 200 can include a plurality of Sub-Agents 220, comprising specialized software modules designed to handle specific tasks or access particular resources. Each Sub- Agent 220 can be configured to receive a Task 301 from the Main Agent 202 or other Sub- Agents higher in the DAG hierarchy. In some embodiments, where the Main Agent 202 shares a common architecture with Sub-Agents 220, Task 301 can be a Complex Research Task 201. The Sub-Agents 220 can be implemented as separate program processes or threads, allowing concurrent execution and scalability.

[0062] An Action Generator 230 can be associated with each agent in the system. The Action Generator 230 can be a software component programmed to determine the optimal course of action for completing an assigned task. The Action Generator 230 can be programmed to evaluate available resources and generate a plan of action, including web searches, database queries, code execution, or further task decomposition.

[0063] Upon receiving Task 301, Action Generator 230 can initially determine whether the Agent can complete Task 301 or must perform Task Decomposition 310. During this step, the Sub Agent 220 can analyze the complexity and requirements of the task, breaking it down into smaller, more manageable subtasks. Whether a task needs to be further decomposed can be determined based on a list of predetermined criteria, such as whether the task has subparts, is too complex to handle by a single agent or other criteria. In some embodiments, the maximum depth of the DAG can be determined in advance, and Sub Agent 220 can determine not to perform Task Decomposition 310 because further task decomposition would result in exceeding the predetermined depth of the DAG.

[0064] Where a Sub Agent 220 has decided to decompose a task, the system can proceed with Sub-Agent Task Assignment 320. In this step, the Sub Agent 220 can delegate the identified subtasks to appropriate Sub-Agents 220 within the DAG structure 210. The assignment process can consider the specializations and capabilities of each Sub-Agent 220, ensuring that subtasks are allocated to the most suitable agents for efficient processing. ATTORNEY DOCKET NO. 093459.0134

[0065] 10

[0066] As Sub-Agents 220 complete their assigned subtasks, they can engage in Result Propagation 350. During this phase, the results and findings from each Sub-Agent 220 can be transmitted back up the DAG hierarchy 210. The Result Propagation 350 process can ensure that information flows efficiently from the leaf nodes of the DAG to higher-level agents, ultimately reaching the Main Agent 202. This step can facilitate the aggregation of results from multiple subtasks and contribute to the overall solution of the complex research task.

[0067] A Result Synthesizer 330 can be included in Sub-Agent 220 to compile and integrate the results from various Sub-Agents 220. This component can aggregate information, resolve conflicts, and organize the collected data into a coherent and structured format. Once the assigned Sub-Agents have completed their tasks, the results can be returned to a Result Synthesizer 330, aggregating the propagated results and resolving any conflicts or inconsistencies in the gathered information. The Sub-Agent Task Assignment 330 process can aim to create a coherent and comprehensive set of findings that address the original Task 301.

[0068] By way of non-limiting example, below is an example of an Action Generator in accordance with an embodiment, coded in Python: class ActionGenerator : def init ( self , 11m) : self . 11m = 11m def generate prompt ( self , tas k : str, depth : int , max depth : int ) -> str :

[0069] "" "Prompt"" " return f"""

[0070] Current depth : { depth }

[0071] Maximum depth : {max depth }

[0072] Maximum number of sub-tas ks : { self . max subtasks }

[0073] Your goal is to determine the best course of action to gather information for this task . Follow these steps :

[0074] 1 . Generate a search query that would help gather information for this task . ATTORNEY DOCKET NO. 093459.0134

[0075] 11

[0076] 2. If the current depth is less than the maximum depth, decide if this task needs to be broken down into sub-research tasks for more in depth research.

[0077] 3. If more research is needed to be done and depth allows, provide a list of sub-research tasks. Limit the number of sub-tasks to a maximum of [{self. max subtasks}] .

[0078] Respond with a JSON object in the following format: ' j son {{

[0079] "search query": "Your generated search query...", "require decomposition": true / false, "subtasks": ["Taskl", "Task2"...] , "reasoning": "Your reasoning for the chosen action ..."

[0080] } }

[0081] Note: Focus on information gathering rather than problem-solving at this stage. If the current depth equals the maximum depth, do not suggest subtasks.

[0082] Task: {task} H TT TT def parse response ( self , response: str, depth: int, max depth: int) -> LLMTaskResponse :

[0083] """Parse response""" try: data = son . loads (response ) require decomposition = data . get ( ' require decomposition ' ) subtasks = data . get (' subtasks ' ) if require decomposition is True and depth < max depth: assert subtasks is not None and len ( subtasks ) <= self. max subtasks, \ ATTORNEY DOCKET NO. 093459.0134

[0084] 12 f' Subtasks required for decomposition at depth: {depth} ' elif require decomposition is False or depth == max depth: assert subtasks is None, f' Subtasks are not required for this task at depth: {depth} ' return LLMTaskResponse ( search query=data [' search query' ] , require decomposition=require decomposition, subtasks=subtasks , reasoning=data [ ' reasoning ' ] except j son . JSONDecodeError as e: raise ValueError ( f ' Failed to parse JSON response: {e} ' ) def generate ( self , task: str, depth: int, max depth: int) -> LLMTaskResponse :

[0085] """Generate action""" prompt = self . generate prompt (task=task, depth=depth, max depth=max depth) response = self .11m. generate (prompt=prompt ) return self. parse response ( response=response , depth=depth, max depth=max depth)

[0086] 2.2.2 TASK EXECUTION

[0087] If no further task decomposition is necessary or permitted, the Action Generator 230 can be programmed to perform Task Execution 340 to formulate a plan of action that may include various operations such as web searches, database queries, or further task decomposition, depending on the nature of the subtask and the agent’s capabilities. This step involves carrying out the planned actions to gather information, perform analysis, or generate results relevant to the assigned subtask. The Task Execution 340 phase can involve interactions with a set of Available Resources 350, comprising various tools and data sources that the agent can leverage to complete its tasks. These resources can be internal and external to the system, providing a comprehensive toolkit for information gathering, analysis, and task execution. The ATTORNEY DOCKET NO.

[0088] 093459.0134

[0089] 13

[0090] Available Resources 350 can be dynamically updated and expanded as new capabilities or data sources become available.

[0091] The system 200 can have access to a variety of Data Sources 351, which can include both internal and external resources. These Data Sources 351 can comprise web-based information, proprietary databases, PDF documents, and other structured and unstructured data repositories. Data Sources 351 can include LLM Server 130 (FIG. 1), web search engine 140, and database server 150. The system 200 can be programmed to efficiently parse and extract relevant information from diverse data formats, including the ability to handle PDF documents and other complex file types.

[0092] In some embodiments, the Available Resources 350 can have various types or kinds with specialized capabilities, like web search, access to certain databases, the ability to execute code snippets, or combinations thereof. The Available Resources 350 can be a Web Search Module 352 in some embodiments. Agents can programmatically call the Web Search Module

[0093] 352 to enable the agents to perform targeted searches of web servers or other resources available via programmatic calls into the network 120 or on the internet to gather relevant information for their assigned tasks. The Web Search Module 352 can incorporate advanced search algorithms and filtering mechanisms to ensure that the information retrieved is both relevant and reliable. It can interface with various search engines and web crawlers to provide comprehensive coverage of online information sources.

[0094] In some embodiments, the Available Resources 350 can be a Database Query Module

[0095] 353 to access databases within the system, such as SQL, NoSQL, Document Management Systems, or other similar data repositories. This module can allow agents to retrieve structured data, historical records, or pre-processed information relevant to their tasks.

[0096] To support advanced analysis and data processing tasks, the Available Resources 350 can incorporate a Code Execution Module 354. This module can allow agents to write and execute custom code snippets, typically in Python or other programming languages. In some embodiments, the code snippets can be written by the agent itself. In some embodiments, the code snippets can be selected from a pre-generated list of available code snippets. The Code Execution Module 354 can provide a secure sandboxed environment for running code, ensuring that the execution does not compromise the system’s integrity or security. ATTORNEY DOCKET NO. 093459.0134

[0097] 14

[0098] Given the importance of extracting information from PDF documents, the Available Resources 350 can include a dedicated PDF Parsing Module 390. This module can be specialized in handling PDF files, capable of extracting both text and structured data from various PDF formats. The PDF Parsing Module 355 can employ advanced optical character recognition (OCR) techniques and layout analysis algorithms to accurately extract information from complex PDF documents, including tables, charts, and formatted text.

[0099] The Available Resources 350 described above are provided as examples only. The Available Resources 350 can further include any data retrieval, analysis, or extraction system that provides an API that permits the Sub Agent 220 to call upon that resource when necessary.

[0100] After either the completion of the Task Execution 340 by the Sub Agent 220, or the completion of the Result Synthesizer 330 after task decomposition, the Sub Agent is programmed to produce an Agent Output 360.

[0101] In some embodiments, the system can incorporate an LLM Evaluator 370, which can be a language model configured to assess the outputs of the agents. The LLM Evaluator 370 can analyze the Agent Output 360 and determine whether tasks have been successfully completed, failed, or require further refinement. This component can provide an objective assessment of the quality and relevance of the information gathered and processed by the agents. This component is discussed in further detail herein.

[0102] After completion of the LLM Evaluator 370 (in embodiments with such a component) or after the generation of an Agent Output 360, the Sub Agent can perform Result Propagation 350 by passing its final result and optional evaluation from the LLM Evaluator 370 to the component that created it. Where the Sub Agent 220 is not the Main Agent 202, the Result Propagation 350 step returns the results to an Agent in the next higher level of the DAG. Where the Sub Agent 220 is the Main Agent 220, the Result Propagation 350 step returns the result to the Final Report Generator 270.

[0103] The workflow depicted in FIG. 3 can support parallel processing of subtasks. Multiple Sub-Agents 220 can simultaneously execute their respective workflows for their respective subtasks. This parallel processing capability can significantly enhance the efficiency of the system 200, allowing it to handle complex research tasks with reduced overall processing time. ATTORNEY DOCKET NO.

[0104] 093459.0134

[0105] 15

[0106] 2.2.3 EVALUATOR

[0107] In an embodiment, the LLM Evaluator 370 is programmed as an independent language model configured to objectively assess the quality and relevance of information gathered and processed by the agents. The LLM Evaluator 370 can support the overall effectiveness and accuracy of the system’s outputs. The LLM Evaluator 370 can be implemented as a separate module within the system 200. It can be designed to operate independently from the agents, providing an assessment of their performance. In some embodiments, the LLM Evaluator 370 can be based on a large language model that has been fine-tuned for the specific task of evaluating research outputs and determining task completion status.

[0108] An evaluation process can begin with the LLM Evaluator 370 receiving an Agent Output 360. This output can result from a task or subtask completed by any agent within the system, including the Main Agent 202 or any of the Sub-Agents 220 described in Figure 1. The Agent Output 360 can contain various types of information, such as collected data, analysis results, or synthesized reports, depending on the nature of the assigned task.

[0109] To assess the Agent Output 360, the LLM Evaluator 370 can utilize a set of evaluation criteria. These criteria can be predefined guidelines or benchmarks that help determine the output's quality, completeness, and relevance. The evaluation criteria can include factors such as information accuracy, analysis comprehensiveness, adherence to task requirements, and clarity of presentation. These criteria can be customized based on the specific domain or type of research task being evaluated.

[0110] The LLM Evaluator 370 is programmed to perform a task success assessment in an embodiment. During this phase, the evaluator can analyze Agent Output 360 against the evaluation criteria to determine if the assigned task has been successfully completed. The task success assessment can involve checking if all required information has been gathered, if the analysis is thorough and accurate, and if the output adequately addresses the original task objectives.

[0111] If the LLM Evaluator 370 determines that the task has been successfully completed, it can generate a positive assessment. This assessment can include a detailed explanation of how the output meets or exceeds the evaluation criteria, highlighting particular strengths or noteworthy aspects of the agent’s performance. The successful task completion status can then ATTORNEY DOCKET NO. 093459.0134

[0112] 16 be communicated back to the relevant components of the system, such as the Main Agent 202 or the Result Synthesizer 330.

[0113] In cases where the Agent Output 360 does not meet the necessary standards, the LLM Evaluator 370 can conduct a task failure assessment. This assessment can identify specific areas where the output falls short of the evaluation criteria. The task failure assessment can provide a detailed analysis of the deficiencies in the agent’s output, such as missing information, inaccuracies, or inadequate analysis.

[0114] The failure assessment can be designed to be constructive, offering specific insights into why the task is considered incomplete or unsuccessful. This information can improve the agents' performance and refine the overall system. The task failure assessment can be communicated to the relevant agent or to the Main Agent 202, potentially triggering a reassignment of the task or a refinement of the agent’s approach.

[0115] By way of non-limiting example, below is an example embodiment in Python of an LLM Evaluator in accordance with an embodiment: class LLMEvaluationResultModel ( BaseModel ) : evaluation : str = Field ( . . . , description="Yes ' or ' No ' - evaluation of the LLM" ) confidence : str = Field ( . . . , description="High ' , ' Medium' or ' Low ' - confidence level" ) explanation : str = Field ( . . . , description="Brief explanation of the evaluator" ) def generate j son model (model : BaseModel ) -> str : return j son . dumps (model , indent=2 ) class LLMEvaluator : def init ( self , 11m: LLM) : self . 11m = 11m def evaluate ( self , response : str , goal : str ) -> bool : H TT TT

[0116] Uses the LLM to evaluate if the response answers the goal and determines the confidence level . ATTORNEY DOCKET NO. 093459.0134 eval prompt = self, get evaluation prompt (response, goal) evaluation response = self .11m. generate response ( eval prompt) evaluation = self, parse evaluation response (evaluation response) return evaluation result . evaluation . lower ( ) == "yes def get evaluation prompt (self, response: str, goal: str) -> str : n n n

[0117] Evaluate if the answer satisfactorily addresses the question .

[0118] Provide your evaluation in a JSON object that conforms to the following schema:

[0119] {

[0120] "evaluation": "Yes / No",

[0121] "confidence " : "High / Medium / Low" , "explanation" : "explanation" }

[0122] ' tool code print (f ' {response} ' ) n n n def parse evaluation response ( self , response: diet [str, Any] ) - > LLMEvaluation: try: eval data = son . loads ( response [ "content "] ) return LLMEvaluation ( **eval data) except son . JSONDecodeError as e: ATTORNEY DOCKET NO. 093459.0134

[0123] 18 return LLMEvaluation ( evaluation ' No ' , conf idence= ' Low ' , explanation=f ' Error in LLM evaluation : { e } ' ) def print evaluation result ( self , eval result : LLMEvaluation) : print ( f ' Evaluation : { eval result . evaluation } ' ) print ( f ' Conf idence : { eval result . conf idence } ' ) print ( f ' Explanation : { eval result . explanation } ' )

[0124] 2.2.4 FINAL REPORT GENERATION

[0125] The workflow can culminate in Final Report Generation 270. The synthesized results can be organized and formatted during this phase into a structured output, such as the Final Report 270 described in FIG. 2. The Final Report Generation 270 process can involve ordering or arranging the information logically, generating appropriate summaries and analyses, and including relevant source links for transparency and traceability. The resulting report can provide a comprehensive response to the initial Complex Research Task 201, leveraging the collective efforts of the agent network.

[0126] The Result Synthesizer 330 in the Main Agent 202 can execute part of Final Report Generation 270. The Result Synthesizer 330 can be implemented as a software module that handles diverse information and data formats. It can employ advanced natural language processing and data integration techniques to combine and harmonize the results from various sources, such as a large language model.

[0127] The Result Synthesizer 330 can receive results from the Result Propagation 350 of its Sub-Agents as its primary input. The Sub-Agent results can include a wide range of information types, such as raw data, analysis results, summaries, and recommendations, depending on the nature of the assigned subtasks and the capabilities of each Sub-Agent 220.

[0128] Upon receiving the Sub-Agent Results 610, the Result Synthesizer 330 can compile information generated in prior phases. During this phase, the system can organize and categorize the collected information based on relevance, importance, and logical flow. The information compilation process can involve identifying common themes, eliminating redundancies, and coherently structuring the information. This step can be crucial in transforming the diverse inputs from multiple Sub-Agents 220 into a unified and comprehensive body of information. ATTORNEY DOCKET NO.

[0129] 093459.0134

[0130] 19

[0131] Following the compilation of information, the system 200 is programmed to proceed with analysis integration. This step can involve combining and reconciling the analytical insights provided by various Sub-Agents 220. The analysis integration process can identify complementary findings, resolve potential conflicts in interpretations, and synthesize a cohesive analytical narrative. The system can employ advanced algorithms to weigh different analyses based on their relevance and reliability, ensuring that the integrated analysis provides a balanced and accurate representation of the research findings.

[0132] The system 200 can be programmed to collect source links to maintain transparency and traceability in the final report. This process can involve gathering and organizing the references and sources used by the Sub-Agents 220 in their research. The source link collection can ensure that each information or analysis in the final report can be traced back to its original source. This feature can enhance the credibility of the report and allow users to verify information or delve deeper into specific aspects of the research.

[0133] The system 200 can be programmed to generate a structured report next, where the compiled information, integrated analysis, and collected source links can be organized into a predefined structured format. As mentioned in the description of Figure 1, this structured format can be a JSON object, facilitating easy parsing and integration with other systems. The structured report generation process can involve mapping different types of content to appropriate fields within the JSON structure, ensuring consistency, and facilitating programmatic access to the report’s contents.

[0134] Below is an example of a Final Report generator in accordance with an embodiment, in Python, intended for use in producing a report regarding potential investment strategies:

[0135] @dataclass class Source : url : str description : str

[0136] @dataclass class SecurityBaseModel : ticker : str name : str ATTORNEY DOCKET NO. 093459.0134

[0137] 20 description: str side: Side sources: List [Source] = field (default_factory=list) relevance_score : float = field(ge=-2, le=2) analysis: str = "" pros: List [str] = field (default_factory=list) cons: List [str] = field (default_factory=list)

[0138] @dataclass class ThematicPortf olio (BaseModel ) : theme: str theme_summary : str economic_summary : str securities: List [ SecurityBaseModel ] = field (def ault_f actory=list ) sources: List [Source] = field (default_factory=list) def > post_init (self) :

[0139] # this only applies to strings, requires to have the field a minimum length. if len ( self . theme_summary ) < 1: raise ValueError ( "Theme summary is required") if len ( self . economic_summary) < 1: raise ValueError ( "Economic summary is required") if len ( self . securities ) < 1: raise ValueError ( "At least one security is required" ) return self def not_empty_theme_summary ( self ) -> bool: return self . theme_summary and self . theme_summary != ATTORNEY DOCKET NO. 093459.0134

[0140] 21 def not_empty_economic_summary ( self ) -> bool: return self . economic_summary and self . economic_summary

[0141] I _ n n def not_empty_securities ( self ) -> bool: return self . securities and len ( self . securities ) > 0 def not_empty_sources ( self ) -> bool: return self. sources and len ( self . sources ) > 0 def _is_valid ( securities : List, sources: List, info: diet) -> bool : min_len = 1 def check_length ( field : str, info: diet) : if field not in info: raise ValueError ( f 'Missing required field:

[0142] " { field} " ’ ) if len ( info . get ( field) ) < min_len: raise ValueError ( f Field "{field}" must contain at least

[0143] {min_len} item(s) ') check_length (' securities ' , info) return True def parse_j son_response ( response : str) -> ThematicPort folio : try : portfolio = j son . loads ( response ) if _is_valid (port folio . get (' securities ' , [] ) , port folio . get (' sources ' , [] ) , portfolio) : return ThematicPort folio (* *port folio ) else : ATTORNEY DOCKET NO. 093459.0134

[0144] 22 raise ValueError ( "Expected portfolio in JSON response" ) except j son . JSONDecodeError as e: raise ValueError ( f ' Failed to parse JSON response: { e } ’ )

[0145] SYSTEM_PROMPT = """You are an Al assistant specialized in synthesizing web search results to address the following goals :

[0146] 1. Your goal is to provide a concise, relevant summary of the search results that directly address the task at hand.

[0147] 2. **DO NOT** generate, modify, guess, or invent any of the exact results provided in the context.

[0148] 3. Focus on qualitative information and avoid specific financial figures or metrics.

[0149] 4. Prioritize the most recent information available in your search results.

[0150] 5. If the information seems outdated, make a note of it in your summary.

[0151] You are an Al assistant specialized in generating qualitative investment reports for stocks and ETFs. Your task is to advise me on { ticker } .

[0152] Generate a report with the following sections:

[0153] 1. Company Overview (150-250 words)

[0154] 2. Market Position and Competitive Landscape (200-300 words)

[0155] 3. Management and Governance (150-250 words)

[0156] 4. Innovation and Growth Prospects (200-300 words)

[0157] 5. Recent Developments (150-250 words)

[0158] 6. ESG Considerations (150-250 words)

[0159] 7. Risks and Challenges (250-350 words) ATTORNEY DOCKET NO. 093459.0134

[0160] 23

[0161] 8. Investment Thesis (250-350 words)

[0162] For each section:

[0163] A. Provide qualitative analysis, focusing on descriptive information rather than specific financial data.

[0164] B. Use comparative language (e.g., "strong performer" in its sector, rather than specific growth percentage) .

[0165] C. Use hedging language to acknowledge the limitations of the analysis (e.g., "may," "could," "appears to be") .

[0166] D. Provide context and explanation for all assertions.

[0167] E. Prioritize the most recent information available. If using older information, clearly state its age and its relevance.

[0168] F. Include at least three hyperlinks to reputable sources that support the information in your report.

[0169] Format your response as a JSON object with a single key "report" containing the InvestmentReport data. The InvestmentReport object should conform to the following schema :

[0170] ' ' ' j son

[0171] {

[0172] "report" : { {

[0173] "company_name" :

[0174] "ticker":

[0175] "report_type" : "market_overview" : "company_position" : "management_and_governance" : " innovation_and_growth" : " . .

[0176] "recent_developments" : "..." "esg_considerations" :

[0177] "risks_and_challenges" : "... ATTORNEY DOCKET NO.

[0178] 093459.0134

[0179] 24

[0180] " investment_thesis " :

[0181] " sources " : [

[0182] { { "url" : "description" :

[0183] } } }

[0184] 2.2.5 FACT VALIDATION

[0185] Before finalizing the report, the system 200 can be programmed to employ a Fact Validation Pipeline 280. This component can serve as an additional layer of quality control, independently verifying the accuracy of key facts and assertions made in the report. The Fact Validation Pipeline 280 can utilize external databases, trusted sources, and potentially additional LLM queries to cross-check important information. This process can help identify and correct any potential inaccuracies that may have been introduced during the research and synthesis phases.

[0186] The culmination of the result synthesis and report generation process can be the Final Report Output 270. This output can be a comprehensive document that addresses the original Complex Research Task 201 described in Figure 2. The Final Report 270 can be structured in a way that presents the synthesized information, integrated analysis, and validated facts in a clear and logical manner. It can include an executive summary, detailed findings, supporting evidence, and a complete list of sources, all organized within the JSON structure generated in the previous step.

[0187] The result synthesis and report generation process can be designed with customization and flexibility. The system can support different report templates or structures based on the specific requirements of different research domains or user preferences. The Structured Format Generation 650 phase can be configured to produce outputs in various formats beyond JSON, such as XML or custom data structures, to accommodate diverse integration needs.

[0188] 3. IMPLEMENTATION EXAMPLE - HARDWARE OVERVIEW

[0189] According to one embodiment, the techniques described herein are implemented by at least one computing device. The techniques may be implemented in whole or in part using a combination of at least one server computer and / or other computing devices coupled using a network, such as a packet data network. The computing devices may be hard-wired to perform ATTORNEY DOCKET NO. 093459.0134

[0190] 25 the techniques or may include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field programmable gate array (FPGA) that is persistently programmed to perform the techniques or may include at least one general purpose hardware processor programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. To accomplish the described techniques, such computing devices may combine custom hard-wired logic, ASICs, or FPGAs with custom programming. The computing devices may be server computers, workstations, personal computers, portable computer systems, handheld devices, mobile computing devices, wearable devices, body-mounted or implantable devices, smartphones, smart appliances, internetworking devices, autonomous or semi-autonomous devices such as robots or unmanned ground or aerial vehicles, any other electronic device that incorporates hard-wired and / or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and / or a network of server computers and / or personal computers.

[0191] FIG. 4 is a block diagram that illustrates an example computer system with which an embodiment may be implemented. In the example of FIG. 4, a computer system 400 and instructions for implementing the disclosed technologies in hardware, software, or a combination of hardware and software, are represented schematically, for example, as boxes and circles, at the same level of detail that is commonly used by persons of ordinary skill in the art to which this disclosure pertains for communicating about computer architecture and computer systems implementations.

[0192] Computer system 400 includes an input / output (I / O) subsystem 402, which may include a bus and / or other communication mechanism(s) for communicating information and / or instructions between the components of the computer system 400 over electronic signal paths. The I / O subsystem 402 may include an VO controller, a memory controller, and at least one I / O port. The electronic signal paths are represented schematically in the drawings, such as lines, unidirectional arrows, or bidirectional arrows.

[0193] At least one hardware processor 404 is coupled to the VO subsystem 402 for processing information and instructions. Hardware processor 404 may include, for example, a general- purpose microprocessor or microcontroller and / or a special-purpose microprocessor such as an embedded system or a graphics processing unit (GPU), or a digital signal processor or ARM ATTORNEY DOCKET NO.

[0194] 093459.0134

[0195] 26 processor. Processor 404 may comprise an integrated arithmetic logic unit (ALU) or be coupled to a separate ALU.

[0196] Computer system 400 includes one or more units of memory 406, such as a main memory, coupled to I / O subsystem 402 for electronically storing data and instructions to be executed by processor 404. Memory 406 may include volatile memory such as various forms of random-access memory (RAM) or other dynamic storage devices. Memory 406 may also be used for storing temporary variables or other intermediate information during the execution of instructions to be executed by processor 404. Such instructions, when stored in non-transitory computer-readable storage media accessible to processor 404, can render computer system 400 into a special-purpose machine customized to perform the operations specified in the instructions.

[0197] Computer system 400 includes non-volatile memory such as read-only memory (ROM) 408 or other static storage devices coupled to I / O subsystem 402 for storing information and instructions for processor 404. The ROM 408 may include various forms of programmable ROM (PROM), such as erasable PROM (EPROM) or electrically erasable PROM (EEPROM). A unit of persistent storage 410 may include various forms of non-volatile RAM (NVRAM), such as FLASH memory, solid-state storage, magnetic disk, or optical disks such as CD-ROM or DVD-ROM and may be coupled to I / O subsystem 402 for storing information and instructions. Storage 410 is an example of a non-transitory computer-readable medium that may be used to store instructions and data, which, when executed by the processor 404, cause performing computer-implemented methods to execute the techniques herein.

[0198] The instructions in memory 406, ROM 408, or storage 410 may comprise one or more instructions organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps. The instructions may comprise an operating system and / or system software; one or more libraries to support multimedia, programming, or other functions; data protocol instructions or stacks to implement TCP / IP, HTTP, or other communication protocols; file format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, internet access applications, design and ATTORNEY DOCKET NO. 093459.0134

[0199] 27 manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. The instructions may implement a web server, web application server, or web client. The instructions may be organized as a presentation, application, and data storage layer, such as a relational database system using a structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system, or other data storage.

[0200] Computer system 400 may be coupled via I / O subsystem 402 to at least one output device 412. In one embodiment, output device 412 is a digital computer display. Examples of a display that may be used in various embodiments include a touchscreen display, a lightemitting diode (LED) display, a liquid crystal display (LCD), or an e-paper display. Computer system 400 may include other types of output devices 412, alternatively or in addition to a display device. Examples of other output devices 412 include printers, ticket printers, plotters, projectors, sound cards or video cards, speakers, buzzers or piezoelectric devices or other audible devices, lamps or LED or LCD indicators, haptic devices, actuators or servos.

[0201] At least one input device 414 is coupled to the I / O subsystem 402 for communicating signals, data, command selections, or gestures to the processor 404. Examples of input devices 414 include touch screens, microphones, still and video digital cameras, alphanumeric and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and / or various types of sensors such as force sensors, motion sensors, heat sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors and / or various types of transceivers such as wireless, such as cellular or Wi-Fi, radio frequency (RF) or infrared (IR) transceivers and Global Positioning System (GPS) transceivers.

[0202] Another type of input device is a control device 416, which may perform cursor control or other automated control functions such as navigation in a graphical interface on a display screen, alternatively or in addition to input functions. The control device 416 may be a touchpad, a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processor 404 and for controlling cursor movement on an output device 412, such as a display. The input device may have at least two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. Another type of input device is a wired, wireless, or optical control device such as a joystick, wand, console, steering wheel, pedal, gearshift mechanism, or other ATTORNEY DOCKET NO.

[0203] 093459.0134

[0204] 28 control device. An input device 414 may include a combination of multiple input devices, such as a video camera and a depth sensor.

[0205] In another embodiment, computer system 400 may comprise an Internet of Things (loT) device in which one or more of the output device 412, input device 414, and control device 416 are omitted. Or, in such an embodiment, the input device 414 may comprise one or more cameras, motion detectors, thermometers, microphones, seismic detectors, other sensors or detectors, measurement devices or encoders, and the output device 412 may comprise a specialpurpose display such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator or a servo.

[0206] When computer system 400 is a mobile computing device, input device 414 may comprise a global positioning system (GPS) receiver coupled to a GPS module that is capable of triangulating to a plurality of GPS satellites, determining and generating geo-location or position data such as latitude-longitude values for a geophysical location of the computer system 400. Output device 412 may include hardware, software, firmware, and interfaces for generating position reporting packets, notifications, pulse or heartbeat signals, or other recurring data transmissions that specify a position of the computer system 400, alone or in combination with other application-specific data, directed toward host computer 424 or server computer 430.

[0207] Computer system 400 may implement the techniques described herein using customized hard-wired logic, at least one ASIC or FPGA, firmware, and / or program instructions or logic which, when loaded and used or executed in combination with the computer system, cause or program the computer system to operate as a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 400 in response to processor 404 executing at least one sequence of at least one instruction contained in main memory 406. Such instructions may be read into main memory 406 from another storage medium, such as storage 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0208] The term “storage media,” as used herein, refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such storage ATTORNEY DOCKET NO. 093459.0134

[0209] 29 media may comprise non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage 410. Volatile media includes dynamic memory, such as memory 406. Common forms of storage media include, for example, a hard disk, solid state drive, flash drive, magnetic data storage medium, any optical or physical data storage medium, memory chip, or the like.

[0210] Storage media is distinct but may be used with transmission media. Transmission media participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, and wires comprising a bus of the I / O subsystem 402. Transmission media can also be acoustic or light waves generated during radio-wave and infrared data communications.

[0211] Various forms of media may carry at least one sequence of at least one instruction to processor 404 for execution. For example, the instructions may initially be carried on a remote computer's magnetic disk or solid-state drive. The remote computer can load the instructions into its dynamic memory and send them over a communication link such as a fiber optic, coaxial cable, or telephone line using a modem. A modem or router local to computer system 400 can receive the data on the communication link and convert the data to a format that can be read by computer system 400. For instance, a receiver such as a radio frequency antenna or an infrared detector can receive the data carried in a wireless or optical signal, and appropriate circuitry can provide the data to the VO subsystem 402, such as placing the data on a bus. VO subsystem 402 carries the data to memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by memory 406 may optionally be stored on storage 410 either before or after execution by processor 404.

[0212] Computer system 400 also includes a communication interface 418 coupled to a bus or I / O subsystem 702. Communication interface 418 provides a two-way data communication coupling to a network link(s) 420 directly or indirectly connected to at least one communication network, such as a network 422 or a public or private cloud on the Internet. For example, communication interface 418 may be an Ethernet networking interface, integrated-services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of communications line, for example, an Ethernet cable, a metal cable of any kind, a fiber-optic line or a telephone line. Network 422 broadly represents a local area network (LAN), wide-area network (WAN), campus network, ATTORNEY DOCKET NO. 093459.0134

[0213] 30 internetwork, or any combination thereof. Communication interface 418 may comprise a LAN card to provide a data communication connection to a compatible LAN, a cellular radiotelephone interface that is wired to send or receive cellular data according to cellular radiotelephone wireless networking standards, or a satellite radio interface that is wired to send or receive digital data according to satellite wireless networking standards. In any such implementation, communication interface 418 sends and receives electrical, electromagnetic, or optical signals over signal paths that carry digital data streams representing various types of information.

[0214] Network link 420 typically provides electrical, electromagnetic, or optical data communication directly or through at least one network to other data devices, using, for example, satellite, cellular, Wi-Fi, or BLUETOOTH technology. For example, network link 420 may connect through network 422 to a host computer 424.

[0215] Furthermore, network link 420 may connect through network 422 or to other computing devices via internetworking devices and / or computers operated by an Internet Service Provider (ISP) 426. ISP 426 provides data communication services through a worldwide packet data communication network called the Internet 428. A server computer 430 may be coupled to the Internet 428. Server computer 430 broadly represents any computer, data center, virtual machine, or virtual computing instance with or without a hypervisor, or computer executing a containerized program system such as DOCKER or KUBERNETES. Server computer 430 may represent an electronic digital service that is implemented using more than one computer or instance and that is accessed and used by transmitting web service requests, uniform resource locator (URL) strings with parameters in HTTP payloads, API calls, app services calls, or other service calls. Computer system 400 and server computer 430 may form elements of a distributed computing system that includes other computers, a processing cluster, a server farm, or other organizations of computers that cooperate to perform tasks or execute applications or services. Server computer 430 may comprise one or more instructions organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs, including mobile apps. The instructions may comprise an operating system and / or system software; one or more libraries to support multimedia, programming, or other functions; data protocol instructions or stacks to implement TCP / IP, HTTP, or other communication protocols; file ATTORNEY DOCKET NO. 093459.0134

[0216] 31 format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. Server computer 430 may comprise a web application server that hosts a presentation layer, application layer, and data storage layer, such as a relational database system using a structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system or other data storage.

[0217] Computer system 400 can send messages and receive data and instructions, including program code, through the network(s), network link 420, and communication interface 418. In the Internet example, server computer 430 might transmit a requested code for an application program through Internet 428, ISP 426, local network 422, and communication interface 418. The received code may be executed by processor 404 as it is received and / or stored in storage 410 or other non-volatile storage for later execution.

[0218] The execution of instructions, as described in this section, may implement a process in the form of an instance of a computer program that is being executed and consists of program code and its current activity. Depending on the operating system (OS), a process may be made up of multiple threads of execution that execute instructions concurrently. In this context, a computer program is a passive collection of instructions, while a process may be the actual execution of those instructions. Several processes may be associated with the same program; for example, opening up several instances of the same program often means more than one process is being executed. Multitasking may be implemented to allow multiple processes to share the processor 404. While each processor 404 or core of the processor executes a single task at a time, the computer system 400 may be programmed to implement multitasking to allow each processor to switch between tasks that are being executed without having to wait for each task to finish. In an embodiment, switches may be performed when tasks perform input / output operations when a task indicates that it can be switched or on hardware interrupts. Time-sharing may be implemented to allow fast response for interactive user applications by rapidly performing context switches to provide the appearance of concurrent execution of multiple processes. In an embodiment, for security and reliability, an operating system may ATTORNEY DOCKET NO.

[0219] 093459.0134

[0220] 32 prevent direct communication between independent processes, providing strictly mediated and controlled inter-process communication functionality.

[0221] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.

Claims

ATTORNEY DOCKET NO. 093459.013433CLAIMSWhat is claimed is:

1. A computer-implemented generative artificial intelligence system programmed for generating qualitative reports and comprising: a network of a plurality of specialized program agents organized in a hierarchical directed acyclic graph (DAG) structure in a memory of a first computing device; a main agent executed using the first computing device and programmed to execute receiving a complex research task, decomposing the task into subtasks, automatically instantiating a plurality of sub-agents corresponding to the subtasks, and delegating the subtasks to the sub-agents; wherein each agent of the plurality of sub-agents is programmed to execute: receiving a subtask; determining an optimal course of action using an action generator; executing actions to complete the subtask; and propagate output results back up a hierarchy of the DAG; an evaluator executed using the first computing device and programmed to assess the output results of the agents; and a synthesizer executed using the first computing device and programmed to compile the output results of the sub-agents into a final report and to cause presenting electronic digital text corresponding to the final report using an output device coupled to a second computing device.

2. The computer-implemented generative artificial intelligence system of claim 1, wherein the action generator is programmed to evaluate available resources and determine the optimal course of action to complete an assigned task.

3. The computer-implemented generative artificial intelligence system of claim 1, wherein the actions executed by the sub-agents include at least one of: web searches, querying internal databases, code execution, and further task decomposition.ATTORNEY DOCKET NO. 093459.0134344. The computer-implemented generative artificial intelligence system of claim 1, wherein the evaluator is an independent agent configured to determine whether a task was successfully completed, failed, or requires further refinement.

5. The computer-implemented generative artificial intelligence system of claim 1, wherein the final report includes relevant information, analysis, and source links.

6. The computer-implemented generative artificial intelligence system of claim 1, wherein the sub-agents are configured to retrieve data from a variety of sources, including proprietary data sources, PDF parsing capabilities, web search modules, and code execution capabilities.

7. The computer-implemented generative artificial intelligence system of claim 1, further comprising a system prompt that provides instructions to guide the agents in their research.

8. The computer-implemented generative artificial intelligence system of claim 7, wherein the system prompt is tailored for specific tasks or agents.

9. The computer-implemented generative artificial intelligence system of claim 1, wherein the final report is returned in a structured format.

10. The computer-implemented generative artificial intelligence system of claim 9, wherein the structured format is a JSON object.

11. The computer-implemented generative artificial intelligence system of claim 1, wherein the system is configured to incorporate data from PDF documents.

12. The computer-implemented generative artificial intelligence system of claim 1, further comprising a fact validator pipeline configured to independently validate assertions made in the final report.ATTORNEY DOCKET NO. 093459.01343513. A computer-implemented method for generating qualitative reports using generative artificial intelligence, comprising: executing using a first computing device: receiving a complex research task at a main agent; decomposing the task into subtasks using the main agent; automatically instantiating a plurality of sub-agents; delegating the subtasks to a plurality of sub-agents organized in a hierarchical directed acyclic graph (DAG) structure; each sub-agent executing: determining an optimal course of action using an action generator; executing actions to complete the assigned subtask; propagating results up a hierarchy of the DAG; evaluating outputs of the agents using an evaluator; synthesizing results from the sub-agents into a final report; and transmitting electronic digital text corresponding to the final report to a second computing device.

14. The method of claim 13, wherein executing actions includes at least one of: performing web searches, querying internal databases, executing code, and further decomposing tasks.

15. The method of claim 13, wherein evaluating outputs includes determining whether a task was successfully completed, failed, or requires further refinement.

16. The method of claim 13, further comprising providing a system prompt to guide the agents in their research.

17. The method of claim 13, wherein synthesizing results includes compiling information, analysis, and source links into a structured format.ATTORNEY DOCKET NO.093459.01343618. The method of claim 13, further comprising parsing PDFs to extract structured data.

19. The method of claim 13, further comprising validating assertions made in the final report using a fact validator pipeline.

20. The method of claim 13, wherein the complex research task is related to generating a thematic investment portfolio.

Images