Multi-agent dynamic arrangement method based on multi-modal analysis and adaptive retrieval

Abstract

The invention discloses a multi-agent dynamic arrangement method based on multi-modal analysis and adaptive retrieval, and relates to the technical field of artificial intelligence and information retrieval. Comprising the steps of S1, converting a text, an image, structured data and voice content input by a user into a unified multi-mode semantic representation, S2, converting the unified multi-mode semantic representation into a specific execution process, and S3, automatically scheduling a reasoning agent, a knowledge obtaining agent and an execution agent according to DAG nodes, task elements and available resources, and obtaining the task elements and the execution agent according to the reasoning agent, the knowledge obtaining agent and the execution agent. S4, after task process construction and agent arrangement are completed, dynamic retrieval, evidence convergence and strategy optimization are carried out on information requirements related to a user task, so that a reasoning agent obtains complete knowledge support with consistent context, and S5, knowledge evidence is combined with a task process, so that the task process is completed. The method comprises the following steps: step S6, implementing problem solving, strategy generation and task closed-loop execution through a reasoning agent, step S6, performing actual operation on a target task by an execution agent according to an executable instruction sequence output by the reasoning agent, and outputting a result, and step S7, performing result verification according to an output result returned by the execution agent, and the correctness, integrity and consistency of an output result are examined through rule verification, model evaluation and evidence alignment.

Description

Technical Field

[0001] This invention discloses a multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval, which relates to the fields of artificial intelligence and information retrieval technology. Background Technology

[0002] With the widespread application of large-scale language models (LLM), graph-structured knowledge bases, and multimodal understanding technologies, agent-based approaches to handling complex tasks are gradually becoming mainstream. However, existing multi-agent systems still have significant shortcomings: agents lack dynamic collaboration and flexible orchestration mechanisms, typically relying on pre-defined processes and failing to automatically select the optimal agent combination based on task type and contextual changes; retrieval methods are limited, lacking unified scheduling of multi-source information such as enterprise knowledge bases, internal documents, online searches, and local memory; in multimodal task scenarios involving mixed text and graph data, and structured data, the system lacks unified semantic parsing capabilities, making it difficult to construct stable inference chains; furthermore, inference results often remain at the text level, failing to automatically generate executable operations, leading to a disconnect between inference and execution. Moreover, the retrieval depth and strategies required for complex tasks are difficult to meet through static methods, affecting overall accuracy and real-time performance. Summary of the Invention

[0003] This invention addresses the problems of existing technologies by providing a multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval. In task processing, this method automatically identifies task structure, integrates multi-source knowledge, generates executable actions, and performs real-time optimization to improve the efficiency and accuracy of handling complex tasks. It is particularly suitable for application in intelligent office scenarios such as contract review, policy Q&A, knowledge graph querying, and approval process generation.

[0004] The specific solution proposed in this invention is as follows: This invention provides a multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval, comprising: Step S1: Convert the user-input text, images, structured data, and speech content into a unified multimodal semantic representation for subsequent knowledge retrieval, inference scheduling, and agent collaboration processes. This involves preprocessing the user-input text, images, structured data, and speech content, followed by multimodal data structure alignment, multimodal data encoding, and finally multimodal semantic fusion to obtain the unified multimodal semantic representation. Step S2: Transform the unified multimodal semantic representation into a specific execution flow: perform fine-grained classification of user intent, extract task elements, and match and generate candidate flow sequences in multiple flow orchestration node libraries. Based on the candidate flow sequences, construct a directed acyclic graph (DAG) that satisfies dependencies and constraints, and perform complexity evaluation and inject alternative paths into the DAG for subsequent agent orchestration and execution scheduling. Step S3: Based on DAG nodes, task elements, and available resources, automatically schedule the reasoning agent, knowledge acquisition agent, and execution agent to achieve the optimal collaboration scheme. Generate an agent combination A={ based on the main objective, task requirements, node priorities, and resource status. ,..., It dynamically allocates task loads, thereby improving the efficiency of multi-agent collaboration and task completion rate. Step S4: After completing the task flow construction and agent orchestration, dynamically retrieve, aggregate evidence, and optimize strategies for the user's task-related information needs, enabling the reasoning agent to obtain complete and context-consistent knowledge support. Step S5: Integrate knowledge evidence with the task flow, and use an inference agent to solve problems, generate strategies, and execute tasks in a closed loop. Through inference chain construction, logic verification, result generation, and execution feedback, an interpretable and verifiable inference path is formed in a multimodal input scenario, and finally, an executable result or operation command that meets the user's needs is output. Step S6: The executing agent performs the actual operation on the target task according to the executable instruction sequence output by the inference agent, and outputs the result. Step S7: Based on the output returned by the executing agent, verify the results by reviewing the correctness, completeness and consistency of the output results through rule verification, model evaluation and evidence alignment.

[0005] Furthermore, step S1 of the multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval specifically includes: S11: Preprocess the user-input text, images, structured data, and voice content, including: Image preprocessing is performed: image denoising, distortion correction, and brightness enhancement are carried out, and the text information and its corresponding spatial coordinates in the image are extracted using the PaddleOCR-VL model. Document structure analysis was performed: the PaddleOCR-VL model was used to classify and group document regions, forming a hierarchical structure: title, paragraph, heading, image region, table region, page number, footnote, and citation. Speech preprocessing is performed: speech segments are separated using endpoint detection, and then the speech is transcribed into text using an ASR (Qwen3-Audio) model. The preprocessed data is transformed into preliminary semantic fragments.

[0006] S12: Perform multimodal data structure alignment, including: after obtaining preliminary semantic fragments, perform structural alignment on heterogeneous information from text, images, and speech, including establishing a document hierarchical tree structure based on document layout analysis results. The text lines and block elements obtained from OCR are mapped to the corresponding structure nodes, and the image region and table region are decomposed into content units. The speech-to-text is structured into a sequence structure according to the timestamps of the speech segments. By unifying node types and linking across modal references, a unified structural alignment graph is constructed:

[0007] in Represents a preliminary semantic fragment. Align(·) represents the structural alignment function between cross-modal nodes, ensuring that the same semantic unit has a one-to-one or one-to-many mapping relationship in different modalities. S13: Perform multimodal data encoding: This involves aligning the representation... Encoding is performed by calling a text encoder, a vision-document joint encoder, and a speech encoder to vectorize the initial semantic fragments of each modality, thereby obtaining text features. Visual / Layout Features With speech features And form a cross-modal correlation-enhanced coding representation. The formula is as follows:

[0008] S14: Perform multimodal semantic fusion: Map all modalities to a shared semantic space, and assign dynamic weights to different modalities through a gated fusion unit. Calculate the global fusion representation: This forms a unified semantic representation vector: .

[0009] Furthermore, step S2 of the multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval specifically includes: S21: Extract task elements: Analyze the key objectives, constraints and available resources in the input to form a task element triple T=(G,C,R) for subsequent process matching. The objective G describes the user's core needs, the constraint C describes the limiting information in the task execution process, and the available resources R include the currently callable intelligent agent capabilities, tool nodes, knowledge sources and external interfaces. S22: Generate candidate nodes: Based on the generated task element triple T=(G,C,R), perform candidate matching on each process node in the preset process node library. The preset library contains m predefined process orchestration nodes N={ ,..., }, Calculate the node semantic vector for each node. With task vector The similarity, combined with the hard rule constraints of the nodes. The matching scores are weighted to generate a candidate matching score for each node: , cos(·) represents the cosine similarity; rule_match∈{0,1} indicates a hard rule match, which is 1 if the task element meets the mandatory prerequisites of the node; λ∈[0,1] is the similarity and rule weight balancing parameter. Nodes that score above a preset threshold will be selected into the candidate node set. And sort them by matching score to provide a preliminary node sequence for subsequent process construction; S23: Based on the candidate node set A directed acyclic graph (DAG) is constructed to represent the execution order and data dependencies between nodes. This involves analyzing the input-output dependencies, resource requirements, and permission constraints of each node, and establishing directed edges E based on the preconditions between nodes, forming a complete DAG = (V, E), where V⊆ Let E be a set of nodes, and let E represent the execution dependencies between nodes; S24: Calculate the execution priority p(n) of each candidate node based on the matching score, execution cost and historical stability, and configure backup paths B(n) for nodes that may fail to execute or the tool is unavailable.

[0010] Furthermore, step S3 of the multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval specifically includes: S31: Agent selection and capability matching: Based on the functional requirements, input / output types, and required tool invocation capabilities of each node, automatically select the most suitable agent type from the agent capability library to execute the node. This involves first describing the node's task. Semantic encoding is performed and matched with the capability vector Aj of each agent to calculate the capability fit score. , Where E(⋅) represents the deep semantic representation of capabilities and needs, the matching process comprehensively considers the agent's cognitive ability, knowledge ability, and execution ability, and distinguishes reasoning agents, knowledge agents, and execution agents based on ability differences to ensure that each node is executed by the agent with the most advantages. S32: Assign roles and bind tasks to agents: Bind specific agents to corresponding nodes according to the topological order of the process DAG, and assign clear roles and responsibilities to each agent. First, classify the nodes into inference nodes, knowledge acquisition nodes, and execution nodes according to their functional categories, and match the node requirements with the agent type to achieve role-level binding. Then, configure the task context for the agent, including the input data entry point, the knowledge sources that can be accessed, the set of tool nodes that can be called, and the output format requirements, so as to ensure that the agent can strictly follow the expected behavior of the node during execution. S33: Agent Interaction and Collaborative Control: After the multi-agent orchestration is completed, interaction and collaborative control between agents are realized based on a unified communication protocol and a global context management mechanism. When each agent is scheduled to execute the corresponding node task, it sends the task result or intermediate inference information to the communication center through a message channel. The output of each agent is structured and encapsulated and stored in the global context. It determines the range of data that subsequent nodes can access based on node dependencies and data read / write strategies.

[0011] Furthermore, step S4 of the multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval specifically includes: S41: Retrieval Intent Identification and Query Structuring: Before entering the knowledge retrieval stage, the user's task intent is classified, decomposed, and structured using a deep semantic analysis model to generate a standardized retrieval request vector. First, multimodal semantic analysis is performed on the user's input text, image fragments, document structural elements, and speech-to-text content to extract core information requirement elements relevant to the retrieval, including entities, attributes, relations, time windows, and constraints. Then, a suitable query template is determined based on the task type, and a structured query expression is constructed. Used for subsequent knowledge source selection and retrieval strategy generation. S42: Perform multi-source knowledge selection and dynamic path planning: After obtaining the structured query expression, the suitability of different knowledge sources for the current query is determined through a knowledge source evaluation mechanism, and the optimal retrieval path is dynamically planned. The first step is to calculate the matching degree between the query and each knowledge source. in This represents the coverage, update frequency, reliability, and access cost of a knowledge source; λ is a balancing factor.

[0012] The primary search source is selected based on the scores, and backup search sources are configured for key knowledge points to ensure automatic path switching in case of missing information, access failure, or response delay exceeding a threshold. Then, a dynamic programming approach is used to generate a Directed Acyclic Graph (DAG) of search paths. S43: Perform semantically enhanced retrieval and evidence aggregation: After determining the retrieval path, semantic enhancement is performed on the retrieval statement based on the structured query results. This includes synonym expansion, logical completion, time inference, context completion, and multi-hop query rewriting. The enhanced query statement is represented as follows: in To provide a global context, the enhancement process integrates the reasoning results of preceding agents with user intent. Subsequently, multi-level retrieval is performed based on selected knowledge sources, including keyword retrieval, vector semantic retrieval, graph database retrieval, and cross-source collaborative retrieval. The retrieval results are then represented as a set of evidence. , The evidence set is subjected to quality assessment, confidence calculation, temporal consistency alignment, and multi-source redundancy denoising, and a unified evidence representation is generated through a weighted fusion strategy. To ensure the credibility, comprehensiveness, and consistency of the output content, S44: Perform adaptive retrieval strategy updates and feedback reinforcement: After the retrieval is completed, the retrieval strategy is adaptively optimized based on the task execution results, user feedback signals, and the judgment results of the agent inference nodes, automatically improving the retrieval quality and task success rate during continuous operation.

[0013] Furthermore, step S5 of the multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval specifically includes: S51: Inference Chain Construction and Context Fusion: At the start of inference execution, the global context generated by the preceding steps is first used as the basis for the inference chain construction. , Retrieve evidence set And the input requirements of the inference nodes in the process DAG, to construct the initial inference chain structure. S52: Reasoning Expansion and Logic Verification: After the reasoning chain is constructed, the reasoning expansion and logic verification process begins. Intermediate reasoning steps are generated through step-by-step deduction, and a logic consistency check is performed at each step. S53: Generate Reasoning Results: When the reasoning chain reaches the endpoint condition, automatically generate a structured or natural language reasoning result based on the task type. Simultaneously, generate interpretable metadata for the reasoning result, including evidence citation chains, reasoning path summaries, conflict branch analysis, etc., to facilitate user tracing and auditing. S54: Execution Feedback and Result Verification: For execution-type tasks, the agent performs operations according to the action sequence and verifies the completion status as a feedback vector. Return to the system; for reasoning tasks, compare the final answer with the retrieved evidence, user expectations, and the terminal state of the process DAG to assess its rationality and sufficiency.

[0014] Furthermore, step S6 of the multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval specifically includes: firstly, performing structured parsing on the operational intent, action parameters, and associated objects in the inference results to generate standardized execution instruction templates. Subsequently, the executing agent selects the appropriate execution capability module based on the instruction type and executes the corresponding actions sequentially or in parallel according to the dependencies of the task nodes. During execution, the execution status is monitored in real time through status monitoring and exception capture mechanisms. It automatically judges and recovers from possible situations such as insufficient permissions, data conflicts, page errors, or execution failures, and triggers retry logic or reschedules process nodes when necessary. After completing all actions, the executing agent will uniformly transmit the execution log, output data, and status information back for result verification and closed-loop optimization, realizing the automated implementation from reasoning and decision-making to actual operation.

[0015] Furthermore, step S7 of the multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval specifically includes: first, selecting the corresponding verification strategy according to the task type, including fact verification, business rule verification, temporal consistency check, multimodal content comparison and format specification detection; Secondly, for tasks where the reasoning results involve external evidence, cross-comparison is performed using the retrieved evidence set or knowledge base content to identify potential biases, omissions, or semantic conflicts; when a problem is found, a backtracking mechanism is automatically triggered to initiate a re-reasoning or supplementary recall request to the reasoning agent or retrieval agent, thereby making secondary corrections to the results; Finally, the verified content is structured, formatted, optimized, styled, or visualized, and the final result that meets the task requirements is output. Meanwhile, the feedback data generated throughout the result verification process will be used to update the policy parameters and task graph nodes, so as to realize the continuous optimization and adaptive evolution of the agent collaboration link.

[0016] This invention also provides a multi-agent dynamic orchestration system based on multimodal parsing and adaptive retrieval, characterized by including a semantic conversion module, a process conversion module, a scheduling module, an execution module, an output module, and a verification module. The semantic transformation module converts user-input text, images, structured data, and speech content into a unified multimodal semantic representation for subsequent knowledge retrieval, inference scheduling, and agent collaboration processes. This process involves preprocessing the user-input text, images, structured data, and speech content, followed by multimodal data structure alignment, multimodal data encoding, and finally multimodal semantic fusion to obtain the unified multimodal semantic representation. The process transformation module converts the unified multimodal semantic representation into a specific execution flow: it performs fine-grained classification of user intent, extracts task elements, matches and generates candidate flow sequences in multiple flow orchestration node libraries, constructs a directed acyclic graph (DAG) that satisfies dependencies and constraints based on the candidate flow sequences, and evaluates the complexity of the DAG and injects alternative paths for subsequent agent orchestration and execution scheduling. The scheduling module automatically schedules inference agents, knowledge acquisition agents, and execution agents based on DAG nodes, task elements, and available resources to achieve the optimal collaboration scheme. Based on the main objectives, it generates an agent combination A={ ,..., It dynamically allocates task loads, thereby improving the efficiency of multi-agent collaboration and task completion rate. After completing the task flow construction and agent orchestration, the system dynamically retrieves, aggregates evidence, and optimizes strategies to meet the user's task-related information needs, enabling the reasoning agent to obtain complete and context-consistent knowledge support. The execution module combines knowledge evidence with task flow, using an inference agent to solve problems, generate strategies, and execute tasks in a closed loop. Through inference chain construction, logic verification, result generation, and execution feedback, it forms interpretable and verifiable inference paths in multimodal input scenarios, and ultimately outputs executable results or operation instructions that meet user needs. The output module outputs the results after the executing agent performs the actual operation on the target task based on the executable instruction sequence output by the inference agent. The verification module verifies the results based on the output returned by the executing agent, and reviews the correctness, completeness and consistency of the output results through rule verification, model evaluation and evidence alignment.

[0017] The advantages of this invention are: This invention can automatically identify task structure, integrate multi-source knowledge, generate executable actions, and perform real-time optimization during task processing to improve the efficiency and accuracy of handling complex tasks. Especially in intelligent office scenarios such as contract review, policy Q&A, knowledge graph query, and approval process generation, it can significantly improve the intelligence and execution efficiency of complex task processing. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention. Example

[0020] This invention provides a multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval, comprising: Step S1: Convert the user-input text, images, structured data, and speech content into a unified multimodal semantic representation for subsequent knowledge retrieval, inference scheduling, and agent collaboration processes. This involves preprocessing the user-input text, images, structured data, and speech content, followed by multimodal data structure alignment, multimodal data encoding, and finally multimodal semantic fusion to obtain a unified multimodal semantic representation.

[0021] Specifically, it may include: S11: Preprocessing of the multimodal raw input: Receive user input data: image data Text / document data With voice data Each module then calls its corresponding preprocessing module to remove noise and obtain preliminary semantic fragments. This may include: Image preprocessing: Perform image denoising, distortion correction, brightness enhancement, etc., and extract text information and its corresponding spatial coordinates from the image using the PaddleOCR-VL model.

[0022] Document layout structure analysis: The PaddleOCR-VL model is used to classify and group document regions to form a hierarchical structure: title, paragraph, heading, image region, table region, page number, footnote, citation, etc.

[0023] Speech preprocessing: Speech segments are separated using endpoint detection, and then the speech is transcribed into text using the ASR (Qwen3-Audio) model.

[0024] All types of preprocessed data are converted into preliminary semantic fragments. , where m∈{t,i,v}. Its formal representation is as follows: =

[0025] For example, when a user uploads a meeting minutes document and supplementary audio descriptions, the meeting text is extracted through document layout structure analysis, and the audio content is obtained through ASR, thus forming two preliminary semantic segments for the next stage of processing.

[0026] S12: Perform multimodal structure alignment: After obtaining preliminary semantic fragments Subsequently, heterogeneous information from text, images, and speech is structurally aligned to achieve a unified organization, facilitating subsequent encoding and retrieval. This process includes three types of operations: hierarchical structure alignment, temporal and spatial alignment, and semantic span unification.

[0027] First, a hierarchical tree structure is established based on the document layout analysis results, such as headings, paragraphs, tables, and image areas. The OCR-derived text lines and block elements are mapped to corresponding structure nodes. Image and table regions are further broken down into content units, such as table cells, objects detected in images, or caption text. Simultaneously, the speech-to-text is structured into a sequence based on the timestamps of speech segments. By unifying node types and using cross-modal reference links, a unified structural alignment graph is constructed:

[0028] Align(·) represents the structural alignment function between cross-modal nodes, ensuring that the same semantic unit has a one-to-one or one-to-many mapping relationship in different modalities.

[0029] For example, when a user uploads an annual performance review report PDF and adds a voice explanation that "the data in Part Two comes from the financial system," the system automatically binds "Part Two" in the voice to the corresponding title node in the document's layout structure, achieving cross-modal structural alignment between the voice and the document content.

[0030] S13: Perform multimodal data encoding: After completing the structural alignment, the aligned representation will be... The input is a multimodal encoding module to obtain deep semantic features from text, images / documents, and speech. Specifically, the text encoder, the visual-document joint encoder, and the speech encoder are invoked respectively to vectorize the initial semantic fragments of each modality, thereby obtaining text features. Visual / Layout Features With speech features The text encoder is responsible for extracting linguistic semantics; the visual-document co-encoder utilizes page layout, OCR text, and visual region information to achieve visual semantic modeling of the document; and the speech encoder obtains a deep semantic representation of the speech content through an acoustic-language dual-channel network. To enhance consistency between modalities, a cross-modal attention mechanism is further adopted, enabling the features of each modality to incorporate contextual information from other modalities during computation, thereby forming a cross-modal association-enhanced encoded representation. .

[0031]

[0032] The encoding result not only has the ability to express semantics within a modality, but also the ability to understand cross-modal associations, providing a high-quality vector basis representation for subsequent semantic fusion, knowledge retrieval and reasoning scheduling.

[0033] S14: Perform multimodal semantic fusion: In obtaining multimodal deep features Then, all modalities are uniformly mapped to a shared semantic space to form a global multimodal representation Z that can support retrieval, reasoning, and multi-agent scheduling.

[0034] First, dynamic weights are assigned to different modalities using a gated fusion unit:

[0035] Then calculate the global fused representation: This ultimately forms a unified semantic representation vector that can be directly used by downstream modules: .

[0036] For example, when a user uploads "a scanned copy of meeting minutes + supplementary audio descriptions + a screenshot of a flowchart", the Z vector output by multimodal fusion is automatically captured: Key paragraphs in document titles and page layout; The process nodes and arrow relationships in the image. The audio recording included supplementary information such as "Budget matters were not discussed at this meeting." This results in a highly consistent global semantic representation, which can be used for subsequent knowledge retrieval, task decomposition, and agent collaboration.

[0037] Step S2: Transform the unified multimodal semantic representation into a specific execution flow: perform fine-grained classification of user intent, extract task elements, match and generate candidate flow sequences in multiple flow orchestration node libraries, construct a directed acyclic graph (DAG) that satisfies dependencies and constraints based on the candidate flow sequences, and perform complexity evaluation and inject backup paths into the DAG for subsequent agent orchestration and execution scheduling.

[0038] Specifically, it may include: S21: Task Element Extraction After obtaining a unified multimodal semantic representation of user input, the process first enters the task element extraction stage. Through intent analysis models and structured parsing rules, key objectives, constraints, and available resources in the input are analyzed to form a task element triple T=(G,C,R) for subsequent process matching. Here, objective G describes the user's core needs, including task types such as document generation, information retrieval, knowledge reasoning, risk analysis, or tool invocation; constraints C describes limitations during task execution, such as time span, permission boundaries, regulatory requirements, or data usage scope; and available resources R include currently available agent capabilities, tool nodes, knowledge sources, and external interfaces. Through methods such as semantic role labeling, intent slot parsing, and context-based collaborative reasoning, these three types of elements are automatically extracted from the multimodal representation, ultimately generating a structured task description that provides a clear target direction and execution boundaries for process node matching.

[0039] For example, when a user enters "Help me generate a summary of key points based on this meeting minutes and interpret the chart data", the system automatically identifies "summary of key points" and "interpretation of charts" as the main task objectives, uses "meeting minutes content" and "chart area" as available resources, and identifies potential constraints on time or expression style in combination with the context, thereby constructing a complete task element triplet.

[0040] S22: Generate candidate nodes: After task element extraction is completed, candidate matching is performed on each process node in the predefined process node library based on the generated task element triple T=(G,C,R). A predefined library of m process orchestration nodes N={ ,..., Examples of nodes include: intent recognition, parameter filling, multi-source retrieval, evidence merging, logical judgment, risk scoring, result verification, report generation, tool invocation, and manual review.

[0041] First, calculate the semantic vector for each node. With task vector The similarity, combined with the hard rule constraints of the nodes. The matching scores are weighted to generate a candidate matching score for each node:

[0042] Where cos(·) is the cosine similarity; rule_match∈{0,1} indicates a hard rule match (1 if the task element meets the mandatory prerequisite of the node); λ∈[0,1] is the similarity and rule weight balance parameter.

[0043] Nodes that score above a preset threshold will be selected into the candidate node set. The nodes are then sorted by matching scores to provide an initial sequence of nodes for subsequent process construction. This process considers both the consistency between nodes and task objectives and constraints, as well as the adaptability of nodes to available resources, ensuring that candidate nodes meet task requirements in terms of functional coverage, execution feasibility, and resource utilization.

[0044] For example, for the task of "contract risk identification and modification suggestions", nodes such as "multi-source knowledge retrieval", "evidence merging", "logical judgment", "risk scoring" and "result generation" are selected into the candidate set to form an executable basic process sequence.

[0045] S23: DAG Construction Process: After the node candidates are generated, based on the candidate node set A directed acyclic graph (DAG) is constructed to represent the execution order and data dependencies between nodes. Specifically, the input-output dependencies, resource requirements, and permission constraints of each node are analyzed, and directed edges E are established based on the preconditions between nodes, forming a complete DAG=(V,E), where V⊆ Let E be a set of nodes, representing the execution dependencies between nodes. During the construction process, constraints such as node permissions, time windows, and resource availability are injected, and backup nodes are set up for nodes that may fail or become unavailable, thereby ensuring the robustness and sustainable execution of the process. After the DAG is constructed, a validity check is performed to ensure that there are no circular dependencies, and the process complexity, such as the sum of node execution costs and edge communication overhead, is calculated to trigger process simplification or parallel optimization strategies when necessary.

[0046] For example, when processing the "Contract Risk Analysis" task, the "Multi-Source Knowledge Retrieval" node in the DAG must be executed before the "Evidence Merging" and "Risk Scoring" nodes. At the same time, if the online retrieval tool is unavailable, it automatically switches to the local knowledge base retrieval node and maintains the integrity of the DAG to ensure that the task can be executed continuously.

[0047] S24: Node Priority Ranking and Backup Path Setting: After the DAG of the process is constructed, the execution priority p(n) of each candidate node is calculated based on the matching score, execution cost, and historical stability. Backup paths B(n) are configured for nodes that may fail or the tool may become unavailable. The priority calculation comprehensively considers the semantic matching degree between the node and the task objective, the node's execution resource consumption, and the historical success rate to ensure that critical nodes are executed first, while improving the overall efficiency and reliability of the process. The formula is as follows:

[0048] Where: stability(n) represents the historical availability (0–1) of the tool / agent corresponding to the node; These are the weighting coefficients (normalized).

[0049] Backup paths include functionally equivalent replacement nodes and degraded execution schemes, enabling automatic switching to backup schemes to continue task execution without interrupting the entire process when the primary node becomes unavailable.

[0050] For example, in the "Contract Review and Risk Identification" task, if the online retrieval node cannot be executed due to rate limitations or network unavailability, the system automatically switches to the local knowledge base retrieval node and inserts a manual review step, thereby ensuring the continuity and integrity of the task.

[0051] Step S3: Based on DAG nodes, task elements, and available resources, automatically schedule the reasoning agent, knowledge acquisition agent, and execution agent to achieve the optimal collaboration scheme. Generate an agent combination A={ based on the main objective, task requirements, node priorities, and resource status. ,..., It dynamically allocates task loads, thereby improving the efficiency of multi-agent collaboration and task completion rate.

[0052] S31: Agent Selection and Capability Matching Once the process nodes are determined, the most suitable agent type for executing the node is automatically selected from the agent capability library based on the functional requirements, input / output types, and required tool invocation capabilities of each node. First, the node task is described. Semantic encoding is performed and matched with the capability vectors AjA_jAj of each agent to calculate the capability fit score. , Here, E(⋅) represents the deep semantic representation of capabilities and requirements. The matching process comprehensively considers the agent's cognitive capabilities (such as logical reasoning and planning), knowledge capabilities (such as network retrieval, local knowledge base querying, and memory access), and execution capabilities (such as code execution, web scraping, and environment invocation). Based on capability differences, it distinguishes between reasoning-oriented agents, knowledge-oriented agents, and execution-oriented agents to ensure that each node is executed by the most advantageous agent. Simultaneously, considering real-time load, response latency, resource consumption, and the agent's historical success rate, it selects the optimal agent for each node and generates a candidate agent set, ensuring a balance between performance, accuracy, and resource efficiency throughout the entire task process.

[0053] S32: Agent role assignment and task binding: After selecting the agents, specific agents are bound to corresponding nodes according to the topological order of the process DAG, and each agent is assigned a clear role and responsibility. During the binding process, nodes are first categorized into types such as "reasoning nodes," "knowledge acquisition nodes," and "execution nodes" based on their functional categories, and node requirements are mapped one-to-one with agent types to achieve role-level binding. Subsequently, a task context is configured for the agents, including input data entry points, allowed knowledge sources, a set of callable tool nodes, and output format requirements, to ensure that the agents strictly adhere to the expected behavior of the nodes during execution. To improve the robustness of task execution, a backup agent is configured for each critical node. If the primary agent fails due to resource consumption, model anomalies, or insufficient capabilities, the process automatically switches to the backup agent to re-execute the node, thus ensuring uninterrupted flow.

[0054] For example, in complex contract comparison tasks, the "evidence extraction node" will be preferentially bound to a multimodal agent with document parsing and structured extraction capabilities, while the "risk judgment node" will be bound to a large language model agent with strong reasoning capabilities, so as to form a clear division of roles and significantly improve the overall task chain execution effect.

[0055] S33: Agent Interaction and Collaborative Control After the multi-agent orchestration is completed, interaction and collaborative control between agents are realized based on a unified communication protocol and a global context management mechanism. Specifically, when each agent is scheduled to execute a corresponding node task, it sends the task result or intermediate inference information to the communication center through a message channel. The output of each agent is then structured, encapsulated, and stored in the global context. The system determines the data range accessible to subsequent nodes based on node dependencies and data read / write strategies. After any agent completes execution, its output is merged through a context update function, formally represented as:

[0056] To ensure consistent management of content generated by different agents across semantics, temporal order, and dependency consistency, subsequent agents retrieve necessary information via a context reading function before execution. This includes task elements, historical reasoning evidence, retrieved knowledge fragments, and previous node outputs, ensuring a complete and accurate environmental state for their reasoning process. Furthermore, conflict detection and version control mechanisms are introduced during agent collaboration. When multiple agents simultaneously attempt to update the same knowledge slot, timestamps, confidence levels, or node priorities are automatically compared to determine the final content to be written, preventing information overwriting and logical contradictions. These mechanisms enable highly consistent collaboration among multiple agents in complex processes, allowing information to flow smoothly between different reasoning stages and ensuring the overall process's controllability, interpretability, and robustness.

[0057] Step S4: After completing the task flow construction and agent orchestration, dynamically retrieve, aggregate evidence, and optimize strategies for the user's task-related information needs, so that the reasoning agent can obtain complete and context-consistent knowledge support.

[0058] Specifically, it can include: S41 Search Intent Identification and Query Structuring: Before entering the knowledge retrieval stage, the user's task intent is classified, decomposed, and structured using a deep semantic analysis model to generate standardized retrieval request vectors. First, multimodal semantic analysis is performed on the user's input text, image fragments, document structural elements, and speech-to-text content to extract core information requirement elements relevant to the retrieval, including entities, attributes, relations, time windows, and constraints. Then, based on the task type (such as fact query, comparison query, rule retrieval, evidence retrieval, etc.), a suitable query template is determined, and a structured query expression is constructed. Used for subsequent knowledge source selection and retrieval strategy generation.

[0059] For example, when dealing with the task of "identifying whether there are unfair clauses in a contract", the user's needs are broken down into three semantic elements: "contract clause entity", "correspondence with legal rules", and "unfair risk characteristics", thereby generating a structured query Q and proceeding to the next stage of retrieval strategy inference.

[0060] S42: Multi-source knowledge selection and dynamic path planning: After obtaining the structured query expression, a knowledge source evaluation mechanism is used to determine the suitability of different knowledge sources (including online retrieval, local knowledge bases, long-term memory, external APIs, and industry-specific databases) for the current query, and the optimal retrieval path is dynamically planned. First, the matching degree between the query and each knowledge source is calculated. in This represents the coverage, update frequency, reliability, and access cost of a knowledge source; λ is the balancing factor.

[0061] The primary retrieval source is selected based on scores, and backup retrieval sources are configured for key knowledge points to ensure automatic path switching in case of missing information, access failure, or response delay exceeding a threshold. A dynamic programming approach is then used to generate a Directed Acyclic Graph (DAG) retrieval path, including steps such as "primary query nodes," "semantic expansion nodes," "cross-source alignment nodes," and "evidence fusion nodes," to form an executable retrieval chain. For multimodal scenarios, such as reviewing mixed text and image contracts, document structure alignment nodes and table parsing nodes will be automatically inserted into the path to complete cross-modal knowledge completion.

[0062] S43: Semantic Enhancement Retrieval and Evidence Aggregation: After determining the retrieval path, the retrieval statement is semantically enhanced based on the structured query results, including synonym expansion, logical completion, time inference, context completion, and multi-hop query rewriting. The enhanced query statement is represented as follows: in To provide a global context, the reasoning results of preceding agents and user intent are integrated during the enhancement process. Subsequently, multi-level searches are performed based on selected knowledge sources, including keyword search, vector semantic search, graph database search, and cross-source collaborative search, and the search results are represented as an evidence set. .

[0063] The evidence set is subjected to quality assessment, confidence calculation, temporal consistency alignment, and multi-source redundancy denoising, and a unified evidence representation is generated through a weighted fusion strategy. This ensures the credibility, comprehensiveness, and consistency of the output content. For example, in regulatory comparison queries, the system automatically timestamps content from different versions of regulations, retaining the latest valid legal provisions and removing outdated content, thus improving the reliability of evidence.

[0064] S44: Adaptive retrieval strategy update and feedback reinforcement: After the retrieval is completed, the retrieval strategy is adaptively optimized based on the task execution results, user feedback signals, and the judgment results of the agent's inference nodes, automatically improving retrieval quality and task success rate during continuous operation. First, the difference between the retrieval results and the user's final task goal is assessed, and feedback signals are calculated by comparing indicators such as "target information gap," "confidence decay of inference nodes," and "reasons for execution failure." Based on feedback signals, retrieval parameters are updated, including query expansion strategies, knowledge source selection weights, path switching thresholds, and evidence fusion weights. Reinforcement learning or memory-based parameter update methods are used to dynamically optimize the retrieval strategy, gradually stabilizing the retrieval process and adapting it to the characteristics of different task types.

[0065] For example, when "slow online search response and lack of key evidence" are detected multiple times, the priority of online search will be automatically reduced, and the weight of local knowledge base and graph database will be increased to achieve continuous improvement in search efficiency.

[0066] Step S5: Combine knowledge evidence with the task flow, and realize problem solving, strategy generation and task closed-loop execution through reasoning agent. Through reasoning chain construction, logic verification, result generation and execution feedback, an interpretable and verifiable reasoning path is formed in multimodal input scenarios, and finally the executable result or operation instruction that meets the user's needs is output.

[0067] Specifically, it may include: S51: Reasoning Chain Construction and Context Fusion At the start of inference execution, the global context generated in the preceding steps is first used as the basis. , Retrieve evidence set The initial inference chain structure is constructed based on the input requirements of the inference nodes in the process DAG. The inference agent automatically selects the inference mode according to the node instructions, including symbolic inference, language model inference, multimodal inference, or hybrid inference, and maps the input content into a sequence of inference tokens or logical symbol representations. Subsequently, based on... Generate initial inference state During the reasoning process, cross-modal information (spatial layout extracted from images, document structure blocks, speech-to-text logical paragraphs, etc.) is dynamically integrated to ensure that the reasoning chain can unfold based on the complete context.

[0068] For example, when a user uploads a picture of meeting minutes and asks "What are the key decision-making bases for the meeting?", the reasoning agent integrates document paragraphs, OCR titles, voice supplementary content, and retrieved meeting rules and regulations to generate the starting point of the reasoning chain.

[0069] S52: Reasoning Development and Logical Verification: After the inference chain is constructed, the process of inference unfolding and logical verification begins. Intermediate inference steps are generated through step-by-step deduction, and a logical consistency check is performed at each step. The inference agent generates the inference result at step t. Afterwards, the logic verification module is invoked to perform interpretability checks, including causal consistency verification, semantic contradiction detection, temporal sequence verification, and numerical operation checks. If a contradiction or anomaly is detected, such as the inference citing non-existent evidence, incorrectly combining two conflicting facts, or making a mathematical inference error, the system rolls back to the nearest valid inference node and re-unfolds the inference chain according to a preset recovery strategy. For complex tasks, a tree-of-thought mechanism is enabled, generating multiple inference paths and using a scoring function. Choose the best branch of reasoning to ensure that the reasoning results are rigorous and reliable.

[0070] S53: Generate reasoning results: Once the reasoning chain reaches its endpoint, such as finding the exact answer, completing the strategy derivation, or triggering a termination rule, the system automatically generates a structured or natural language reasoning result based on the task type. For text tasks, it outputs a complete explanation chain and conclusion; for mixed text and image tasks, it maps the reasoning result to the original document structure, such as highlighting corresponding evidence paragraphs in the OCR document; for behavioral tasks such as automatically generating scripts, operation instructions, and schedules, it transforms the reasoning conclusion into an executable sequence of actions. And then it is handed over to the executing agent for processing.

[0071] Simultaneously, interpretable metadata is generated for the reasoning results, including evidence citation chains, reasoning path summaries, conflict branch analysis, etc., to facilitate user tracing and auditing.

[0072] For example, in contract review tasks, the output will not only be "This clause has the risk of being unfair", but will also be linked to the specific location of the clause, the source of evidence of the risk, and the reasoning chain.

[0073] S54: Execution Feedback and Result Verification: For execution-type tasks, the agent performs operations based on a sequence of actions and returns the completion status as a feedback vector. Return to the system; for reasoning tasks, compare the final answer with the retrieved evidence, user expectations, and the terminal state of the process DAG to assess its rationality and sufficiency. If a reasoning gap is found, such as insufficient evidence or a jump in the reasoning chain, a backtracking or supplementary retrieval mechanism will be automatically triggered, re-entering the corresponding module in S4 or S3 to complete the information. Finally, the reasoning result, process path, evidence usage, and error recovery record are written to the global context. As a long-term memory for future task execution, it supports the agent's habit transfer, path optimization, and policy updates.

[0074] Step S6: The executing agent performs actual operations on the target task based on the executable instruction sequence output by the inference agent and outputs the results. Specifically, this may include: first, performing structured parsing of the operation intent, action parameters, and associated objects in the inference results to generate standardized execution instruction templates. Then, the executing agent selects the appropriate execution capability module based on the instruction type, such as a document processing module, business system interaction module, data writing module, text generation module, or external API call module, and executes the corresponding actions sequentially or in parallel according to the dependencies of task nodes. During execution, the execution status is monitored in real time through status monitoring and exception capture mechanisms, automatically judging and recovering from potential issues such as insufficient permissions, data conflicts, page anomalies, or execution failures, triggering retry logic or rescheduling process nodes when necessary. After completing all actions, the executing agent uniformly transmits the execution log, output data, and status information back to the system for result verification and closed-loop optimization, achieving automated implementation from inference decision-making to actual operation.

[0075] Step S7: Based on the output returned by the executing agent, perform result verification. Review the correctness, completeness, and consistency of the output through rule validation, model evaluation, and evidence alignment. Specifically, this may include: First, selecting appropriate verification strategies based on the task type, including fact verification, business rule verification, temporal consistency checks, multimodal content comparison, and format specification checks; second, for tasks where the inference result involves external evidence, cross-referencing is performed using retrieved evidence sets or knowledge base content to identify potential biases, omissions, or semantic conflicts; when problems are found, an automatic backtracking mechanism is triggered, initiating a re-inference or supplementary recall request to the inference agent or retrieval agent, thereby performing secondary correction of the result. In the final stage, the verified content is structured, formatted, optimized, style-enhanced, or visualized, and a final result meeting the task requirements is output. Simultaneously, the feedback data generated throughout the result verification process will be used to update strategy parameters and task graph nodes, achieving continuous optimization and adaptive evolution of the agent collaboration link.

[0076] In practical applications, such as enterprise knowledge graph question answering tasks, the process can be referenced as follows: S1: A user enters a query request in the enterprise portal system: "Please summarize all the company's partners and collaborations related to artificial intelligence." First, the user's input text is segmented, entity-based, and semantically cleaned to extract the core query intent. Simultaneously, if the user uploads relevant business structure diagrams, news PDFs, or meeting minutes images, the document's layout structure is analyzed, and entity names, chart titles, business process arrows, and image content are extracted. The multimodal encoder maps text, images, and structured graph nodes to a unified semantic vector space, forming a multimodal semantic representation that can be used for knowledge retrieval and graph reasoning.

[0077] S2: Based on the input content and semantic vector, the task is classified into the "Enterprise Knowledge Graph Question Answering" type. Then, relevant nodes are selected from the process node library, including knowledge graph entity matching nodes, relationship retrieval nodes, graph structure traversal nodes, evidence verification nodes, and answer generation nodes. Based on the upstream and downstream dependencies of the nodes, a task execution DAG is constructed, realizing a complete reasoning chain from entity recognition and graph retrieval to final answer generation.

[0078] S3: Automatically selects a collaborative combination of inference agents, knowledge acquisition agents, and execution agents based on the task DAG. The inference agent is responsible for parsing the query intent and planning the graph query path; the knowledge acquisition agent is responsible for initiating searches to the enterprise's internal knowledge base, knowledge graph database, and external trusted data sources; the execution agent is responsible for result rendering, formatted output, and graph visualization generation. Dynamic load assessment and execution order adjustment are performed through the agent scheduling engine to ensure efficient collaboration among agents under complex tasks involving multi-source data.

[0079] S4: Once the reasoning agent has planned the required entity and relationship types, it determines the specific retrieval strategy based on the confidence level of graph nodes, the degree of ambiguity of entity aliases, historical query patterns, and the degree of knowledge gaps. For example: if there are multiple levels of business classifications for the topic of "artificial intelligence," a hierarchical expansion retrieval is performed; if a cooperative entity is not clearly defined in the internal graph, an external network retrieval is triggered to complete the information; if the user's query involves cross-business line associations, the retrieval agent will automatically perform multi-hop relationship reasoning and complete any missing link nodes to ensure the completeness and breadth of the recalled information.

[0080] S5: The reasoning agent combines unified semantic representation with graph retrieval results to perform path reasoning on the enterprise knowledge graph according to the task DAG. It automatically identifies all partners of the enterprise in the artificial intelligence business line and analyzes the relationship attributes between each pair of nodes, such as cooperation type, cooperation time, cooperation content, contract amount, and cooperation results. The reasoning agent further screens unreliable relationship links based on evidence confidence and performs fusion and disambiguation on duplicate entities, name aliases, and cross-domain relationships to obtain a structured knowledge answer set, preparing for the final generation of natural language summaries.

[0081] S6: The executing agent generates the final output based on the reasoning results, including: a partner list, a statistical graph of partnership types, a structured table of multiple relationship links, key partnership document references, and an automatically generated knowledge graph visualization (including node, edge, and weight information). It supports outputting results in graph mode, table mode, or business interpretation mode, and can export to PDF, image, or Markdown formats to meet different business needs.

[0082] S7: Perform a verification process on the answer results, including: checking whether the cooperation relationship conforms to the internal knowledge graph definition, confirming whether the business lines are correctly categorized, verifying whether the cooperation year and amount are consistent with the source data, and checking whether cross-departmental content has been deduplicated and integrated. If information conflicts or omissions are found, supplementary searches or secondary reasoning are automatically triggered to improve the accuracy of the results. The final output of the enterprise knowledge graph Q&A results has undergone format optimization and content enhancement to ensure that users obtain a complete, accurate, and visualized interpretation of business relationships.

[0083] For example, the workflow for intelligent contract review tasks can be referenced as follows: This specific application scenario illustrates the application process of the dynamic multi-agent orchestration system described in this invention in the context of contract review.

[0084] S1: The user uploads a PDF contract document and enters a review request: "Please identify all risk points and provide modification suggestions." First, the contract document undergoes layout structure analysis, automatically identifying areas such as titles, paragraphs, subheadings, tables, and signature areas, and extracting text content and its spatial coordinate information. Then, through text cleaning, paragraph logic restoration, and image-text alignment, text fragments, page image areas, and table structures in the contract are uniformly aligned. Finally, a multimodal encoder is used to transform the above fragments into a unified semantic vector representation, providing a computable basic semantic input for subsequent task recognition and reasoning analysis.

[0085] S2: Identify the user's input intent, determine that the task belongs to the "Intelligent Contract Review" category, and automatically extract key elements such as "identifying risk points" and "providing modification suggestions." Based on the extracted task structure information, select contract review-related nodes from a predefined process node library, including contract structure analysis nodes, legal clause risk detection nodes, regulatory comparison nodes, and modification suggestion generation nodes. Construct a task-specific process DAG based on node dependencies, forming a complete analysis chain from contract information parsing to risk identification, evidence supplementation, and suggestion generation.

[0086] S3: Based on the task flow DAG, automatically determine the agent combination required for this contract review, including a knowledge acquisition agent, a reasoning agent, and an execution agent. First, select the reasoning agent as the leading execution unit, and during the reasoning process, schedule the knowledge acquisition agent in real time to complete auxiliary tasks such as regulatory retrieval, business entity lookup, and contract template comparison. Through internal state management and message routing mechanisms, control the communication order and resource scheduling between agents to ensure the stability and efficiency of the multi-agent collaborative process.

[0087] S4: When the reasoning agent identifies potentially risky areas in a contract, such as breach of contract liability, payment terms, or confidentiality clauses, it dynamically selects the optimal search strategy based on the specific risk type. Risks related to entity consistency trigger a search of business registration information; payment risks trigger a search of internal company regulations; and legal clause risks trigger a search of national legal databases. Based on the search confidence level, the agent automatically determines whether a secondary, expanded search is necessary and submits the retrieved legal provisions, policy regulations, or historical cases as external evidence to the reasoning agent for further evaluation.

[0088] S5: Supported by unified semantic vector representation and externally retrieved evidence, the reasoning agent sequentially performs legal risk analysis on each clause of the contract. The analysis includes, but is not limited to, whether the subject information is consistent, whether the payment terms are complete and reasonable, whether the penalty for breach of contract is within acceptable limits, whether there are conflicts in the contract term, and whether confidentiality obligations are missing. For each risk point, the reasoning agent provides the risk type and triggering cause, and generates corresponding modification suggestions based on the retrieved laws and regulations, forming a structured risk analysis result.

[0089] S6: The agent receives the inference results and generates the final output report. Specifically, this includes: combining the location of risk text in the contract with the coordinate information of the OCR model to achieve visual annotation of risk content in the original contract text; classifying and sorting risks according to their type; generating suggested clauses for each risk that can be directly used for contract revision; and supporting the export of the risk list and modification suggestions as PDF or Word documents for further user use.

[0090] S7: After generating preliminary results, perform quality verification, including comparing whether the legal content cited by the reasoning agent is consistent with the retrieved evidence, checking whether the generated modification suggestions conflict with other clauses of the contract, and identifying potential risks such as mislabeling, omissions, or low confidence levels. For uncertain content, automatically trigger supplementary searches or re-inference to improve the accuracy of the results. Finally, output a verified and optimized contract review report, including a risk list, legal basis, modification suggestions, and a visually annotated contract summary.

[0091] Example 2 This invention also provides a multi-agent dynamic orchestration system based on multimodal parsing and adaptive retrieval, characterized by including a semantic conversion module, a process conversion module, a scheduling module, an execution module, an output module, and a verification module. The semantic transformation module converts user-input text, images, structured data, and speech content into a unified multimodal semantic representation for subsequent knowledge retrieval, inference scheduling, and agent collaboration processes. This process involves preprocessing the user-input text, images, structured data, and speech content, followed by multimodal data structure alignment, multimodal data encoding, and finally multimodal semantic fusion to obtain the unified multimodal semantic representation. The process transformation module converts the unified multimodal semantic representation into a specific execution flow: it performs fine-grained classification of user intent, extracts task elements, matches and generates candidate flow sequences in multiple flow orchestration node libraries, constructs a directed acyclic graph (DAG) that satisfies dependencies and constraints based on the candidate flow sequences, and evaluates the complexity of the DAG and injects alternative paths for subsequent agent orchestration and execution scheduling. The scheduling module automatically schedules inference agents, knowledge acquisition agents, and execution agents based on DAG nodes, task elements, and available resources to achieve the optimal collaboration scheme. Based on the main objectives, it generates an agent combination A={ ,..., It dynamically allocates task loads, thereby improving the efficiency of multi-agent collaboration and task completion rate. After completing the task flow construction and agent orchestration, the system dynamically retrieves, aggregates evidence, and optimizes strategies to meet the user's task-related information needs, enabling the reasoning agent to obtain complete and context-consistent knowledge support. The execution module combines knowledge evidence with task flow, using an inference agent to solve problems, generate strategies, and execute tasks in a closed loop. Through inference chain construction, logic verification, result generation, and execution feedback, it forms interpretable and verifiable inference paths in multimodal input scenarios, and ultimately outputs executable results or operation instructions that meet user needs. The output module outputs the results after the executing agent performs the actual operation on the target task based on the executable instruction sequence output by the inference agent. The verification module verifies the results based on the output returned by the executing agent, and reviews the correctness, completeness and consistency of the output results through rule verification, model evaluation and evidence alignment.

[0092] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0093] Similarly, the system of this invention can automatically identify task structure, integrate multi-source knowledge, generate executable actions, and perform real-time optimization during task processing to improve the efficiency and accuracy of handling complex tasks. Especially in enterprise intelligent office scenarios such as contract review, policy Q&A, knowledge graph query, and approval process generation, it can significantly improve the intelligence level and execution efficiency of complex task processing.

[0094] It should be noted that not all steps and modules in the above processes and system structures are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments can be physical or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0095] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval, characterized by: include: Step S1: Convert the user-input text, images, structured data, and speech content into a unified multimodal semantic representation for subsequent knowledge retrieval, inference scheduling, and agent collaboration processes. This involves preprocessing the user-input text, images, structured data, and speech content, followed by multimodal data structure alignment, multimodal data encoding, and finally multimodal semantic fusion to obtain the unified multimodal semantic representation. Step S2: Transform the unified multimodal semantic representation into a specific execution flow: perform fine-grained classification of user intent, extract task elements, and match and generate candidate flow sequences in multiple flow orchestration node libraries. Based on the candidate flow sequences, construct a directed acyclic graph (DAG) that satisfies dependencies and constraints, and perform complexity evaluation and inject alternative paths into the DAG for subsequent agent orchestration and execution scheduling. Step S3: Based on DAG nodes, task elements, and available resources, automatically schedule the reasoning agent, knowledge acquisition agent, and execution agent to achieve the optimal collaboration scheme. Generate an agent combination A={ based on the main objective, task requirements, node priorities, and resource status. ,..., It dynamically allocates task loads, thereby improving the efficiency of multi-agent collaboration and task completion rate. Step S4: After completing the task flow construction and agent orchestration, dynamically retrieve, aggregate evidence, and optimize strategies for the user's task-related information needs, enabling the reasoning agent to obtain complete and context-consistent knowledge support. Step S5: Integrate knowledge evidence with the task flow, and use an inference agent to solve problems, generate strategies, and execute tasks in a closed loop. Through inference chain construction, logic verification, result generation, and execution feedback, an interpretable and verifiable inference path is formed in a multimodal input scenario, and finally, an executable result or operation command that meets the user's needs is output. Step S6: The executing agent performs the actual operation on the target task according to the executable instruction sequence output by the inference agent, and outputs the result. Step S7: Based on the output returned by the executing agent, verify the results by reviewing the correctness, completeness and consistency of the output results through rule verification, model evaluation and evidence alignment.

2. The multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval according to claim 1, characterized in that: Step S1 specifically includes: S11: Preprocess the user-input text, images, structured data, and voice content, including: Image preprocessing is performed: image denoising, distortion correction, and brightness enhancement are carried out, and the text information and its corresponding spatial coordinates in the image are extracted using the PaddleOCR-VL model. Document structure analysis was performed: the PaddleOCR-VL model was used to classify and group document regions, forming a hierarchical structure: title, paragraph, heading, image region, table region, page number, footnote, and citation. Speech preprocessing is performed: speech segments are separated using endpoint detection, and then the speech is transcribed into text using an ASR (Qwen3-Audio) model. The preprocessed data is transformed into preliminary semantic fragments; S12: Perform multimodal data structure alignment, including: after obtaining preliminary semantic fragments, perform structural alignment on heterogeneous information from text, images, and speech, including establishing a document hierarchical tree structure based on document layout analysis results. The text lines and block elements obtained from OCR are mapped to the corresponding structure nodes, and the image region and table region are decomposed into content units. The speech-to-text is structured into a sequence structure according to the timestamps of the speech segments. By unifying node types and linking cross-modal references, a unified structural alignment graph is constructed: ,in Represents a preliminary semantic fragment. Align(·) represents the structural alignment function between cross-modal nodes, ensuring that the same semantic unit has a one-to-one or one-to-many mapping relationship in different modalities. S13: Perform multimodal data encoding: This involves aligning the representation... Encoding is performed by calling a text encoder, a vision-document joint encoder, and a speech encoder to vectorize the initial semantic fragments of each modality, thereby obtaining text features. Visual / Layout Features With speech features And form a cross-modal correlation-enhanced coding representation. The formula is as follows: ， S14: Perform multimodal semantic fusion: Map all modalities to a shared semantic space, and assign dynamic weights to different modalities through a gated fusion unit. , Compute the global fusion representation: This forms a unified semantic representation vector: .

3. The multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval according to claim 1, characterized in that: Step S2 specifically includes: S21: Extract task elements: Analyze the key objectives, constraints and available resources in the input to form a task element triple T=(G,C,R) for subsequent process matching. The objective G describes the user's core needs, the constraint C describes the limiting information in the task execution process, and the available resources R include the currently callable intelligent agent capabilities, tool nodes, knowledge sources and external interfaces. S22: Generate candidate nodes: Based on the generated task element triple T=(G,C,R), perform candidate matching on each process node in the preset process node library. A preset library of m predefined process orchestration nodes N={ ,..., }, Calculate the node semantic vector for each node. With task vector The similarity, combined with the hard rule constraints of the nodes. The matching scores are weighted to generate a candidate matching score for each node: , cos(·) represents the cosine similarity; rule_match∈{0,1} indicates a hard rule match, which is 1 if the task element meets the mandatory prerequisites of the node; λ∈[0,1] is the similarity and rule weight balancing parameter. Nodes that score above a preset threshold will be selected into the candidate node set. And sort them by matching score to provide a preliminary node sequence for subsequent process construction; S23: Based on candidate node set A directed acyclic graph (DAG) is constructed to represent the execution order and data dependencies between nodes. This involves analyzing the input-output dependencies, resource requirements, and permission constraints of each node, and establishing directed edges E based on the preconditions between nodes, forming a complete DAG = (V, E), where V⊆ Let E be a set of nodes, and let E represent the execution dependencies between nodes; S24: Calculate the execution priority p(n) of each candidate node based on the matching score, execution cost and historical stability, and configure backup paths B(n) for nodes that may fail to execute or the tool is unavailable.

4. The multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval according to claim 1, characterized in that: Step S3 specifically includes: S31: Agent selection and capability matching: Based on the functional requirements, input / output types, and required tool invocation capabilities of each node, automatically select the most suitable agent type from the agent capability library to execute the node. This involves first describing the node's task. Semantic encoding is performed and matched with the capability vectors Aj of each agent to calculate the capability fit score. , Where E(⋅) represents the deep semantic representation of capabilities and needs, the matching process comprehensively considers the agent's cognitive ability, knowledge ability, and execution ability, and distinguishes reasoning agents, knowledge agents, and execution agents based on ability differences to ensure that each node is executed by the agent with the most advantages. S32: Assign roles and bind tasks to agents: Bind specific agents to corresponding nodes according to the topological order of the process DAG, and assign clear roles and responsibilities to each agent. First, classify the nodes into inference nodes, knowledge acquisition nodes, and execution nodes according to their functional categories, and match the node requirements with the agent type to achieve role-level binding. Then, configure the task context for the agent, including the input data entry point, the knowledge sources that can be accessed, the set of tool nodes that can be called, and the output format requirements, so as to ensure that the agent can strictly follow the expected behavior of the node during execution. S33: Agent Interaction and Collaborative Control: After the multi-agent orchestration is completed, interaction and collaborative control between agents are realized based on a unified communication protocol and a global context management mechanism. When each agent is scheduled to execute the corresponding node task, it sends the task result or intermediate inference information to the communication center through a message channel. The output of each agent is structured and encapsulated and stored in the global context. It determines the range of data that subsequent nodes can access based on node dependencies and data read / write strategies.

5. The multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval according to claim 1, characterized in that: Step S4 specifically includes: S41: Retrieval Intent Identification and Query Structuring: Before entering the knowledge retrieval stage, the user's task intent is classified, decomposed, and structured using a deep semantic analysis model to generate a standardized retrieval request vector. First, multimodal semantic analysis is performed on the user's input text, image fragments, document structural elements, and speech-to-text content to extract core information requirement elements relevant to the retrieval, including entities, attributes, relations, time windows, and constraints. Then, a suitable query template is determined based on the task type, and a structured query expression is constructed. Used for subsequent knowledge source selection and retrieval strategy generation. S42: Perform multi-source knowledge selection and dynamic path planning: After obtaining the structured query expression, the suitability of different knowledge sources for the current query is determined through a knowledge source evaluation mechanism, and the optimal retrieval path is dynamically planned. The first step is to calculate the matching degree between the query and each knowledge source. in The metric represents the coverage, update frequency, reliability, and access cost of a knowledge source; λ is a balancing factor, which selects the primary retrieval source based on its score and configures backup retrieval sources for key knowledge points to ensure automatic path switching in case of missing information, access failure, or response delay exceeding a threshold. Then, a dynamic programming method is used to generate a Directed Acyclic Graph (DAG) of retrieval paths. S43: Perform semantically enhanced retrieval and evidence aggregation: After determining the retrieval path, semantic enhancement is performed on the retrieval statement based on the structured query results. This includes synonym expansion, logical completion, time inference, context completion, and multi-hop query rewriting. The enhanced query statement is represented as follows: in To provide a global context, the enhancement process integrates the reasoning results of preceding agents with user intent. Subsequently, multi-level retrieval is performed based on selected knowledge sources, including keyword retrieval, vector semantic retrieval, graph database retrieval, and cross-source collaborative retrieval. The retrieval results are then represented as a set of evidence. , The evidence set is subjected to quality assessment, confidence calculation, temporal consistency alignment, and multi-source redundancy denoising, and a unified evidence representation is generated through a weighted fusion strategy. To ensure the credibility, comprehensiveness, and consistency of the output content, S44: Perform adaptive retrieval strategy updates and feedback reinforcement: After the retrieval is completed, the retrieval strategy is adaptively optimized based on the task execution results, user feedback signals, and the judgment results of the agent inference nodes, automatically improving the retrieval quality and task success rate during continuous operation.

6. The multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval according to claim 1, characterized in that step S5 specifically includes: S51: Inference Chain Construction and Context Fusion: At the start of inference execution, the global context generated by the preceding steps is first used as the basis for the inference chain construction. , Retrieve evidence set And the input requirements of the inference nodes in the process DAG, to construct the initial inference chain structure. S52: Reasoning Expansion and Logic Verification: After the reasoning chain is constructed, the reasoning expansion and logic verification process begins. Intermediate reasoning steps are generated through step-by-step deduction, and a logic consistency check is performed at each step. S53: Generate Reasoning Results: When the reasoning chain reaches the endpoint condition, automatically generate a structured or natural language reasoning result based on the task type. Simultaneously, generate interpretable metadata for the reasoning result, including evidence citation chains, reasoning path summaries, conflict branch analysis, etc., to facilitate user tracing and auditing. S54: Execution Feedback and Result Verification: For execution-type tasks, the agent performs operations according to the action sequence and verifies the completion status as a feedback vector. Return to the system; for reasoning tasks, compare the final answer with the retrieved evidence, user expectations, and the terminal state of the process DAG to assess its rationality and sufficiency.

7. The multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval according to claim 1, characterized in that: Step S6 specifically includes: firstly, performing structured parsing of the operational intent, action parameters, and associated objects in the reasoning results to generate standardized execution instruction templates. Subsequently, the executing agent selects the appropriate execution capability module based on the instruction type and executes the corresponding actions sequentially or in parallel according to the dependencies of the task nodes. During execution, the execution status is monitored in real time through status monitoring and exception capture mechanisms. It automatically judges and recovers from possible situations such as insufficient permissions, data conflicts, page errors, or execution failures, and triggers retry logic or reschedules process nodes when necessary. After completing all actions, the executing agent will uniformly transmit the execution log, output data, and status information back for result verification and closed-loop optimization, realizing the automated implementation from reasoning and decision-making to actual operation.

8. The multi-agent dynamic orchestration method based on multimodal parsing and adaptive retrieval according to claim 1, characterized in that: Step S7 specifically includes: First, selecting the appropriate verification strategy based on the task type, including fact verification, business rule verification, time sequence consistency check, multimodal content comparison and format specification detection; Secondly, for tasks where the reasoning results involve external evidence, cross-comparison is performed using the retrieved evidence set or knowledge base content to identify potential biases, omissions, or semantic conflicts; when a problem is found, a backtracking mechanism is automatically triggered to initiate a re-reasoning or supplementary recall request to the reasoning agent or retrieval agent, thereby making secondary corrections to the results; Finally, the verified content is structured, formatted, optimized, styled, or visualized, and the final result that meets the task requirements is output. Meanwhile, the feedback data generated throughout the result verification process will be used to update the policy parameters and task graph nodes, so as to realize the continuous optimization and adaptive evolution of the agent collaboration link.

9. A multi-agent dynamic orchestration system based on multimodal parsing and adaptive retrieval, characterized in that: It includes a semantic conversion module, a process conversion module, a scheduling module, an execution module, an output module, and a verification module. The semantic transformation module converts user-input text, images, structured data, and speech content into a unified multimodal semantic representation for subsequent knowledge retrieval, inference scheduling, and agent collaboration processes. This process involves preprocessing the user-input text, images, structured data, and speech content, followed by multimodal data structure alignment, multimodal data encoding, and finally multimodal semantic fusion to obtain the unified multimodal semantic representation. The process transformation module converts the unified multimodal semantic representation into a specific execution flow: it performs fine-grained classification of user intent, extracts task elements, matches and generates candidate flow sequences in multiple flow orchestration node libraries, constructs a directed acyclic graph (DAG) that satisfies dependencies and constraints based on the candidate flow sequences, and evaluates the complexity of the DAG and injects alternative paths for subsequent agent orchestration and execution scheduling. The scheduling module automatically schedules inference agents, knowledge acquisition agents, and execution agents based on DAG nodes, task elements, and available resources to achieve the optimal collaboration scheme. Based on the main objectives, it generates an agent combination A={ ,..., It dynamically allocates task loads, thereby improving the efficiency of multi-agent collaboration and task completion rate. After completing the task flow construction and agent orchestration, the system dynamically retrieves, aggregates evidence, and optimizes strategies to meet the user's task-related information needs, enabling the reasoning agent to obtain complete and context-consistent knowledge support. The execution module combines knowledge evidence with task flow, using an inference agent to solve problems, generate strategies, and execute tasks in a closed loop. Through inference chain construction, logic verification, result generation, and execution feedback, it forms interpretable and verifiable inference paths in multimodal input scenarios, and ultimately outputs executable results or operation instructions that meet user needs. The output module outputs the results after the executing agent performs the actual operation on the target task based on the executable instruction sequence output by the inference agent. The verification module verifies the results based on the output returned by the executing agent, and reviews the correctness, completeness and consistency of the output results through rule verification, model evaluation and evidence alignment.

Images