A multi-modal agent process orchestration system and process orchestration method

Abstract

The application discloses a multi-modal intelligent agent process arrangement system and a process arrangement method. Since the system is applied to an end-side device and is deployed with a lightweight offline structured processing large model and a process arrangement large model, the system can complete data structured processing and process arrangement operation independently at the local end-side without relying on cloud network computing power, can break away from external network restrictions, has low deployment threshold, has fast running response speed, can reduce cloud calling cost and network delay, is suitable for offline office, edge terminal and other network-free use scenarios, completely solves the problem of intelligent agent work stagnation when the network is interrupted, and guarantees the continuity and stability of multi-modal intelligent agent process arrangement. The intelligent agent process arrangement module combines the reasoning and analysis capability of the process arrangement large model, realizes standardized output of the multi-modal intelligent agent process arrangement scheme, breaks away from the dependence on personal experience, and improves the efficiency and accuracy of process arrangement.

Description

Technical Field

[0001] This application relates to the fields of data processing and deep learning technology, and in particular to a multimodal intelligent agent process orchestration system and process orchestration method. Background Technology

[0002] With the rapid development of multimodal big data models and intelligent agent technologies, intelligent agents that integrate multiple modalities of data, such as text, images, and videos, have been widely applied in complex business scenarios such as smart offices, intelligent monitoring, content production, and industrial quality inspection, becoming the core carrier supporting multi-source data collaborative processing and automated task execution. In practical applications, the core link to ensuring the efficient and accurate completion of multimodal tasks is that multimodal intelligent agents need to complete process orchestration according to different business scenarios.

[0003] Currently, the orchestration and workflow configuration of multimodal intelligent agents mainly rely on manual processes. Technical personnel need to manually analyze the collaborative logic of different modalities and combine their professional experience to determine the appropriate orchestration scheme for different business scenarios. The entire process lacks a unified automated orchestration mechanism. This manual orchestration mode is not only time-consuming and labor-intensive, but also prone to errors in orchestration logic due to the complexity of modal types. This results in slow process orchestration speed, failing to meet the needs of efficient multimodal business advancement, and causing untimely response and execution of multimodal tasks, significantly reducing the overall efficiency of intelligent agent process orchestration.

[0004] Meanwhile, the knowledge system for orchestrating multimodal intelligent agents suffers from significant fragmentation. Its core content, such as orchestration knowledge, scenario adaptation rules, inter-agent collaborative skills, and fault handling solutions, is scattered across multiple media, including paper documents, electronic manuals, and online platforms. Manual orchestration requires consulting various resources one by one, significantly increasing the workload. More importantly, the industry currently lacks unified standards for process orchestration and intelligent agent collaborative logic specifications. Manual orchestration results heavily rely on individual experience, leading to significant differences in orchestration schemes among different technicians. This directly results in insufficient accuracy and low standardization in process orchestration, impacting the collaborative execution effectiveness of multimodal intelligent agents and increasing the operational costs of multimodal businesses.

[0005] In summary, existing technical solutions suffer from prominent problems such as low efficiency in multimodal intelligent agent process orchestration and insufficient standardization and accuracy in orchestration. This results in low efficiency and intelligence levels in multimodal intelligent agent process orchestration and multimodal task execution, making it difficult to meet the actual needs of high-quality and large-scale promotion of current multimodal businesses. Summary of the Invention

[0006] To address this, this application provides a multimodal intelligent agent process orchestration system and method to solve the problems of low efficiency, insufficient standardization, and inadequate accuracy in existing multimodal intelligent agent process orchestration solutions.

[0007] To achieve the above objectives, this application provides the following technical solution:

[0008] In a first aspect, this application provides a multimodal intelligent agent process orchestration system, characterized in that the system is applied to an edge device, and the system includes:

[0009] The structured processing module is equipped with a lightweight offline structured processing model, which is used to transform the acquired multimodal data into structured orchestration requirement data through the structured processing model.

[0010] The intelligent agent orchestration skill configuration module is used to store standardized orchestration knowledge graphs, scene adaptation rules, and intelligent agent collaborative logic libraries;

[0011] The intelligent agent process orchestration module deploys a lightweight, offline process orchestration model. This model, based on the structured orchestration requirement data and combined with the intelligent agent orchestration skill configuration module, generates an intelligent agent process orchestration scheme. The process of generating the intelligent agent process orchestration scheme includes:

[0012] Based on the structured orchestration requirement data, corresponding target scenario adaptation rules are matched from the agent orchestration skill configuration module; according to the target scenario adaptation rules and the orchestration knowledge graph, multiple target agents to be invoked are determined; based on the agent collaboration logic library, the collaborative execution logic and data flow relationship of the multiple target agents are determined; and according to the collaborative execution logic and data flow relationship, the multiple target agents are logically assembled to generate an agent process orchestration scheme.

[0013] Secondly, this application provides a process orchestration method based on any of the multimodal intelligent agent process orchestration systems described above, characterized in that the method includes:

[0014] Acquire multimodal data;

[0015] Based on the multimodal data, an intelligent agent process orchestration scheme is obtained through a multimodal intelligent agent process orchestration system.

[0016] Thirdly, this application provides a computer device including a processor, which executes a computer program stored in a memory to implement the steps of the process orchestration method described above.

[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the process orchestration method described above.

[0018] The beneficial effects of this application are as follows:

[0019] 1. By using structured processing of large models, we can achieve structured processing of multimodal data, eliminate differences in data formats of different modalities, unify the expression of requirements, and accurately transform scattered and complex raw data into standard arrangement requirements, thereby improving the accuracy and universality of requirement parsing.

[0020] 2. Since the system is applied to edge devices and is equipped with lightweight offline structured processing and process orchestration models, it can independently complete data structured processing and process orchestration calculations on the local edge without relying on cloud network computing power. It is free from external network restrictions, has a low deployment threshold, and a fast running response speed. At the same time, it reduces cloud call costs and network latency, and is suitable for offline office, edge terminal and other network-free usage scenarios. It completely solves the problem of intelligent agent work stagnation when the network is interrupted, and ensures the continuity and stability of multimodal intelligent agent process orchestration.

[0021] 3. The intelligent agent orchestration skill configuration module has a built-in standardized orchestration knowledge graph, scene adaptation rules, and intelligent agent collaboration logic, which enables unified management, reuse, and rapid retrieval of resources, avoids repetitive R&D and configuration work, and greatly reduces the time spent on early rule building and logic sorting.

[0022] 4. The intelligent agent process orchestration module combines the reasoning and analysis capabilities of the large process orchestration model to achieve standardized output of multimodal intelligent agent process orchestration solutions. This eliminates reliance on personal experience, accurately matches business scenario requirements, and formulates scientific orchestration logic, thereby improving the efficiency and accuracy of process orchestration. It also saves the workload of manually sorting out the division of labor, docking logic, and data transmission relationships of intelligent agents, reducing the error rate of manual orchestration and the operating costs of multimodal business. Attached Figure Description

[0023] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0024] Figure 1 A schematic diagram of a system structure for multimodal intelligent agent process orchestration provided in this application embodiment;

[0025] Figure 2 A schematic diagram of a system structure for another multimodal agent process orchestration provided in this application embodiment;

[0026] Figure 3 A schematic diagram of a process arrangement provided in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application. Detailed Implementation

[0028] The following specific embodiments illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] To achieve automatic process orchestration for multimodal data and improve the efficiency, standardization, and accuracy of multimodal data process orchestration, this application provides a multimodal intelligent agent process orchestration system and process orchestration method.

[0030] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Example:

[0032] Figure 1 A schematic diagram of a system architecture for multimodal intelligent agent process orchestration provided in this application embodiment, the system comprising:

[0033] The structured processing module 11 is equipped with a lightweight offline structured processing model, which is used to transform the acquired multimodal data into structured orchestration requirement data through the structured processing model.

[0034] The intelligent agent orchestration skill configuration module 12 is used to store standardized orchestration knowledge graphs, scene adaptation rules, and intelligent agent collaborative logic libraries;

[0035] The agent process orchestration module 13 is equipped with a lightweight offline process orchestration model. This model, based on the structured orchestration requirement data and combined with the agent orchestration skill configuration module 12, generates an agent process orchestration scheme. The process of generating the agent process orchestration scheme includes:

[0036] Based on the structured orchestration requirement data, the corresponding target scenario adaptation rules are matched from the agent orchestration skill configuration module 12; according to the target scenario adaptation rules and the orchestration knowledge graph, multiple target agents to be invoked are determined; based on the agent collaboration logic library, the collaborative execution logic and data flow relationship of the multiple target agents are determined; and according to the collaborative execution logic and data flow relationship, the multiple target agents are logically assembled to generate an agent process orchestration scheme.

[0037] To achieve automated and standardized workflow orchestration of multimodal data, reduce manual intervention costs, and improve the adaptability and accuracy of orchestration schemes, this application provides a multimodal intelligent agent workflow orchestration system (hereinafter referred to as the system). This system is deployed on edge devices, such as mobile terminals, personal computers, and industrial control terminals, and does not rely on cloud computing power, enabling efficient offline operation. The system includes a structured processing module 11, an intelligent agent orchestration skill configuration module 12, and an intelligent agent workflow orchestration module 13. Through the collaborative work of these modules, the entire process of multimodal data acquisition, structured processing, automatic intelligent agent workflow orchestration, and scheme generation can be automated, effectively solving the problems of low efficiency, low standardization, and error susceptibility in traditional multimodal workflow orchestration. The functions of each module are described in detail below:

[0038] (1) Structured processing module 11

[0039] The structured processing module 11 provides data structuring services, transforming unstructured and semi-structured multimodal data into structured orchestration requirement data in a unified format that can be directly used for subsequent process orchestration. This facilitates the subsequent intelligent agent process orchestration module 13 in quickly identifying requirements and matching adaptation rules, improving the efficiency and accuracy of process orchestration. Simultaneously, it achieves standardized processing of multimodal data, avoiding orchestration errors caused by inconsistent data formats. The types of multimodal data include text, images, and audio, such as business requirement text, scene images / videos, intelligent agent execution logs, and task progress data. This provides complete raw data support for edge-side intelligent agent process orchestration, enabling comprehensive perception of multimodal data and intelligent agent states.

[0040] To achieve offline operation on the edge, reduce reliance on network bandwidth and cloud computing power, and ensure the accuracy and speed of structured processing, this application deploys a lightweight offline structured processing large model in the structured processing module 11. Lightweight offline processing refers to model compression of the original large model, such as pruning, quantization, and knowledge distillation, reducing the model parameter scale and computational complexity. This allows the model to adapt to the hardware resources of the edge device and independently complete data structured processing without network connectivity, ensuring processing efficiency while improving data privacy and security, and preventing information leakage caused by the external transmission of multimodal data. For example, the structured processing module 11 inputs the acquired multimodal data into the lightweight offline structured processing large model. This large model performs integrated offline recognition, classification, feature extraction, and semantic / visual parsing of the input multimodal data, transforming the multimodal features into structured orchestration requirement data, and finally outputting it to the intelligent agent process orchestration module 13. Among them, the structured orchestration requirement data represents the core requirements of the process orchestration corresponding to the multimodal data, such as key information such as requirement type, processing target, constraints, and data specifications. It can be directly used by the intelligent agent process orchestration module 13 to identify requirements, match scenario adaptation rules, and generate orchestration schemes.

[0041] In one possible implementation, the multimodal data acquired by the structured processing module 11 can be obtained by receiving data transmitted from other devices, such as receiving historical multimodal data sent by a cloud server, receiving real-time multimodal data shared by other end devices, or receiving data transmitted by external sensors or acquisition devices through an interface. The transmission method can be wired transmission, such as Ethernet or USB, or wireless transmission, such as WiFi, Bluetooth, or 5G.

[0042] In another possible implementation, the multimodal data acquired by the structured processing module 11 can also be collected by the system itself, without relying on external devices for transmission, thus improving the timeliness and autonomy of data acquisition. For example, the system further includes a multimodal information acquisition module; the multimodal information acquisition module is used to acquire the multimodal data and output it to the structured processing module 11.

[0043] The multimodal information acquisition module can integrate various acquisition units, including image acquisition units such as cameras and scanners, audio acquisition units such as microphones and recording devices, text acquisition units such as keyboard input modules and document reading modules, and sensor acquisition units such as temperature sensors, humidity sensors, and vibration sensors. The corresponding type of acquisition unit can be flexibly activated according to the needs of the actual application scenario. Multimodal data is acquired through each acquisition unit and output to the structured processing module 11.

[0044] In one possible implementation, during the acquisition process, the multimodal information acquisition module can perform preliminary screening and preprocessing on the acquired raw data, removing invalid data such as blurry images, meaningless audio, and garbled text, and unifying the data format, such as converting images of different formats to JPG format and audio to WAV format. Then, according to the preset acquisition frequency and data storage rules, the processed multimodal data is output to the structured processing module 11 to ensure that the structured processing module 11 can acquire valid data in a timely manner, providing support for subsequent process arrangement.

[0045] In one example, to obtain a lightweight offline structured processing model, this application constructs a multimodal training dataset containing a large number of multimodal data samples of different types and scenarios, along with corresponding labeled structured data, ensuring the diversity and representativeness of the dataset. Then, a basic large model, such as a visual-language model, is selected as a pre-trained model. Based on the constructed multimodal training dataset, knowledge distillation is used to transfer knowledge from the basic large model to the lightweight model. Simultaneously, the lightweight model is compressed, removing redundant model parameters and network layers to reduce model complexity. The trained lightweight offline structured processing model is validated and fine-tuned using an offline validation dataset to optimize the model's structured processing accuracy, ensuring that the model can quickly and accurately transform multimodal data into structured orchestration data in an offline state, while meeting the speed and resource requirements of edge devices. After obtaining the validated structured processing model, it can be deployed to the system to facilitate subsequent data structured processing services.

[0046] (2) Intelligent agent orchestration skill configuration module 12

[0047] The intelligent agent orchestration skill configuration module 12 is used to store standardized orchestration knowledge graphs, scenario adaptation rules, and intelligent agent collaboration logic libraries to ensure the standardization and normalization of process orchestration, while improving the scenario adaptability of orchestration schemes and the rationality of intelligent agent collaboration. Among them, the orchestration knowledge graph is a structured knowledge representation form, including agent attribute data, process orchestration node data, multimodal data and agent association data, and scene-orchestration scheme mapping data, etc., which provide knowledge support for agent selection and process node orchestration. It can help the process orchestration module 13 quickly locate agents and orchestration logic that match the current requirements, avoid ineffective screening, and improve orchestration efficiency. The scene adaptation rules are a set of standardized rules preset based on different application scenarios, including scene type definitions, scene requirement characteristics, the scope of adapted agents, orchestration priorities, etc., which are used to quickly match the corresponding application scenarios based on structured orchestration requirement data, determine the overall direction and constraints of the orchestration scheme, and ensure that the orchestration scheme can adapt to the requirements of specific scenarios. The agent collaboration logic library is a data set that stores the logic related to the collaborative operation between agents. It is used to define the interaction rules and operating mechanisms when multiple agents work together, including agent execution order, data flow path, collaboration trigger conditions, and abnormal branch handling logic, etc. Through this data, the responsibilities and collaboration methods of each agent can be clarified, avoiding agent collaboration chaos and ensuring that the process orchestration scheme can be executed smoothly. The standardized data storage and management described above facilitates the standardization of process orchestration, reduces the complexity of manual intervention, and enhances the reusability and scalability of the orchestration scheme. When adding new scenarios or intelligent agents in the future, only the corresponding data set needs to be updated, without modifying the core architecture of the system.

[0048] In one possible implementation, the collaborative execution logic includes: agent execution order, data flow path, collaborative triggering conditions, and exception branch handling logic. Among them, the execution order of intelligent agents refers to the sequential execution of their respective tasks by multiple target intelligent agents according to a preset logical order, specifying which intelligent agent executes first and which executes last. For example, in a multimodal data processing scenario, the data parsing intelligent agent is executed first, followed by the data verification intelligent agent, and finally the data storage intelligent agent. The data flow path refers to the data transmission path and method between intelligent agents, specifying how the output data of the previous intelligent agent is transmitted to the next intelligent agent, as well as the format, frequency, and priority of data transmission, to ensure efficient and accurate data flow between intelligent agents. The collaborative triggering condition refers to the preconditions for triggering collaborative work between intelligent agents. For example, when the data parsing intelligent agent completes data parsing and outputs qualified data, it triggers the data verification intelligent agent to start working. If the parsed data is unqualified, subsequent intelligent agents are not triggered, and exception handling is initiated directly. The exception branch handling logic refers to the preset handling rules when an exception occurs during the process execution, such as intelligent agent execution failure, data transmission interruption, or requirement mismatch. For example, when an intelligent agent fails to execute, a backup intelligent agent is started to replace it. If there is no backup intelligent agent, the current process is terminated and an exception prompt is output, while the exception information is recorded to provide a basis for subsequent investigation and optimization.

[0049] In one example, considering that orchestration requirements may change dynamically in real-world application scenarios, such as adjustments to user needs, changes in scenario parameters, and updates to data formats, the absence of a corresponding process adjustment mechanism would prevent the generated orchestration scheme from adapting to the changed requirements, thus affecting the process execution effect. Therefore, in this application, the intelligent agent orchestration skill configuration module 12 is also used to store standardized process adjustment criteria. Process adjustment guidelines are pre-defined, standardized rules used to guide adjustments to generated orchestration schemes. These guidelines include data such as adjustment trigger conditions, adjustment scope, adjustment priority, and post-adjustment collaboration logic adaptation rules. Adjustment trigger conditions clarify when adjustments to the orchestration scheme are necessary, such as changes to structured orchestration requirement data, updates to scenario adaptation rules, agent state anomalies, the matching degree between the current agent's process orchestration scheme and the current business, agent resource utilization, and task execution risks. The adjustment scope specifies the details of the adjustment, such as adjusting only the agent's execution order, modifying only the data flow path, or adjusting the entire target agent selection. Adjustment priority determines which aspects should be adjusted first when multiple adjustment requirements exist, ensuring the adjusted scheme prioritizes core needs. Post-adjustment collaboration logic adaptation rules ensure that the adjusted agent collaboration logic matches the updated requirements and rules, preventing collaboration chaos after adjustment. By storing these process adjustment guidelines, the system can automatically adjust generated agent process orchestration schemes based on dynamic changes in requirements and scenarios, improving system flexibility and adaptability without manual re-orchestration, further enhancing process orchestration efficiency.

[0050] (3) Intelligent Agent Process Orchestration Module 13

[0051] The intelligent agent process orchestration module 13 provides automatic orchestration services for intelligent agents. Based on structured orchestration requirement data and relevant data stored in the intelligent agent orchestration skill configuration module 12, it automatically generates standardized intelligent agent process orchestration schemes that meet scenario requirements and have reasonable collaborative logic. This replaces manual orchestration, improving the efficiency, accuracy, and standardization of process orchestration while lowering the orchestration threshold, allowing for the orchestration of complex multimodal scenarios without the need for professional personnel. The intelligent agent process orchestration module 13 deploys a lightweight offline process orchestration model. Using this model, based on structured orchestration requirement data and combined with the intelligent agent orchestration skill configuration module 12, it generates intelligent agent process orchestration schemes. Similar to the lightweight offline model of the structured processing module 11, this large-scale process orchestration model undergoes model compression, quantization, and knowledge distillation, making it adaptable to the hardware resources of edge devices. It can independently complete process orchestration without a network connection, ensuring both orchestration speed and privacy security during the orchestration process, preventing the external transmission of orchestration requirements and scheme data.

[0052] To ensure the rationality and adaptability of the process orchestration scheme, the agent process orchestration module 13 receives structured orchestration requirement data output by the structured processing module 11 and inputs this structured orchestration requirement data into the large process orchestration model. The process orchestration model processes the structured orchestration requirement data to generate the agent process orchestration scheme, including:

[0053] First, based on the input structured orchestration requirement data, the corresponding target scene adaptation rules are matched from the agent orchestration skill configuration module 12. For example, the input structured orchestration requirement data is parsed to extract core requirements, such as processing objectives, constraints, scene types, and data specifications. Based on the parsed core requirements, the corresponding target scene adaptation rules are matched from the agent orchestration skill configuration module 12.

[0054] Next, based on the matched target scenario adaptation rules and combined with the orchestration knowledge graph, multiple target agents that can meet the current task requirements are selected from the orchestration knowledge graph. The function, attributes and applicable scope of each target agent are clarified to ensure that the selected agents can cover all processing links of the current requirements.

[0055] Based on the intelligent agent collaboration logic library, and combined with the selected target intelligent agents, the collaborative execution logic and data flow relationship of multiple target intelligent agents are determined to ensure that each intelligent agent can work together smoothly and avoid execution conflicts or data gaps.

[0056] Finally, based on the determined collaborative execution logic and data flow relationship, multiple target intelligent agents are logically assembled to generate a complete intelligent agent process orchestration scheme.

[0057] In one example, the agent orchestration skill configuration module 12 is also used to store standardized process adjustment criteria;

[0058] The process by which the agent process orchestration module 13 generates the agent process orchestration scheme through the process orchestration big model also includes:

[0059] Determine the compatibility evaluation parameters between the intelligent agent process orchestration scheme and the current business; wherein, the compatibility evaluation parameters include one or more of the following information: the matching degree between the current intelligent agent process orchestration scheme and the current business, intelligent agent resource utilization rate, and task execution risk;

[0060] Based on the adaptation evaluation parameters and the process adjustment criteria, the agent process orchestration scheme is adjusted to execute the current task according to the adjusted agent process orchestration scheme.

[0061] In this application, the intelligent agent process orchestration module 13, in the process of generating an intelligent agent process orchestration scheme through the large process orchestration model, additionally adds a scheme adaptation evaluation and dynamic adjustment step to further improve the scenario adaptability and execution reliability of the scheme. For example, after generating the intelligent agent process orchestration scheme, the large process orchestration model can determine the adaptation evaluation parameters between the intelligent agent process orchestration scheme and the current business. These adaptation evaluation parameters are core indicators used to measure the degree of matching between the scheme and the current business and its execution feasibility, and mainly include one or more of the following information: the degree of matching between the current intelligent agent process orchestration scheme and the current business, intelligent agent resource utilization, and task execution risk. The matching degree between the current intelligent agent process orchestration scheme and the current business refers to the degree to which the generated intelligent agent process orchestration scheme fits the actual business needs, scenarios, constraints, etc. The higher the matching degree, the better the current intelligent agent process orchestration scheme fits the actual business needs and the better the execution effect. Intelligent agent resource utilization rate refers to the ratio of the resource occupation of multiple target intelligent agents called in the current intelligent agent process orchestration scheme to the actual available resources. The higher the resource utilization rate, the more reasonable the intelligent agent resource allocation, which can effectively avoid resource waste and reduce the resource consumption of edge devices. Task execution risk refers to the probability of abnormal situations that may occur during the execution of the scheme, such as intelligent agent execution failure, data transmission interruption, process lag, etc. The lower the risk, the stronger the execution stability and reliability of the scheme. After determining the adaptation evaluation parameters, the process orchestration big model will adjust the generated intelligent agent process orchestration scheme based on the adaptation evaluation parameters and the process adjustment criteria stored in the intelligent agent orchestration skill configuration module 12. If the adaptation evaluation parameters meet the preset qualification standards in the process adjustment guidelines, it means that the solution is suitable for the current business and does not need to be adjusted; it can be directly used to execute the current task. If the adaptation evaluation parameters do not meet the qualification standards, the process orchestration module will adjust the solution content in a targeted manner according to the adjustment rules in the process adjustment guidelines. For example, when the solution is not well matched with the current business, the scenario adaptation rules will be re-matched and the target intelligent agent will be selected. When the intelligent agent resource utilization is too low, the intelligent agent collaborative execution order will be adjusted and resource allocation will be optimized. When the task execution risk is too high, the abnormal branch handling logic will be improved and the backup intelligent agent configuration will be added until the adaptation evaluation parameters corresponding to the adjusted solution meet the qualification standards. Then, the current task will be executed according to the adjusted intelligent agent process orchestration solution.

[0062] In one possible implementation, the adjustment of the agent process orchestration scheme includes one or more of the following: orchestration node optimization, agent collaboration mode adjustment, and task execution order change. Orchestration node optimization refers to optimizing the execution nodes corresponding to each agent in the scheme, including adding necessary execution nodes, deleting redundant nodes, and optimizing node execution parameters to ensure that the function of each node accurately matches business requirements and improves the scheme's execution efficiency. Agent collaboration mode adjustment refers to changing the collaborative interaction mode between multiple target agents, including adjusting the communication methods, data transmission formats, and collaborative triggering conditions between agents to solve problems such as poor agent collaboration and delayed response. Task execution order change refers to adjusting the order in which tasks corresponding to each agent are executed, optimizing the execution logic based on business requirements and agent resource usage, avoiding task conflicts, and improving resource utilization.

[0063] The beneficial effects of this application are as follows:

[0064] 1. By using structured processing of large models, we can achieve structured processing of multimodal data, eliminate differences in data formats of different modalities, unify the expression of requirements, and accurately transform scattered and complex raw data into standard arrangement requirements, thereby improving the accuracy and universality of requirement parsing.

[0065] 2. Since the system is applied to edge devices and is equipped with lightweight offline structured processing and process orchestration models, it can independently complete data structured processing and process orchestration calculations on the local edge without relying on cloud network computing power. It is free from external network restrictions, has a low deployment threshold, and a fast running response speed. At the same time, it reduces cloud call costs and network latency, and is suitable for offline office, edge terminal and other network-free usage scenarios. It completely solves the problem of intelligent agent work stagnation when the network is interrupted, and ensures the continuity and stability of multimodal intelligent agent process orchestration.

[0066] 3. The intelligent agent orchestration skill configuration module 12 has a built-in standardized orchestration knowledge graph, scene adaptation rules, and intelligent agent collaboration logic, which realizes unified management, reuse and rapid retrieval of resources, avoids repeated R&D and configuration work, and greatly reduces the time spent on early rule building and logic sorting.

[0067] 4. The intelligent agent process orchestration module 13 combines the reasoning and analysis capabilities of the process orchestration big model to achieve standardized output of multimodal intelligent agent process orchestration solutions. It eliminates the reliance on personal experience, accurately matches business scenario requirements, and formulates scientific orchestration logic, thereby improving the efficiency and accuracy of process orchestration. It also saves the workload of manually sorting out the division of labor, docking logic, and data transmission relationships of intelligent agents, reducing the error rate of manual orchestration and the operating costs of multimodal business.

[0068] Example 2:

[0069] To ensure the successful implementation of the intelligent agent workflow orchestration scheme and improve user experience, in addition to the above embodiments, the system in this application further includes:

[0070] The orchestration execution feedback module is used to generate and issue execution instructions corresponding to the intelligent agent process orchestration scheme.

[0071] In this application, the system also includes an orchestration execution feedback module. This module is primarily responsible for generating and issuing execution instructions corresponding to the agent process orchestration scheme, enabling the generated scheme to directly drive various agents to perform their tasks in an orderly manner. The execution instructions generated by the orchestration execution feedback module are executable scheduling instructions derived from the final determined agent process orchestration scheme. Each instruction defines the specific tasks that each target agent needs to complete, clarifies data interaction requirements and start / stop control conditions, and accurately corresponds to all execution logic within the orchestration scheme. The orchestration execution feedback module can issue execution instructions via targeted push according to a unified communication protocol or in batches based on the agent distribution status, ensuring timely and stable instruction transmission.

[0072] In one example, to fully retain the entire lifecycle of the process and facilitate subsequent problem tracing, process review, and model iteration optimization, the orchestration execution feedback module in this application, in addition to issuing execution instructions, can also acquire the running status data, process progress data, and final task completion result data generated by each target agent during task execution, while summarizing various abnormal information that occurs during process execution. The orchestration execution feedback module saves all collected feedback data as log data for the agent process orchestration scheme. For example, this log data includes, but is not limited to, one or more of the following: orchestration requirement data, agent running status, task execution progress, execution process data, and execution results. Orchestration requirement data refers to the structured requirements corresponding to the initial process orchestration. Agent running status records the working conditions of each target agent participating in the task. Task execution progress reflects the completion degree of the overall orchestration process and individual branch processes. Execution process data records various data generated during the process, such as data interaction content between agents, resource call status, and parameter change information. Execution results save the final completion status of the entire orchestration task, business output content, and relevant information on task achievement.

[0073] In one possible implementation, the system further includes:

[0074] The orchestration strategy dynamic generation module is used to determine the compatibility evaluation parameters between the current agent process orchestration scheme and the current business based on the log data through the process orchestration big model, and to adjust the current agent process orchestration scheme based on the compatibility evaluation parameters and the process adjustment criteria, so as to execute the current task according to the adjusted agent process orchestration scheme.

[0075] To continuously optimize orchestration logic based on historical execution data, overcome the adaptation limitations of fixed orchestration patterns, and further enhance the adaptability and long-term operational effectiveness of process orchestration, this application also includes a dynamic orchestration strategy generation module. This module can comprehensively analyze and deeply examine log data through a large-scale process orchestration model. From the orchestration requirement data, agent running status, task execution progress, execution process data, and execution results contained in the logs, it determines the corresponding adaptation evaluation parameters between the currently used agent process orchestration scheme and the actual business operations. After obtaining the adaptation evaluation parameters, the large-scale process orchestration model retrieves the process adjustment criteria stored internally in the agent orchestration skill configuration module 12, and strictly follows the established standards and adjustment specifications to make targeted optimizations to the current agent process orchestration scheme. After the scheme adjustment is completed, the system directly adopts the optimized new orchestration scheme to guide the orderly execution of various business tasks, realizing that the orchestration strategy follows business changes and autonomously updates operational data.

[0076] This dynamic generation module for orchestration strategies enables autonomous iteration and optimization of the current agent process orchestration scheme. The system can automatically evaluate and optimize the scheme based on historical execution logs without manual intervention to analyze operational data, significantly reducing manual maintenance and strategy optimization costs. Furthermore, it allows the process orchestration scheme to adapt to real-time changes in business development trends, upgrading the traditional "fixed template orchestration" to "state-driven orchestration." This allows for precise optimization of the orchestration process based on business needs and task progress, preventing agent over-execution, execution gaps, or process mismatches, thereby improving agent resource utilization and multimodal task execution efficiency.

[0077] In one possible implementation, the system further includes:

[0078] The self-learning and optimization module is used to iteratively optimize the process orchestration model, the orchestration knowledge graph, the scenario adaptation rules, and the intelligent agent collaborative logic library based on the log data.

[0079] To enable the system to continuously iterate and improve itself, and to steadily enhance orchestration accuracy, adaptability, and operational efficiency over long-term operation, fully leveraging the core value of log data, this application also includes a self-learning and optimization module. This module uses log data collected by the orchestration execution feedback module as its core data source to iteratively optimize the core components of the system. Specific optimization targets include the large-scale process orchestration model, orchestration knowledge graph, scenario adaptation rules, and intelligent agent collaborative logic library. The optimization logic of each part is interconnected and works synergistically to ensure the overall performance improvement of the system. For example, for the large-scale process orchestration model, the self-learning and optimization module uses log data to fine-tune and optimize the parameters of the large-scale process orchestration model, correcting deviations in orchestration logic judgment, agent selection, and adaptation evaluation, thereby improving the accuracy, efficiency, and scenario adaptability of the large-scale process orchestration model in generating agent process orchestration solutions, allowing the large-scale process orchestration model to better align with actual business operation rules. For the orchestration knowledge graph, the module supplements the knowledge graph with information extracted from log data, such as new agent attributes, the relationship between multimodal data and agents, and the mapping relationship between scenarios and orchestration solutions. Simultaneously, it updates outdated and unreasonable association data in the knowledge graph, enriching its content. The module enhances the knowledge graph's support for process orchestration. Regarding scenario adaptation rules, it optimizes existing rules' adaptation conditions and priority settings based on log data reflecting actual execution effects in different scenarios, deleting rules unsuitable for current business needs and adding rules that fit new scenarios and requirements, ensuring that scenario adaptation rules accurately match the core needs of various business scenarios. For the intelligent agent collaboration logic library, the module optimizes the intelligent agent execution order, data flow path, collaboration triggering conditions, and abnormal branch handling logic based on the intelligent agent collaboration operation recorded in log data, resolving issues such as stuttering, conflicts, and delayed responses during collaboration, and improving the smoothness and stability of multi-agent collaborative operation.

[0080] Example 3:

[0081] The workflow of the multimodal intelligent agent process orchestration system provided in this application will be described below through specific embodiments. Figure 2 This is a schematic diagram of a system structure for another multimodal agent process orchestration provided in an embodiment of this application. The system is applied to an edge device and includes: a structured processing module 11, an agent orchestration skill configuration module 12, an agent process orchestration module 13, an orchestration execution feedback module 14, an orchestration strategy dynamic generation module 15, a self-learning and optimization module 16, and a multimodal information acquisition module 17.

[0082] This system is applied to edge devices and supports offline, local, and independent operation. It can complete the entire process orchestration service for intelligent agents without relying on cloud computing power and network environment.

[0083] The device is equipped with a lightweight offline structured large model and a process orchestration large model, enabling data structuring and process orchestration to be completed independently on the local device without relying on cloud network computing power. It is free from external network restrictions, has a low deployment threshold, and a fast running response speed. At the same time, it reduces cloud call costs and network latency, and is suitable for offline office, edge terminal and other network-free use scenarios. It completely solves the problem of intelligent agent work stagnation when the network is interrupted, and ensures the continuity and stability of multimodal intelligent agent process orchestration.

[0084] The intelligent agent orchestration skill configuration module 12 is used to store standardized orchestration knowledge graphs, scenario adaptation rules, intelligent agent collaboration logic libraries, and process adjustment criteria to ensure the standardization and normalization of process orchestration, while improving the scenario adaptability of the orchestration scheme and the rationality of intelligent agent collaboration. Among them, the orchestration knowledge graph is a structured knowledge representation form, including agent attribute data, process orchestration node data, multimodal data and agent association data, and scene-orchestration scheme mapping data, etc., which provide knowledge support for agent selection and process node orchestration. It can help the process orchestration module 13 quickly locate agents and orchestration logic that match the current requirements, avoid ineffective screening, and improve orchestration efficiency. The scene adaptation rules are a set of standardized rules preset based on different application scenarios, including scene type definitions, scene requirement characteristics, the scope of adapted agents, orchestration priorities, etc., which are used to quickly match the corresponding application scenarios based on structured orchestration requirement data, determine the overall direction and constraints of the orchestration scheme, and ensure that the orchestration scheme can adapt to the requirements of specific scenarios. The agent collaboration logic library is a data set that stores the logic related to the collaborative operation between agents. It is used to define the interaction rules and operating mechanisms when multiple agents work together, including agent execution order, data flow path, collaboration trigger conditions, and abnormal branch handling logic, etc. Through this data, the responsibilities and collaboration methods of each agent can be clarified, avoiding agent collaboration chaos and ensuring that the process orchestration scheme can be executed smoothly. The process adjustment guidelines are pre-defined, standardized rules used to guide adjustments to existing orchestration schemes. These guidelines include data such as adjustment trigger conditions, adjustment scope, adjustment priority, and post-adjustment collaboration logic adaptation rules. The adjustment trigger conditions clarify when adjustments to the orchestration scheme are necessary, such as changes to structured orchestration requirements, updates to scenario adaptation rules, agent state anomalies, the matching degree between the current agent's process orchestration scheme and the current business, agent resource utilization, and task execution risks. The adjustment scope specifies the details of the adjustment, such as adjusting only the agent's execution order, modifying only the data flow path, or adjusting the entire target agent selection. The adjustment priority determines which aspects should be adjusted first when multiple adjustment requirements exist, ensuring the adjusted scheme prioritizes core needs. The post-adjustment collaboration logic adaptation rules ensure that the adjusted agent collaboration logic matches the updated requirements and rules, preventing collaboration chaos after the adjustment. The standardized data storage and management described above facilitates the standardization of process orchestration, reduces the complexity of manual intervention, and enhances the reusability and scalability of the orchestration scheme. When adding new scenarios or intelligent agents in the future, only the corresponding data set needs to be updated, without modifying the core architecture of the system.

[0085] In one possible implementation, the collaborative execution logic includes: agent execution order, data flow path, collaborative triggering conditions, and exception branch handling logic. Among them, the execution order of intelligent agents refers to the sequential execution of their respective tasks by multiple target intelligent agents according to a preset logical order, specifying which intelligent agent executes first and which executes last. For example, in a multimodal data processing scenario, the data parsing intelligent agent is executed first, followed by the data verification intelligent agent, and finally the data storage intelligent agent. The data flow path refers to the data transmission path and method between intelligent agents, specifying how the output data of the previous intelligent agent is transmitted to the next intelligent agent, as well as the format, frequency, and priority of data transmission, to ensure efficient and accurate data flow between intelligent agents. The collaborative triggering condition refers to the preconditions for triggering collaborative work between intelligent agents. For example, when the data parsing intelligent agent completes data parsing and outputs qualified data, it triggers the data verification intelligent agent to start working. If the parsed data is unqualified, subsequent intelligent agents are not triggered, and exception handling is initiated directly. The exception branch handling logic refers to the preset handling rules when an exception occurs during the process execution, such as intelligent agent execution failure, data transmission interruption, or requirement mismatch. For example, when an intelligent agent fails to execute, a backup intelligent agent is started to replace it. If there is no backup intelligent agent, the current process is terminated and an exception prompt is output, while the exception information is recorded to provide a basis for subsequent investigation and optimization.

[0086] During the process orchestration, the specific workflows of each module in this system are as follows:

[0087] The multimodal information acquisition module 17 can integrate various acquisition units, including image acquisition units such as cameras and scanners, audio acquisition units such as microphones and recording devices, text acquisition units such as keyboard input modules and document reading modules, and sensor acquisition units such as temperature sensors, humidity sensors, and vibration sensors. The corresponding type of acquisition unit can be flexibly activated according to the needs of the actual application scenario. Multimodal data is acquired through each acquisition unit and output to the structured processing module 11.

[0088] In one possible implementation, during the acquisition process, the multimodal information acquisition module 17 can perform preliminary screening and preprocessing on the acquired raw data, removing invalid data such as blurry images, meaningless audio, and garbled text, and unifying the data format, such as converting images of different formats to JPG format and audio to WAV format. Then, according to the preset acquisition frequency and data storage rules, the processed multimodal data is output to the structured processing module 11 to ensure that the structured processing module 11 can acquire valid data in a timely manner, providing support for subsequent process arrangement.

[0089] The structured processing module 11 inputs the acquired multimodal data into a lightweight offline structured processing model. This model performs integrated offline recognition, classification, feature extraction, and semantic / visual analysis on the input multimodal data, transforming the multimodal features into structured orchestration data, which is then output to the agent process orchestration module 13.

[0090] The agent process orchestration module 13 receives structured orchestration requirement data output by the structured processing module 11 and inputs this structured orchestration requirement data into the large-scale process orchestration model. The process orchestration model processes the structured orchestration requirement data to generate an agent process orchestration scheme, including:

[0091] First, based on the input structured orchestration requirement data, the corresponding target scene adaptation rules are matched from the agent orchestration skill configuration module 12. For example, the input structured orchestration requirement data is parsed to extract core requirements, such as processing objectives, constraints, scene types, and data specifications. Based on the parsed core requirements, the corresponding target scene adaptation rules are matched from the agent orchestration skill configuration module 12.

[0092] Next, based on the matched target scenario adaptation rules and combined with the orchestration knowledge graph, multiple target agents that can meet the current task requirements are selected from the orchestration knowledge graph. The function, attributes and applicable scope of each target agent are clarified to ensure that the selected agents can cover all processing links of the current requirements.

[0093] Based on the intelligent agent collaboration logic library, and combined with the selected target intelligent agents, the collaborative execution logic and data flow relationship of multiple target intelligent agents are determined to ensure that each intelligent agent can work together smoothly and avoid execution conflicts or data gaps.

[0094] Based on the established collaborative execution logic and data flow relationships, multiple target intelligent agents are logically assembled to generate a complete intelligent agent process orchestration scheme.

[0095] Determine the compatibility evaluation parameters between the intelligent agent process orchestration scheme and the current business; wherein, the compatibility evaluation parameters include one or more of the following information: the matching degree between the current intelligent agent process orchestration scheme and the current business, the intelligent agent resource utilization rate, and the task execution risk;

[0096] Based on the adaptation evaluation parameters and process adjustment criteria, the agent process orchestration scheme is adjusted so that the current task can be executed according to the adjusted agent process orchestration scheme.

[0097] In one possible implementation, the adjustment of the agent process orchestration scheme includes one or more of the following: orchestration node optimization, agent collaboration mode adjustment, and task execution order change. Orchestration node optimization refers to optimizing the execution nodes corresponding to each agent in the scheme, including adding necessary execution nodes, deleting redundant nodes, and optimizing node execution parameters to ensure that the function of each node accurately matches business requirements and improves the scheme's execution efficiency. Agent collaboration mode adjustment refers to changing the collaborative interaction mode between multiple target agents, including adjusting the communication methods, data transmission formats, and collaboration triggering conditions between agents to solve problems such as poor agent collaboration and delayed response. Task execution order change refers to adjusting the order in which tasks corresponding to each agent are executed, optimizing the execution logic based on business requirements and agent resource usage, avoiding task conflicts, and improving resource utilization.

[0098] The orchestration and execution feedback module 14 generates and issues execution instructions corresponding to the agent process orchestration scheme, enabling the generated scheme to directly drive various agents to perform their tasks in an orderly manner. The execution instructions generated by the orchestration and execution feedback module 14 are executable scheduling instructions derived from the final determined agent process orchestration scheme. Each instruction defines the specific tasks that each target agent needs to complete, clarifies data interaction requirements and start / stop control conditions, and accurately corresponds to all execution logic within the orchestration scheme. The orchestration and execution feedback module 14 can issue execution instructions via targeted push according to a unified communication protocol or in batches based on the agent distribution status, ensuring timely and stable instruction transmission.

[0099] To fully retain information throughout the entire process lifecycle and facilitate subsequent problem tracing, process review, and model iteration optimization, the orchestration execution feedback module 14 in this application, in addition to issuing execution instructions, can also acquire operational status data, process progress data, and final task completion result data generated by each target agent during task execution, while summarizing various abnormal information that occurs during process execution. The orchestration execution feedback module 14 saves all collected feedback data as log data for the agent process orchestration scheme. For example, this log data includes, but is not limited to, one or more of the following: orchestration requirement data, agent operational status, task execution progress, execution process data, and execution results. Orchestration requirement data refers to the structured requirements corresponding to the initial stage of this process orchestration. Agent operational status records the working conditions of each target agent participating in the task. Task execution progress reflects the completion level of the overall orchestration process and individual branch processes. Execution process data records various data generated during the process, such as data interaction content between agents, resource call status, and parameter change information. Execution results save the final completion status of the entire orchestration task, business output content, and relevant information on task achievement.

[0100] The dynamic orchestration strategy generation module 15 can comprehensively analyze and deeply examine log data through the large-scale process orchestration model. From the orchestration requirement data, agent running status, task execution progress, execution process data, and execution results contained in the logs, it determines the corresponding adaptation evaluation parameters between the currently used agent process orchestration scheme and the actual business operations. After obtaining the adaptation evaluation parameters, the large-scale process orchestration model retrieves the process adjustment guidelines stored internally in the agent orchestration skill configuration module 12, and strictly follows the established standards and adjustment specifications to make targeted optimizations to the current agent process orchestration scheme. After the scheme adjustment is completed, the system directly adopts the optimized new orchestration scheme to guide the orderly execution of various business tasks, realizing that the orchestration strategy follows business changes and autonomously updates operational data.

[0101] Meanwhile, the self-learning and optimization module 16 iteratively optimizes the process orchestration model, orchestration knowledge graph, scenario adaptation rules, and intelligent agent collaborative logic library based on log data.

[0102] Through the coordinated operation of seven modules, a closed-loop system for multimodal intelligent agent process orchestration, consisting of "data acquisition, information processing, intelligent orchestration, dynamic adjustment, execution feedback, and self-learning optimization," has been constructed. This breaks down the barriers of independent operation of each module in traditional intelligent agents, achieves deep integration of orchestration and execution, and significantly improves the intelligence and automation level of multimodal intelligent agent process orchestration.

[0103] Example 4:

[0104] This application also provides a process orchestration method based on a multimodal intelligent agent process orchestration system according to any of the above embodiments. Figure 3 A schematic diagram of a process orchestration provided in this application embodiment, the process including:

[0105] S301: Acquire multimodal data.

[0106] S302: Obtain an agent process orchestration scheme based on the multimodal data through a multimodal agent process orchestration system.

[0107] The process orchestration method provided in this application can be applied to edge devices, such as mobile phones and computers.

[0108] Since the principle of the process arrangement method for solving specific technical problems has been described in the above embodiments, please refer to the above embodiments 1-3 for details, and will not be elaborated here.

[0109] Example 5:

[0110] Please see Figure 4 , Figure 4This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application, such as... Figure 4 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0111] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0112] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0113] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0115] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0116] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0117] Example 6:

[0118] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to perform the following steps:

[0119] Acquire multimodal data;

[0120] Based on the multimodal data, an intelligent agent process orchestration scheme is obtained through a multimodal intelligent agent process orchestration system.

[0121] Since the principle of the computer-readable storage medium in solving the problem is similar to the process arrangement method described above, the implementation of the computer-readable storage medium can be found in the embodiments of the method, and repeated details will not be repeated.

[0122] The storage media described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory.

[0123] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this application can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0124] Although this application has been described in detail above with general descriptions and specific embodiments, some modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of this application fall within the scope of protection claimed in this application.

Claims

1. A multimodal intelligent agent process orchestration system, characterized in that, The system is applied to an end-side device, and the system includes: The structured processing module is equipped with a lightweight offline structured processing model, which is used to transform the acquired multimodal data into structured orchestration requirement data through the structured processing model. The intelligent agent orchestration skill configuration module is used to store standardized orchestration knowledge graphs, scene adaptation rules, and intelligent agent collaborative logic libraries; The intelligent agent process orchestration module deploys a lightweight, offline process orchestration model. This model, based on the structured orchestration requirement data and combined with the intelligent agent orchestration skill configuration module, generates an intelligent agent process orchestration scheme. The process of generating the intelligent agent process orchestration scheme includes: Based on the structured orchestration requirement data, corresponding target scenario adaptation rules are matched from the agent orchestration skill configuration module; according to the target scenario adaptation rules and the orchestration knowledge graph, multiple target agents to be invoked are determined; based on the agent collaboration logic library, the collaborative execution logic and data flow relationship of the multiple target agents are determined; and according to the collaborative execution logic and data flow relationship, the multiple target agents are logically assembled to generate an agent process orchestration scheme.

2. The system according to claim 1, characterized in that, The system also includes: a multimodal information acquisition module; The multimodal information acquisition module is used to acquire the multimodal data and output it to the structured processing module.

3. The system according to claim 1, characterized in that, The collaborative execution logic includes: agent execution order, data flow path, collaborative triggering conditions, and abnormal branch handling logic.

4. The system according to claim 1, characterized in that, The intelligent agent orchestration skill configuration module is also used to store standardized process adjustment criteria; The process by which the agent process orchestration module generates the agent process orchestration scheme through the process orchestration big model also includes: Determine the compatibility evaluation parameters between the intelligent agent process orchestration scheme and the current business; wherein, the compatibility evaluation parameters include one or more of the following information: the matching degree between the current intelligent agent process orchestration scheme and the current business, intelligent agent resource utilization rate, and task execution risk; Based on the adaptation evaluation parameters and the process adjustment criteria, the agent process orchestration scheme is adjusted to execute the current task according to the adjusted agent process orchestration scheme.

5. The system according to claim 4, characterized in that, The methods for adjusting the intelligent agent process orchestration scheme include one or more of the following: optimizing orchestration nodes, adjusting the intelligent agent collaboration method, and changing the task execution order.

6. The system according to claim 1 or 4, characterized in that, The system also includes: The orchestration execution feedback module is used to generate and issue execution instructions corresponding to the intelligent agent process orchestration scheme.

7. The system according to claim 6, characterized in that, The orchestration execution feedback module is also used to obtain log data of the execution of the agent process orchestration scheme; wherein, the log data includes, but is not limited to, one or more of the following information: orchestration requirement data, agent running status, task execution progress, execution process data, and execution results.

8. The system according to claim 7, characterized in that, The system also includes: The orchestration strategy dynamic generation module is used to determine the compatibility evaluation parameters between the current agent process orchestration scheme and the current business based on the log data through the process orchestration big model, and to adjust the current agent process orchestration scheme based on the compatibility evaluation parameters and the process adjustment criteria, so as to execute the current task according to the adjusted agent process orchestration scheme.

9. The system according to claim 7, characterized in that, The system also includes: The self-learning and optimization module is used to iteratively optimize the process orchestration model, the orchestration knowledge graph, the scenario adaptation rules, and the intelligent agent collaborative logic library based on the log data.

10. A process orchestration method based on the system described in any one of claims 1-9, characterized in that, The method includes: Acquire multimodal data; Based on the multimodal data, an intelligent agent process orchestration scheme is obtained through a multimodal intelligent agent process orchestration system.

Images