Multi-agent cooperation performance evaluation method and medium

Abstract

The invention discloses a multi-agent cooperation performance evaluation method and a medium. The method comprises the following steps: determining a task structure index according to task splitting information and task scheduling execution information; performing time sequence analysis and behavior pattern recognition according to the communication information among the intelligent agents to determine a cooperative communication index; determining a task consistency index according to the sub-task execution information and the main task execution information of each agent; determining a time resource efficiency index according to the task execution time information of each agent and the execution capability information of each agent; and determining an evaluation result of the multi-agent cooperation performance according to the task structure index, the cooperative communication index, the task consistency index and the time resource efficiency index. According to the invention, evaluation is carried out from multiple dimensions according to related information of multi-agent cooperation, the comprehensiveness of the evaluation result is improved, and the accuracy of the evaluation result is further improved.

Description

Technical Field

[0001] This invention relates to the field of multi-agent cooperation technology, and in particular to a method and medium for evaluating the performance of multi-agent cooperation. Background Technology

[0002] With the development of artificial intelligence technology, multi-agent systems (MAS) have become a key driver for solving complex problems and achieving high levels of automation and intelligent collaboration. Whether in industrial manufacturing, smart city management, financial risk control, or medical diagnosis, MAS demonstrates unprecedented application potential due to its inherent distributed, autonomous, and collaborative advantages. Especially when dealing with complex tasks such as intelligent financial contract review that require the integration of multi-domain knowledge, adherence to multi-step processes, and multi-party collaboration, MAS can effectively simulate the collaborative model of human expert teams, significantly improving work efficiency and decision-making accuracy.

[0003] However, the complexity of Multi-Agent Systems (MAS) also presents significant evaluation challenges. A truly efficient MAS requires not only outstanding performance from each individual agent but also efficient, reasonable, and high-quality collaboration among them. Currently, multi-agent collaboration evaluation typically employs a combination of methods: Hard-coded rule-based and statistical measurement methods: Pre-setting a strict set of rules and thresholds to count and statistically analyze specific events in the logs (such as error codes or specific message types). For example, calculating the number of messages per second or the number of task failures. Evaluation methods based on macroscopic performance indicators: Primarily focusing on the performance of the system's final output, including task completion rate, task completion time, throughput, and resource consumption. Evaluation methods based on simulation and reinforcement learning: Testing and evaluation are conducted in a controlled simulation environment, especially in the field of reinforcement learning, where agents learn and optimize their collaboration strategies through interaction with the environment. For example, running the MAS in a simulated environment and evaluating its performance in different scenarios through multiple experiments; comparing the performance differences of different collaboration strategies or agent architectures in the simulation environment. Log aggregation and visualization tools: Log management systems (such as ELK Stack) are used to aggregate log data, perform time-series and trend analyses, and visually display agent activities and message flows through dashboards and charts. Human expert evaluation: Domain experts invest significant time and effort in manually reviewing massive amounts of agent communication logs and internal decision-making processes, and conduct qualitative evaluations based on their extensive experience.

[0004] Current industry evaluations of multi-agent systems (MAS) largely focus on simulated task performance or output evaluation, which presents the following problems: "Black box" problem: Many MAS operate like a black box, making it difficult for developers to understand their internal collaboration mechanisms, decision-making chains, and information flow. When problems arise, it's hard to pinpoint which agent, which link, or the collaboration itself is flawed. Limitations of traditional evaluation methods: Existing evaluation methods often focus on macro-level performance indicators (such as task completion time and accuracy) but fail to reveal the quality of the collaboration process between agents. For example, a task may be completed, but there may be a large amount of redundant communication, unreasonable division of labor, or inefficient conflict resolution. Unstructured data processing problem: Agent collaboration logs typically contain a large amount of communication content in natural language and internal state descriptions. This complex unstructured data is difficult to analyze and evaluate effectively using traditional tools. High dependence on human experience problem: Evaluating complex multi-agent collaborations often requires domain experts to review logs and manually analyze them for extended periods, which is inefficient and easily influenced by subjectivity. This makes system optimization lack a scientific basis and hinders scalability. Summary of the Invention

[0005] This invention provides a method and medium for evaluating the performance of multi-agent cooperation, in order to solve the problem of inaccurate evaluation results of multi-agent cooperation performance.

[0006] According to one aspect of the present invention, a method for evaluating the performance of multi-agent cooperation is provided, comprising:

[0007] Determine task structure indicators based on task breakdown information and task scheduling and execution information;

[0008] Based on the communication information between the agents, time-series analysis and behavioral pattern recognition are performed to determine the collaborative communication indicators;

[0009] Determine task consistency indicators based on the subtask execution information and main task execution information of each intelligent agent;

[0010] Determine time resource efficiency indicators based on task execution time information and execution capability information of each intelligent agent;

[0011] The evaluation results of multi-agent cooperation performance are determined based on the task structure index, the cooperative communication index, the task consistency index, and the time resource efficiency index.

[0012] According to another aspect of the present invention, an evaluation apparatus for the performance of multi-agent cooperation is provided, comprising:

[0013] The task structure index determination module is used to determine task structure indexes based on task splitting information and task scheduling and execution information.

[0014] The collaborative communication index determination module is used to determine collaborative communication indices based on time-series analysis and behavioral pattern recognition of communication information between agents.

[0015] The task consistency index determination module is used to determine the task consistency index based on the subtask execution information and the main task execution information of each intelligent agent.

[0016] The time resource efficiency index determination module is used to determine the time resource efficiency index based on the task execution time information and the execution capability information of each intelligent agent.

[0017] The overall evaluation module is used to determine the evaluation results of multi-agent collaboration performance based on the task structure index, the collaborative communication index, the task consistency index, and the time resource efficiency index.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multi-agent cooperative performance evaluation method according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the multi-agent cooperative performance evaluation method according to any embodiment of the present invention.

[0023] The technical solution of this invention improves the comprehensiveness of the evaluation results by evaluating them from multiple dimensions based on relevant information of multi-agent collaboration, thereby improving the accuracy of the evaluation results.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a method for evaluating the performance of multi-agent cooperation according to an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the closed-loop implementation path for the entire evaluation process;

[0028] Figure 3 A flowchart of another method for evaluating the performance of multi-agent cooperation according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the task structure indicator evaluation process;

[0030] Figure 5 A flowchart of another method for evaluating the performance of multi-agent cooperation according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the collaborative communication metrics evaluation process;

[0032] Figure 7 This is a schematic diagram of the evaluation process for the collaborative performance of multiple agents;

[0033] Figure 8 This is a schematic diagram of the structure of a multi-agent cooperation performance evaluation device provided in an embodiment of the present invention;

[0034] Figure 9 This is a schematic diagram of the structure of an electronic device that implements the multi-agent cooperative performance evaluation method of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "candidate," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Figure 1 This invention provides a flowchart of a method for evaluating the performance of multi-agent cooperation. This embodiment is applicable to situations where the accuracy of evaluation results for optimizing the effectiveness and performance of multi-agent cooperation is improved. This method can be executed by a multi-agent cooperation performance evaluation device, which can be implemented in hardware and / or software and can be configured in a server with computing power. Figure 1 As shown, the method includes:

[0038] S110. Determine the task structure indicators based on the task splitting information and task scheduling execution information.

[0039] The target task is input into the multi-agent control system (MCS), which generates a task breakdown list and schedules multiple agents to execute subtasks. Based on the task breakdown list, task breakdown information is determined, including at least subtask information, subtask allocation information, and task complexity information. The MCS's execution logs during the target task execution are acquired. These logs include at least task scheduling logs, agent execution time logs, and A2A communication logs between agents. Based on these logs, task scheduling and execution information is determined, including at least subtask execution progress and time information.

[0040] Task structure metrics are used to comprehensively evaluate an MCS system, specifically including the rationality of the MCS system's design decisions, such as the rationality of task decomposition and division of labor, as well as the efficiency of task distribution and scheduling. The MCS acts as the "command center" of the MAS, responsible for the macro-level decomposition, scheduling, resource allocation, and process orchestration of tasks. The evaluation of the MCS primarily focuses on the rationality of its design decisions (such as division of labor and task decomposition) and the efficiency of task distribution and scheduling. For example, task structure metrics include efficiency parameters for task decomposition, task distribution, and overall scheduling determined based on task decomposition information and task scheduling execution information. These efficiency parameters can be determined based on the logical information corresponding to task decomposition, task distribution, and overall scheduling, and are not limited to specific content. In other words, task structure metrics are evaluated from the perspective of the overall structural aspects of multi-agent collaboration.

[0041] In one feasible embodiment, S110 includes:

[0042] The task matching degree is determined based on the task processing capabilities of each agent and the task allocation information.

[0043] The task allocation balance is determined based on the task allocation information of each agent.

[0044] The level of agent participation is determined based on the agent information corresponding to the execution of each sub-task.

[0045] Determine task coverage based on the matching results between task breakdown information and task completion information;

[0046] The redundant task rate is determined by analyzing the proportion of the same task being repeatedly executed by different agents based on task scheduling and execution information.

[0047] Determine the rationality of the collaboration process based on task scheduling and execution information;

[0048] Determine the task decomposition completeness rate based on task breakdown information and original target task information;

[0049] Determine the information of failed tasks based on the task scheduling and execution information, and determine the task failure rate based on the information of failed tasks.

[0050] The integration quality is determined based on the original target task information and sub-task execution information;

[0051] Task structure indicators are determined based on task matching degree, task allocation balance, agent participation, task coverage, redundant task rate, rationality of collaboration process, completeness of task decomposition, task failure rate, and integration quality.

[0052] For example, task matching degree is used to represent the degree of matching between the capabilities of each agent and the assigned tasks. For instance, the task processing capability of each agent is determined based on the attribute information of each agent, the task assignment information of each agent is determined based on the task scheduling and execution information, the task matching degree of each agent is determined based on the matching degree between the attributes of the assigned task information and the corresponding agent's task processing capability, and the total task matching degree is determined by combining the task matching degrees of all agents.

[0053] Task allocation balance is used to represent the rationality of the amount of tasks or execution time undertaken by each agent. For example, based on task scheduling execution information, the sub-task information and execution time information executed by each agent are determined. The quantity balance is determined based on the standard deviation of the number of sub-tasks executed by each agent. The time balance is determined based on the standard deviation of the total duration of the sub-tasks executed by each agent. The task allocation balance is determined based on the quantity balance and time balance. For example, the average of the quantity balance and time balance can be used as the task allocation balance.

[0054] Agent participation is used to represent the number of agents actually executing subtasks during the execution of each subtask, that is, the number of agents actually active within each subtask instance or time period. For example, the execution record information of each agent during the execution of each subtask is determined based on the task scheduling execution information; the number of active agents corresponding to each subtask is determined based on the execution record information; the ratio of active agents to the total number of agents is determined based on the ratio of the number of active agents to the total number of agents; and the agent participation is determined based on the sum or average of the active agent ratios corresponding to all subtasks.

[0055] Task coverage is used to represent the degree of matching between task breakdown results and task completion results. For example, the main task is broken down into a list of subtasks based on task breakdown information, the execution result of each subtask is determined based on task scheduling and execution information, the number of completed subtasks is determined based on the execution result of each subtask, and the task coverage is determined by the ratio of the number of completed subtasks to the total number of subtasks in the subtask list.

[0056] Redundancy rate is used to represent the proportion of the same task being repeatedly executed by multiple agents. For example, the number of agents executing the same subtask is determined based on the task execution log in the task scheduling execution information; the redundancy rate of the subtask is determined based on the ratio of the number of subtask agents to the total number of agents; and the redundancy rate is determined based on the sum or average of the redundancy rates of all subtasks.

[0057] The rationality of the collaborative process is used to represent the proportion of cyclical or invalid execution processes that exist during the process of multiple agents executing sub-tasks. For example, based on the task execution log in the task scheduling execution information, the number of agents repeatedly executing the same task at different time points and the number of invalid agents whose instructions generated during task execution do not match the corresponding sub-task information are determined. The rationality of the collaborative process is then determined based on the ratio of the sum of the number of repeated agents and the number of invalid agents to the total number of agents.

[0058] Task decomposition completeness rate represents the proportion of the total requirements corresponding to the original target task information that are covered by the sub-task information derived from the original target task information. In other words, it indicates whether the sub-tasks derived from the original target task cover all key requirements. For example, the task decomposition completeness rate is determined by determining the list of key sub-tasks corresponding to the original target task information, determining the list of sub-tasks based on the task decomposition information, and determining the coverage of the list of key sub-tasks by the list of sub-tasks.

[0059] The task failure rate is used to represent the proportion of subtasks that return exceptions or timeouts during execution. For example, subtasks that return failures or timeouts are identified from the task execution logs in the task scheduling execution information and recorded as failed task information. The task failure rate is then determined based on the ratio of the number of failed task information subtasks to the total number of subtasks.

[0060] Integration quality is used to represent whether the final output covers the results of all subtasks. For example, the original semantic information of the original target task is determined based on the original target task information, the subtask semantic information of the output results of all subtasks is determined based on the subtask execution information, and the integration quality is determined based on the degree of matching between the subtask semantic information and the original semantic information.

[0061] Task structure indicators are determined based on a comprehensive analysis of factors including task matching degree, task allocation balance, agent participation, task coverage, redundant task rate, collaborative process rationality, task decomposition completeness rate, task failure rate, and integration quality. For example, these factors can be normalized, and the average value after normalization can be used to determine the task structure indicators. Alternatively, these factors can be input into a large language model, which then outputs the final task structure indicators.

[0062] S120. Based on the communication information between the agents, perform time-series analysis and behavior pattern recognition to determine the collaborative communication indicators.

[0063] A2A communication is a manifestation of dynamic collaboration between intelligent agents, reflecting the communication information between them. The evaluation of A2A mainly focuses on the quality of communication, behavioral patterns, and the smoothness of task handover.

[0064] Collaborative communication metrics are used to evaluate the communication efficiency between multiple agents. By analyzing real-time A2A communication logs and agent-internal logs, time-series analysis and behavioral pattern recognition are performed based on runtime logs, communication logs, subtask lists, and results. The evaluation is completed by constructing a task dependency graph based on the subtask list, A2A message details, and communication logs.

[0065] In one feasible embodiment, S120 includes:

[0066] The total number of communication messages and the retry message ratio are determined based on the communication information between the agents.

[0067] The waiting time is determined based on the timing analysis results of the communication information between the agents.

[0068] Collaborative communication metrics are determined based on the total number of communication messages, the retry message ratio, and the waiting time.

[0069] For example, the total number of communication messages is used to represent the total number of messages exchanged between agents. For instance, the total number of message entries in an A2A communication log can be used as the total number of communication messages.

[0070] The retry message ratio is used to represent the number of retry messages sent between agents. For example, the message tag corresponding to each retry message is determined, and the ratio of the number of messages with that message tag in the A2A communication log to the total number of messages is used as the retry message ratio.

[0071] Waiting time is used to represent the duration of waiting for other agents to complete a scheduled task. For example, by constructing a task dependency graph, recording the start and end times of each task, calculating the time difference between the completion of all prerequisites and the start of execution of each task, and using the sum of the time differences of all tasks as the waiting time.

[0072] The collaborative communication metric is determined by combining the total number of communication messages, the retry rate, and the waiting time. For example, the total number of communication messages, the retry rate, and the waiting time can be normalized, and the collaborative communication metric can be determined based on the average of these normalized values. Alternatively, the total number of communication messages, the retry rate, and the waiting time can be input into a large language model, which will then output the final collaborative communication metric.

[0073] S130. Determine the task consistency index based on the sub-task execution information and main task execution information of each intelligent agent.

[0074] Among them, the task consistency index is used to represent the consistency of the output of all subtasks and the main task. For example, if the output information of any subtask is not very similar to the output information of the main task, it will have a greater impact on task consistency.

[0075] For example, the semantic description information of each agent's subtask execution information is determined, the semantic description information of the main task execution information is determined, the similarity between the semantic description information of the subtasks and the semantic description information of the main task is determined, and a task consistency index is determined based on the similarity. For example, the sum or average of the similarities can be directly used as the task consistency index, or the number of subtasks whose similarity between the semantic description information of the subtasks and the total semantic description information of the task is less than a similarity threshold can be determined, and the task consistency index is determined based on the ratio of the number of such subtasks to the total number of subtasks.

[0076] In one feasible embodiment, S130 includes:

[0077] The style consistency parameters are determined based on the output information of each agent's subtask and the output information of the main task.

[0078] The redundancy output rate is determined based on the similarity matching results between the subtask execution information and the main task execution information of each intelligent agent.

[0079] Based on style consistency parameters and redundancy output rate, determine task consistency metrics.

[0080] For example, LLM (Large Language Model) reviews the consistency of the outputs of all subtasks and the main task. LLM directly provides style consistency parameters (e.g., 0-100 points). [0,100]. The LLM, acting as a review expert, analyzes the tone, terminology, professionalism, and writing style of all texts to determine whether they maintain consistency throughout the document or meet specific style requirements. The input information of the large language model includes: the output text of all subtasks, the final main task output text, and the expected style index; the output information of the large language model includes: style consistency scores and corresponding inconsistency details and adjustment suggestions, which are used as style consistency parameters.

[0081] The redundancy output rate is determined based on the logical information of the redundancy output rate; the redundancy output rate is determined based on the large language model; the final redundancy output rate is determined based on the redundancy output rate logical evaluation result and the redundancy output rate large language model evaluation result, or either the redundancy output rate logical evaluation result or the redundancy output rate large language model evaluation result is used as the final redundancy output rate.

[0082] For example, in NLP text extraction, the proportion of duplicate or invalid content in the similarity matching results between the subtask execution information and the main task execution information of each agent is used as the logical evaluation result of the redundancy output rate. That is, the same or nearly duplicated segments in the subtask execution information and main task execution information of each agent are detected by string matching and similarity algorithm (such as Levenshtein distance) and are regarded as invalid content. The first ratio is determined by the ratio of the number of characters in invalid content to the total number of characters in the corresponding agent's subtask execution information, and the second ratio is determined by the ratio of the number of characters in invalid content to the total number of characters in the main task execution information. The combined result of the first ratio and the second ratio is used as the logical evaluation result of the redundancy output rate.

[0083] The Prompt Engineering (LEM) evaluation method, based on a large model, analyzes the text content, outputs a redundancy score (e.g., 0-100), and explains the reasons for its output. The LEM redundancy output score is as follows: [0,100]. LLM Evaluation Method: LLM can perform semantic deduplication and content validity judgment. It can identify not only literal repetition, but also semantic repetition (i.e., using different words to express the same meaning), as well as invalid information (content that is irrelevant, unnecessary, or misleading to the task objective). Input information of the large language model: the final output text, and the output of related subtasks; corresponding output information: redundancy rate score, specific redundant or invalid content information, and optimization suggestions. The output information of the large language model is used as the redundancy output rate evaluation result of the large language model.

[0084] The style consistency parameter and redundancy output rate can be used as input information into a large language model, which will then output the final task consistency index; alternatively, the style consistency parameter and redundancy output rate can be used directly as the task consistency index.

[0085] S140. Determine the time resource efficiency index based on the task execution time information and the execution capability information of each agent.

[0086] The time resource efficiency index is used to represent the degree of matching between the time and execution capability of each agent in executing tasks. For example, the time consumption of each subtask is determined based on the task execution time information of each agent; the capability parameters of each agent are determined based on the execution capability information of each agent; the complexity of each subtask is determined based on the subtask information; the efficiency parameter corresponding to each subtask is determined based on its time consumption, complexity, and the capability parameters of the agents participating in that subtask; and the time resource efficiency index is determined based on the sum or average of the efficiency parameters of all subtasks. For instance, the time consumption, complexity, and capability parameters of each subtask are input into a large language model, which then determines the efficiency parameter of each subtask.

[0087] In one feasible embodiment, S140 includes:

[0088] The average time of each subtask is determined based on the time information from distribution to execution of the corresponding subtask for each agent.

[0089] Based on the execution time and capability information of the corresponding subtasks of each agent, the resource redundancy rate is determined; whereby the resource redundancy rate is used to represent the ratio of agents with the same capability being repeatedly scheduled.

[0090] The time resource efficiency index is determined based on the average time spent on subtasks and the resource redundancy rate.

[0091] Specifically, the logical information of the average time of a subtask is the average time from distribution to execution. The logical evaluation result of the average time of a subtask is determined based on the logical information corresponding to the average time of the subtask. The time information from distribution to execution of the subtask for each agent is used as input information and input into the large language model to obtain the evaluation result of the large language model of the average time of the subtask. The average time of the subtask is taken as the combined result of the logical evaluation result of the average time of the subtask and the evaluation result of the large language model of the average time of the subtask.

[0092] An example logical evaluation method: The average time taken for each subtask is determined according to the following formula: Specifically, the sum of the differences between the completion time of all tasks and the time it takes for the MCS to assign the task to the agent is determined. The ratio of this total time to the total number of tasks is used as the logical evaluation result of the average subtask time. This indicates the time it takes for the MCS to assign tasks to the agents. This indicates the time it takes for the agent to report the completion of the task.

[0093] Large Language Model Evaluation Method: Based on the average time taken, LLM evaluates its reasonableness (e.g., 0-100 points) and combines it with task complexity to score the time efficiency of LLM subtasks. [0, 100]. LLM Evaluation Method: LLM analyzes the time consumption of each subtask and, combined with an understanding of the semantic complexity of the task, judges whether the time consumption is reasonable. It can identify abnormal time consumption (too long or too short) and attempt to analyze its causes (e.g., low agent processing efficiency, complex input data, slow external service response, etc.). Input information of the large language model: detailed MCS scheduling logs, agent internal execution logs (including task start / end times), and corresponding output information: average time consumption, time consumption efficiency score, abnormal time consumption analysis, and optimization suggestions.

[0094] The logical information of resource redundancy rate indicates whether multiple agents with the same capability are repeatedly scheduled. The logical evaluation result of resource redundancy rate is determined based on the logical information corresponding to resource redundancy rate. The execution time information and capability information of the corresponding sub-tasks of each agent are used as input information and input into the large language model to obtain the large language model evaluation result of resource redundancy rate. The resource redundancy rate is the combined result of the logical evaluation result of resource redundancy rate and the large language model evaluation result of resource redundancy rate, or any one of the results.

[0095] An exemplary logical evaluation method involves: statistically analyzing agents with identical capabilities to determine whether they were all assigned the same task within the same time period, or whether any agents remained idle for extended periods despite possessing the required and available capabilities. The resource redundancy rate is then determined by combining the scheduling records of the MCS with the busy status of the agents. For example, the number of first agents repeatedly assigned the same task is determined based on the execution time information of each agent's corresponding subtask; the number of second agents with the same capabilities among those first agents is determined based on the capability information of each agent's corresponding subtask; and the resource redundancy rate is determined based on the ratio of the number of second agents to the number of first agents.

[0096] Large Language Model Evaluation Method: LLM outputs a resource optimization score (e.g., 0-100 points) based on analysis of agent capabilities, task requirements, and scheduling logs. LLM Resource Optimization Score [0, 100]. LLM Evaluation Method: LLM can identify agents with similar capabilities in the system and analyze their utilization in task allocation. It can determine whether there is unnecessary agent parallelism (multiple agents are launched when a single agent is sufficient to handle the task), or a large number of idle agents (leading to resource waste), or uneven load causing some resources to be overloaded while others are idle. The input information of the large language model includes: agent capability description, MCS scheduling log, agent internal workload log; the corresponding output information is: resource optimization score, identified resource waste patterns, and specific optimization suggestions.

[0097] The average time consumption and resource redundancy rate of subtasks are numerically normalized, and the time resource efficiency index is determined based on the average value after normalization. Alternatively, the average time consumption and resource redundancy rate of subtasks can be input into a large language model, which will then output the final time resource efficiency index.

[0098] S150. Determine the evaluation results of multi-agent cooperation performance based on task structure indicators, cooperative communication indicators, task consistency indicators, and time resource efficiency indicators.

[0099] The task structure metrics, collaborative communication metrics, task consistency metrics, and time resource efficiency metrics are numerically normalized, and the evaluation result of the multi-agent collaborative performance is determined based on the average value after normalization. Alternatively, the task structure metrics, collaborative communication metrics, task consistency metrics, and time resource efficiency metrics can be input into a large language model, which then outputs the final evaluation result of the multi-agent collaborative performance.

[0100] For example, the evaluation results of multi-agent collaboration performance include scoring results and indicator visualization results. Specifically, this includes: summarizing indicators of various dimensions to perform LLM scoring; and generating reports, scoring radar charts, anomaly tracking, and other outputs based on indicators of various dimensions. The evaluation results not only include quantitative indicators, but also provide detailed explanations, problem identification, and specific optimization suggestions through natural language.

[0101] In one feasible embodiment, after determining the evaluation results of multi-agent cooperation performance based on task structure metrics, cooperative communication metrics, task consistency metrics, and time resource efficiency metrics, the method further includes:

[0102] Adjust the multi-agent configuration and task scheduling strategy based on the evaluation results.

[0103] Existing methods suffer from a severe disconnect between evaluation and scheduling; the evaluation results of agent cooperation fail to serve as effective feedback to guide agent scheduling, thus failing to achieve a closed-loop evaluation process. This invention proposes a path to achieve a closed-loop evaluation process, such as... Figure 2 The diagram illustrates the closed-loop implementation path of the entire evaluation process, which includes: each agent collecting logs (execution, allocation, communication) after task execution; calculating collaborative evaluation results through the evaluation engine; outputting an evaluation report and feeding back inefficiencies to the MCS (inefficiencies such as redundant tasks, role monopolization, etc., can be determined based on various dimensions of indicators in the evaluation results); establishing an MCS scheduling feedback channel, constructing a self-feedback optimization mechanism based on reinforcement learning, and feeding back the MCS evaluation results to the MCS to optimize the inefficiencies found in the evaluation; the MCS adjusting agent configuration and task partitioning strategy based on reinforcement learning; triggering the next round of collaboration, and cyclically optimizing.

[0104] This embodiment achieves a closed-loop evaluation path throughout the entire process by optimizing and adjusting the agent scheduling based on the evaluation results, thereby improving the scalability and effect feedback of the multi-agent collaborative performance evaluation.

[0105] The technical solution of this embodiment improves the comprehensiveness of the evaluation results by evaluating them from multiple dimensions based on relevant information of multi-agent collaboration, thereby improving the accuracy of the evaluation results.

[0106] Figure 3This is a flowchart illustrating another method for evaluating the performance of multi-agent cooperation provided by an embodiment of the present invention. This embodiment further refines the process for determining the task structure indicators in the above embodiments. For example... Figure 3 As shown, the method includes:

[0107] S310. Based on the logical information of task matching degree, task allocation balance, agent participation degree, task coverage, redundant task rate, reasonableness of collaboration process, task decomposition completeness rate, task failure rate, and integration quality, determine the corresponding first type of logical evaluation result.

[0108] Optionally, based on the logical information of task matching, the logical evaluation result of task matching is determined as the first type of logical evaluation result, including:

[0109] The logical information of task matching degree is the degree of matching between the agent's capabilities and the assigned tasks. It determines the total number of capabilities required for each agent to execute the assigned sub-tasks, as well as the number of matching capabilities that match its own capabilities in the tasks assigned to each agent. The individual matching degree of the agent is determined based on the ratio of the number of matching capabilities to the total number of capabilities. The task matching degree is determined based on the weighted sum of the individual matching degrees of all agents and their corresponding weights. The weights are determined based on the number of sub-tasks assigned to each agent or the complexity of the task.

[0110] For example, the individual matching degree of each agent is determined according to the following formula:

[0111] The formula for determining task matching degree is: , This represents the number of subtasks processed by the i-th agent or the task complexity. This represents the total number of agents. Matching is measured by the degree of overlap between predefined capability labels and task type labels (such as the Jaccard similarity coefficient).

[0112] Optionally, based on the logical information of task allocation balance, the logical evaluation result of task allocation balance is determined as the first type of logical evaluation result, including:

[0113] The logical information for task allocation balance is whether the amount of tasks undertaken or the execution time of each agent is reasonable. The task load rate is determined based on the number of tasks, the task idle rate is determined based on the execution time, and the logical evaluation result of task allocation balance is determined based on the sum or average of the task load rate and the task idle rate.

[0114] For example, determine the standard deviation of the number of tasks. , This represents the number of tasks assigned to the i-th agent. This represents the average number of tasks performed by all agents; a smaller standard deviation indicates better load balance. The agent load rate is determined using the following formula: That is, the load rate of the i-th agent is the ratio of the number of tasks assigned to the agent to the maximum number of tasks actually processed by the agent. The task load rate is determined based on the standard deviation of the number of tasks and the agent load rate.

[0115] Determine the standard deviation of the task time: ,in This represents the total execution time of the i-th agent. This represents the average execution time of all agents; the smaller the standard deviation of task time, the better the balance. The agent idle rate is determined using the following formula: The idle rate of the i-th agent is the ratio of the difference between the total execution time of all agents and the total execution time of the i-th agent to the total execution time. The task idle rate is determined based on the task time standard deviation and the agent idle rate.

[0116] Optionally, based on the logical information of agent participation, the logical evaluation result of agent participation is determined as the first type of logical evaluation result, including:

[0117] The logical information for agent participation is the types and proportions of agents actually executing subtasks in the statistical logs. The number of actually active agents is determined by counting the number of unique agent IDs that leave any execution records (such as receiving messages, sending messages, starting / ending tasks, and generating internal logs) in the logs within a certain task instance or time period. The logical evaluation result of agent participation is determined based on the ratio of the number of actually active agents to the total number of agents.

[0118] Optionally, based on the logical information of task coverage, the logical evaluation result of task coverage is determined as the first type of logical evaluation result, including:

[0119] The logical information for task coverage represents the degree of matching between the expected task breakdown list and the completed task list. Specifically, the expected task breakdown list is obtained through DSL parsing. The set of actually completed subtasks, i.e., the completed task list, is calculated based on the subtask IDs marked as "completed" in the logs. The logical evaluation result of task coverage is determined based on the overlap between the expected task breakdown list and the completed task list. For example, the number of overlapping tasks between the expected task breakdown list and the completed task list is determined, and the logical evaluation result of task coverage is determined based on the ratio of the number of overlapping tasks to the expected number of tasks in the expected task breakdown list.

[0120] Optionally, based on the logical information of the redundancy rate, the logical evaluation result of the redundancy rate is determined as the first type of logical evaluation result, including:

[0121] The logical information of the redundancy rate is the proportion of the same task repeatedly executed by multiple agents. Repeated execution is usually identified by precisely matching the task ID, the processed data block ID, or the function / API name and parameters called by the agent. The number of repeated tasks repeatedly assigned to different agents is determined, and the logical evaluation result of the redundancy rate is determined based on the ratio of the number of repeated tasks to the total number of tasks.

[0122] Optionally, based on the logical information of the task decomposition completeness rate, the logical evaluation result of the task decomposition completeness rate is determined as the first type of logical evaluation result, including:

[0123] The logical information for task decomposition completeness rate is whether the subtasks derived from the original target task cover all key requirements. A list of key subtasks for the original target task needs to be predefined. For example, the number of intersecting subtasks between the actual decomposed task list and the list of key subtasks is determined, and the logical evaluation result of the task decomposition completeness rate is determined based on the ratio of the number of intersecting subtasks to the number of key subtasks.

[0124] Optionally, based on the logical information of the task failure rate, a logical evaluation result of the task failure rate is determined as the first type of logical evaluation result, including:

[0125] The logical information for task failure rate is the proportion of tasks that return exceptions or timeouts during execution. Specifically, failed task information is collected by analyzing specific error codes, exception information, or task status flags (such as FAILED, TIMEOUT) in the monitoring logs of the task scheduling execution information.

[0126] For example, the logical evaluation result of the task failure rate is determined according to the following formula:

[0127] ;

[0128] That is, the number of failed tasks is determined based on specific error codes, exception information, or task status flags, and the task failure rate is determined based on the ratio of the number of failed tasks to the total number of tasks.

[0129] Optionally, based on the logical information of integration quality, the logical evaluation result of integration quality is determined as the first type of logical evaluation result, including:

[0130] The integrated quality logical information indicates whether the final output covers all subtask results without omissions or conflicts. Specifically, it involves: determining content completeness: whether the final task output contains all the key data fields or keywords expected by the original target task, and determining the first parameter based on the number of missing key data fields or keywords, for example, by pre-establishing a mapping relationship between the number of missing fields and the first parameter; determining data consistency: if the subtask output is structured data, comparing the final output with the original subtask output for numerical or Boolean values; if they match, the second parameter is determined as the first value; otherwise, the second parameter is determined as the second value; determining format conformity: whether the final task output conforms to a predefined format specification; if it does, the third parameter is determined as the first value; otherwise, the third parameter is determined as the second value. The integrated quality logical evaluation result is determined based on the sum of the first, second, and third parameters.

[0131] In a feasible embodiment, based on the logical information regarding the rationality of the collaboration process, the corresponding first type of logical evaluation result is determined, including:

[0132] Construct a dependency graph between tasks and agents based on task scheduling and execution information;

[0133] Perform topological sorting on the dependency graph to determine the number of loop paths and / or invalid paths;

[0134] The logical evaluation result of the collaboration process rationality is determined based on the number of loop paths and / or invalid paths, and is used as the first type of logical evaluation result.

[0135] The logical information for the rationality of the collaboration process is to indicate whether there are logical loops or invalid paths in the collaboration graph through topological sorting analysis.

[0136] Specifically, the process begins with constructing a dependency graph: A directed graph (dependent graph) is built based on the MCS scheduling logs or agent A2A communication logs (request-response pattern). Next, topology sorting / cycle detection is performed: the dependency graph is topologically sorted. If a cycle exists in the dependency graph (i.e., a complete topology sort cannot be performed), it indicates a logical cycle or potential deadlock. Invalid path detection is also performed: expected path patterns are defined, and then the actual execution path is checked for deviations or inclusion of nodes that should not exist. Finally, the logical evaluation result of the collaborative process's rationality is based on a Boolean value (existence / non-existence of cycles) and / or the number of cycles.

[0137] S320. Based on the input information and prompt words corresponding to task matching degree, task allocation balance, agent participation degree, task coverage, redundant task rate, collaborative process rationality, task decomposition completeness rate, task failure rate, and integration quality, determine the corresponding evaluation results of the first type of large language model.

[0138] Optionally, based on the input information and prompt word information corresponding to the task matching degree, the evaluation result of the large language model is determined, including:

[0139] The LLM reads the task description and agent capability description, and outputs a matching score (e.g., 0-100). The matching score is directly provided by the LLM, for example, MatchScoreLLM(Task, Agent). [0, 100]. The final matching degree can be the average score or weighted average score of all task-agent pairs. The cue word information can be determined based on the logical information of the task matching degree.

[0140] Optionally, based on the input information and prompt word information corresponding to the task allocation balance, the evaluation result of the large language model for task allocation balance is determined, including:

[0141] The input information for the large language model includes: MCS task allocation logs, agent execution time logs, agent capability descriptions, and task complexity estimates (which can be estimated by LLM or preset). LLM comprehensively considers the number of tasks, execution time, task complexity, and agent capabilities to evaluate overall balance (e.g., 0-100 points) and identifies overloaded or idle agents and their reasons. Cue word information can be determined based on the logical information of task allocation balance.

[0142] Optionally, based on the input information and prompt word information corresponding to the agent's participation, the evaluation result of the large language model for agent participation is determined, including:

[0143] Based on its semantic understanding of the logs, LLM outputs an engagement rationality score (e.g., 0-100 points) along with a reasoning. This score emphasizes "effective" engagement rather than simply "activity." (LLM agent engagement rationality score) [0, 100]. LLM analyzes the behavior of all agents to determine which agents should have participated but did not, or whether their participation was effective. Based on the decomposition of the original task and the capabilities of the agents, it infers the role that each agent should play.

[0144] For example, the input information for a large language model includes: a complete task execution log, including A2A communication and internal activity records for each agent, as well as a list of system agents and the original task objective. The corresponding output information includes: a reasonableness score and a natural language explanation. Cue word information can be determined based on the logical information of agent participation.

[0145] Optionally, based on the input information and prompt word information corresponding to the task coverage, the evaluation result of the large language model for task coverage is determined, including:

[0146] LLM (Label Completeness Model) outputs an integrity score (e.g., 0-100) based on its semantic understanding of the original task and the actual completed subtasks, and lists potentially missing or incomplete key steps. LLM Task Integrity Score [0,100]. LLM leverages its domain knowledge and task understanding to infer which semantically necessary subtasks are required to complete the original macro-task. It then compares these subtasks with the actually completed subtasks to identify missing items.

[0147] For example, the input information for a large language model includes: a raw macro-level task description, and detailed descriptions and results of all completed (or claimed to be completed) subtasks. The corresponding output information includes: a completeness score and a natural language explanation. Cue word information can be determined based on logical information about task coverage.

[0148] Optionally, based on the input information and prompt word information corresponding to the redundancy rate, the large language model evaluation result for the redundancy rate is determined, including:

[0149] LLM uses semantic analysis to output a redundancy score (e.g., 0-100 points) and the nature of the redundancy (necessary or unnecessary). This is the LLM redundancy score. [0,100]. LLM Evaluation Method: LLM analyzes the operation logs and communication content of different agents to identify semantically identical or highly overlapping operations, even if they may use different method names. LLM can also determine whether such duplication is necessary (e.g., cross-validation, fault recovery mechanisms) or unnecessary (inefficient).

[0150] For example, the input information of a large language model includes detailed agent internal logs and A2A communication logs. The corresponding output information includes: redundancy score, identified redundant instances, their necessity assessment, and optimization suggestions. Cue word information can be determined based on the logical information of the redundancy task rate.

[0151] Optionally, based on the input information and prompt word information corresponding to the rationality of the collaboration process, the large language model evaluation result of the collaboration process rationality is determined, including:

[0152] LLM outputs a process rationality score (e.g., 0-100). LLM Process Rationality Score [0, 100]. LLM evaluation method: LLM can understand the temporal logic and semantic intent of logs, and can identify not only graph-based loops, but also semantic logical loops (e.g., agent A queries B, B queries A, but neither can provide valid information, resulting in an invalid loop), or redundant back-and-forth communication. It can also determine whether there are signs of deadlock (long wait times, no response).

[0153] For example, the input information for a large language model includes: a complete task execution log (including task status, agent communication sequences, and timestamps). The corresponding output information is: a reasonableness score and a natural language explanation. Prompt word information can be determined based on the logical information regarding the reasonableness of the collaborative process.

[0154] Optionally, based on the input information and prompt word information corresponding to the task decomposition completeness rate, the large language model evaluation result of the task decomposition completeness rate is determined, including:

[0155] LLM directly outputs a completion rate score (e.g., 0-100) based on the original task objective and its understanding of the domain. LLM decomposes the completion rate score. [0,100]. LLM evaluation method: LLM uses its domain knowledge and general principles of task decomposition to infer all necessary subtasks required to complete the original task. Then, it performs a semantic comparison with the actual decomposed subtasks to identify missing items.

[0156] For example, the input information for a large language model includes: a detailed description of the original macro-task, and a list of subtasks actually generated by the MCS (typically from DSL parsing or task scheduling logs). The corresponding output information is: a completeness score and a detailed explanation. Cue word information can be determined based on the logical information of task decomposition and completeness score.

[0157] Optionally, based on the input information and prompt word information corresponding to the task failure rate, the large language model evaluation result is determined, including:

[0158] LLM not only counts failures, but also performs semantic analysis and classification of the causes of failures, and assesses the severity of failures. LLM failure impact score. [0, 100]. LLM evaluation method: LLM deeply analyzes the context of error logs to determine the root cause of failure (whether it is a data problem, an internal logic error of the agent, or an external service interruption) and its impact on the entire task process and the final result.

[0159] For example, the input information for a large language model includes: records from the runtime log regarding task execution status, error messages, and exception stack traces. The corresponding output information includes: the number of failures, the classification of failure reasons, the impact score, and specific suggestions. The prompt words can be determined based on the logical information of the task failure rate.

[0160] Optionally, based on the input information and prompt word information corresponding to the integration quality, the evaluation result of the large language model for integration quality is determined, including:

[0161] LLM performs a semantic evaluation of the quality, completeness, and consistency of the final output, assigning a score (e.g., 0-100). LLM integrates the quality scores. [0, 100]. LLM Evaluation Method: LLM performs semantic analysis and comparison on all inputs, checking: whether the final result fully reflects the key findings of all subtasks; whether there are any omissions, conflicts, or inconsistencies in information during the integration process (e.g., two subtasks reach contradictory conclusions but the final report does not explain or resolve them); and whether the presentation format and content of the final result conform to the original task objectives and user expectations.

[0162] For example, the input information of a large language model includes: the original task objective, the final output of all subtasks (text or structured data), and the final output of MAS. The corresponding output information includes: integrated quality score and specific problem instances.

[0163] S330. Determine the task structure indicators based on the first type of logic evaluation results and / or the first type of large language model evaluation results.

[0164] The task structure index can be determined by using the first-class logical evaluation results of each dimension parameter as the task structure index. For example, the first-class logical evaluation results of each dimension parameter are normalized, and the task structure index is determined based on the average value after normalization. Alternatively, the first-class logical evaluation results of each dimension parameter can be directly displayed in multiple dimensions as the task structure index. Alternatively, the task structure index can be determined by using the first-class large language model evaluation results of each dimension parameter as the task structure index. For example, the numerical results in the first-class large language model evaluation results of each dimension parameter are normalized, and the numerical task structure index is determined based on the average value after normalization. The textual explanation results in the first-class large language model evaluation results of each dimension parameter are used as the explanatory task structure index. The numerical task result index and the explanatory task structure index together constitute the task structure index. Alternatively, the combined result of the first-type logical evaluation results and the first-type large language model evaluation results for each dimension parameter can be used as the task structure indicator. For example, the normalized average of the numerical results from the first-type large language model evaluation results and the first-type logical evaluation results for each dimension parameter can be used as the combined numerical evaluation result for that dimension. The numerical task structure indicator is determined based on the average of the combined numerical evaluation results for each dimension. This is then combined with the textual explanation results from the first-type large language model evaluation results for each dimension parameter as the explanatory task structure indicator. The numerical task result indicator and the explanatory task structure indicator together constitute the task structure indicator. Alternatively, the first-type logical evaluation results and the first-type large language model evaluation results for each dimension parameter can be directly displayed in multiple dimensions as the task structure indicator.

[0165] like Figure 4 The diagram shows the process of evaluating task structure indicators.

[0166] S340. Based on the communication information between the agents, perform time-series analysis and behavior pattern recognition to determine the collaborative communication indicators.

[0167] S350. Determine the task consistency index based on the sub-task execution information and main task execution information of each intelligent agent.

[0168] S360. Determine the time resource efficiency index based on the task execution time information and the execution capability information of each intelligent agent.

[0169] S370. Determine the evaluation results of multi-agent cooperation performance based on task structure indicators, cooperative communication indicators, task consistency indicators, and time resource efficiency indicators.

[0170] The technical solution of this embodiment determines the task structure indicators through a dual-path indicator implementation mechanism, namely, dual-path evaluation based on a large language model and logical information, which improves the completeness and evaluation depth of the task structure indicator determination.

[0171] Figure 5 This is a flowchart illustrating another method for evaluating the cooperative performance of multiple agents according to an embodiment of the present invention. This embodiment further refines the process for determining the cooperative communication indicators in the above embodiments. Figure 5 As shown, the method includes:

[0172] S510. Determine the task structure indicators based on the task splitting information and task scheduling execution information.

[0173] S520. Based on the logical information of the total number of communication messages, the retry message ratio, and the waiting time, determine the corresponding second type of logical evaluation result.

[0174] Optionally, based on the logical information of the total number of communication messages, a logical evaluation result of the total number of communication messages is determined as the second type of logical evaluation result, including:

[0175] The logical information for the total number of communication messages is obtained by analyzing all A2A communication logs and calculating the total number of message exchanges. The total number of message entries in the A2A communication logs is directly counted as the logical evaluation result for the total number of communication messages.

[0176] Optionally, based on the logical information of the retry message ratio, a logical evaluation result of the retry message ratio is determined as the second type of logical evaluation result, including:

[0177] The retry message ratio is a logical information measure of the proportion of message types containing retry tags. These retry tags can be determined based on the specific scenario, such as "retry" or "resend". For example, by parsing specific keywords containing retry tags in the message metadata or message content, the ratio of the number of retry messages corresponding to that specific keyword to the total number of messages is used as the logical evaluation result of the retry message ratio. For instance, the logical evaluation result of the retry message ratio can be determined using the following formula:

[0178] .

[0179] Optionally, based on the logical information of the waiting time, a logical evaluation result of the waiting time is determined as the second type of logical evaluation result, including:

[0180] The logical information regarding waiting time refers to the duration during which a scheduled task waits for other agents to complete, which can be analyzed in conjunction with dependency graphs. For example, by constructing a task dependency graph, recording the start and end times of each task, and calculating the time difference between the completion of all prerequisites and the start of execution for each task, the logical evaluation result of the waiting time can be determined. For instance, the logical evaluation result of the waiting time can be determined based on the difference between the task's start time and the latest completion time of all prerequisites or dependencies.

[0181] S530. Based on the input information and prompt word information corresponding to the total number of communication messages, the retry message ratio, and the waiting time, determine the corresponding evaluation results of the second type of large language model.

[0182] Optionally, based on the input information and prompt word information corresponding to the total number of communication messages, the large language model evaluation result for the total number of communication messages is determined, including:

[0183] LLM can further evaluate the necessity and efficiency of communication based on statistics, rather than just quantity. LLM Communication Efficiency Scoring [0, 100]. In addition to quantity, LLM can also analyze message content redundancy (whether duplicate or unnecessary information is sent), granularity (whether the message is too fine or too coarse), and timeliness. It can identify invalid or inefficient communication patterns.

[0184] For example, the input information for a large language model includes: A2A communication logs. The corresponding output information includes: total number of messages, efficiency score, and specific suggestions. The prompt word information is determined based on the logical information of the total number of communication messages.

[0185] Optionally, based on the input information and prompt word information corresponding to the retry message ratio, the large language model evaluation result is determined, including:

[0186] LLM can analyze the rationality and reasons for retry behavior based on the statistical retry ratio. LLM retry rationality score [0,100]. LLM evaluation method: LLM not only identifies retry messages, but also analyzes the context in which the retry occurs to determine whether the retry is due to temporary network fluctuations, receiver processing failure, or sender logic defects. It evaluates whether the retry mechanism is excessive or insufficient.

[0187] For example, the input information of a large language model includes: A2A communication logs containing retry / resend tags, and related agent internal logs (which may contain retry trigger reasons). The corresponding output information includes: retry ratio, reasonableness score, retry reason analysis, and optimization suggestions. The prompt word information is determined based on the logical information corresponding to the retry message ratio.

[0188] Optionally, based on the input information and prompt word information corresponding to the waiting time, the long-language model evaluation result is determined, including:

[0189] LLM comprehensively analyzes the timestamps and agent states in the logs and provides a blocking wait time efficiency score (e.g., 0-100 points). LLM Blocking Wait Efficiency Score [0, 100]. LLM evaluation method: LLM can identify semantic patterns in logs that indicate that an agent is in a "waiting" state, and, in conjunction with dependencies (whether explicit or semantically implicit), determine whether the waiting is necessary. It can identify unnecessary waiting caused by improper task scheduling, resource contention, or slow agent response.

[0190] For example, the input information for a large language model includes: detailed task scheduling logs, agent internal state logs (such as "waiting" and "processing"), and A2A communication logs (request-response). The corresponding output information includes: key bottlenecks, estimated waiting times, efficiency scores, and specific cause analysis and optimization suggestions. The prompt word information is determined based on the logical information corresponding to the waiting time.

[0191] S540. Determine collaborative communication metrics based on the results of the second type of logical evaluation and / or the results of the second type of large language model evaluation.

[0192] The second-type logical evaluation results of each dimension parameter can be used as collaborative communication indicators. For example, the second-type logical evaluation results of each dimension parameter are normalized, and the collaborative communication indicator is determined based on the average value after normalization. Alternatively, the second-type logical evaluation results of each dimension parameter can be directly displayed in multiple dimensions as collaborative communication indicators. Alternatively, the second-type large language model evaluation results of each dimension parameter can be used as collaborative communication indicators. For example, the numerical results in the second-type large language model evaluation results of each dimension parameter are normalized, and the numerical collaborative communication indicator is determined based on the average value after normalization. The textual explanation results in the second-type large language model evaluation results of each dimension parameter are used as explanatory collaborative communication indicators. The collaborative communication indicators are thus formed by combining the numerical task result indicator and the explanatory collaborative communication indicator. Alternatively, the combined result of the second-type logical evaluation results and the second-type large language model evaluation results for each dimension parameter can be used as the collaborative communication indicator. For example, the normalized average of the numerical results from the second-type large language model evaluation results and the second-type logical evaluation results for each dimension parameter can be used as the combined numerical evaluation result for that dimension. The numerical collaborative communication indicator is determined based on the average of the combined numerical evaluation results for each dimension. This is then combined with the textual explanation results from the second-type large language model evaluation results for each dimension parameter as the explanatory collaborative communication indicator. The combined numerical task result indicator and the explanatory collaborative communication indicator constitute the collaborative communication indicator. Alternatively, the second-type logical evaluation results and the second-type large language model evaluation results for each dimension parameter can be directly displayed in multiple dimensions as the collaborative communication indicator.

[0193] like Figure 6 The diagram shows the evaluation process for collaborative communication metrics.

[0194] S550: Determine the task consistency index based on the subtask execution information and main task execution information of each intelligent agent.

[0195] S560. Determine the time resource efficiency index based on the task execution time information and the execution capability information of each intelligent agent.

[0196] S570. Determine the evaluation results of multi-agent cooperation performance based on task structure indicators, cooperative communication indicators, task consistency indicators, and time resource efficiency indicators.

[0197] The technical solution of this embodiment determines the collaborative communication indicators through a dual-path indicator implementation mechanism, namely, dual-path evaluation based on a large language model and logical information, which improves the completeness and evaluation depth of the collaborative communication indicator determination.

[0198] like Figure 7The diagram illustrates the evaluation process of multi-agent collaboration performance. This invention constructs an intelligent evaluation system based on Large Models (LLMs) MCS. Unlike existing methods, this system fully leverages the powerful natural language understanding, complex reasoning, and generative capabilities of LLMs to comprehensively, deeply, and intelligently evaluate the structural design (MCS perspective) and runtime behavior (A2A perspective) of multi-agent collaboration, providing interpretable insights and optimization suggestions. Specifically, it includes:

[0199] Data-driven evaluation: The three core data sources are the MCS scheduling system (task orchestration, task distribution, and communication scheduling), A2A communication messages, and runtime logs (actual behavioral data).

[0200] LLM as an intelligent evaluation engine: LLM serves as a central intelligent agent, endowing it with the ability to understand, analyze, and judge complex collaborative patterns.

[0201] Logical evaluation engine: Based on non-LLM algorithm logic, evaluates structural metrics such as task coverage, retry rate, and redundancy to assist in the evaluation of large models.

[0202] Explainability and Action Recommendations: The evaluation results not only include quantitative indicators, but also provide detailed explanations, problem identification, and specific optimization suggestions through natural language.

[0203] Modular and highly scalable: The evaluation system is designed with a modular structure, which is easy to integrate into existing platforms and can be flexibly extended to adapt to new agent types, task definitions and evaluation dimensions.

[0204] Compared with existing multi-agent evaluation methods, this invention:

[0205] This paper innovatively proposes evaluation angles and indicators from both a holistic structural perspective and a communication perspective for multi-agent collaboration. It presents a closed-loop evaluation structure that can be embedded in the scheduler to form a "scheduling-evaluation-optimization" closed loop, offering high completeness and automation. Fine-grained evaluation granularity is achieved, including sub-task, agent, and task dependency levels, enabling more precise exposure of collaboration bottlenecks. Focusing on task dependencies, it constructs task dependency graphs and collaboration topologies, allowing analysis of blocked paths and redundant tasks. Leveraging LLM analysis capabilities—task matching degree—it dynamically and semantically drives task allocation analysis. Building upon logical evaluation, it introduces Big Prophet LLM evaluation, employing a dual-track mechanism combining logic and LLM, resulting in strong interpretability and diverse evaluation dimensions. It focuses not only on quantitative accuracy but also on output quality. Language review of style, consistency, and redundant information enhances subjective quality control. Compared to existing offline analysis methods, it achieves real-time evaluation. Real-time collection of MCS scheduling logs for evaluation supports dynamic MCS tuning and adaptation.

[0206] In summary, this invention constructs a quantifiable, traceable, and tunable multi-agent collaborative evaluation system. It evaluates collaborative effectiveness through structured indicators, quantifies performance differences under different models or configurations, provides interpretable analysis and feedback to guide task decomposition strategies and scheduling optimization, helps identify problems such as low matching degree between agent capabilities and tasks, task duplication, and communication redundancy, and realizes a closed-loop system of "evaluation-scheduling-tuning" by connecting to the scheduling system.

[0207] This invention mainly consists of a core functional module of the evaluation system, which specifically includes:

[0208] First, the MCS scheduling system: The MCS acts as the "command center" of the MAS, responsible for the macro-level decomposition, scheduling, resource allocation, and process orchestration of tasks. The evaluation of the MCS mainly focuses on the rationality of its design decisions (such as division of labor and task decomposition) and the efficiency of task distribution and scheduling.

[0209] Second, the log collection and preprocessing module: standardized log structure (execution, communication, task allocation, dependency graph, etc.). Supports task granularity alignment, role classification, timestamp synchronization, etc.

[0210] Third, the logical evaluation engine: Unlike existing methods, the logical evaluation engine of this invention uses topology graph analysis to determine the depth of dependent paths and the complexity of collaborative graphs; it judges the redundancy rate of tasks based on log comparison of task IDs; it uses counting statistics to evaluate agent participation, subtask failure rate, etc.; and it uses scheduling layer timeline analysis to analyze synchronization blocking, execution time, etc.

[0211] Fourth, large-scale intelligent evaluation engine: This invention innovatively introduces a large model and uses the LLM method (Prompt combined with review) to judge the style consistency of the results: It provides all output fragments and performs semantic and structural analysis on the agent's output results based on LLM to judge style consistency.

[0212] Fifth, division of labor matching analysis: Provide a list of capabilities and task assignments, use LLM to perform semantic understanding of the agent's capability descriptions and assigned tasks, and analyze the agent's division of labor matching.

[0213] Sixth, integration quality assessment: Based on LLM, and combined with engineering logic, content matching analysis is performed on the task output results and agent output results to evaluate the integration quality of the MCS system for the comprehensive output results of multiple agents.

[0214] The innovations of this application mainly include: a multi-dimensional collaborative evaluation standard system: Compared with traditional result-oriented evaluation methods, this system innovatively takes the overall structure and agent collaborative communication as evaluation entry points, and proposes for the first time five dimensions of indicators: structure, communication, division of labor, integration, and style consistency, to systematically evaluate the performance of multiple agents. It introduces large-scale models for high-level semantic evaluation and review: deeply integrating the semantic understanding and reasoning capabilities of large-scale models (LLMs). A dual-path indicator implementation mechanism (large-scale model-based evaluation and logic-based evaluation). A data-driven scheduling feedback loop: establishing a direct linkage mechanism between evaluation results and the MCS (Multi-agent Control System) scheduler, realizing adaptive adjustment of task allocation strategies and a closed-loop evaluation process. The integration of structural graphs and process visualization analysis: utilizing task dependency graphs and communication logs, constructing collaborative graphs and time-series flow graphs, intuitively locating collaborative bottlenecks and failure paths from a visual perspective.

[0215] This proposal presents a multi-agent collaborative evaluation system based on a large-model Multi-Agent Control System (MCS), fundamentally changing the evaluation paradigm. It shifts from shallow evaluation based on hard-coded rules and statistics to deep, intelligent, and interpretable insights based on large-model semantic understanding and reasoning. This transforms the MAS from an elusive black box into a transparent system that can be deeply analyzed, continuously learned, and iteratively optimized, thereby accelerating the application and development of MAS in complex real-world scenarios. Compared to existing multi-agent collaborative evaluation methods, this approach offers the following technical advantages:

[0216] The first aspect is semantic understanding and deep reasoning ability:

[0217] Traditional methods have limitations: Existing methods largely rely on keyword matching, regular expressions, or predefined rules for pattern recognition and statistics. They can identify "errors" or "communication volume," but cannot understand the underlying semantic meaning or perform complex causal reasoning. For example, traditional methods can identify high communication volume between two agents, but cannot determine whether it is efficient collaboration or an invalid loop.

[0218] Advantages of this solution: The LLM intelligent evaluation engine supports:

[0219] Understanding Natural Language: Directly parse unstructured text such as dialogues between intelligent agents, internal logs, and task descriptions to understand their true intentions and context.

[0220] Perform domain reasoning: combine general knowledge and domain knowledge learned in pre-training to reason about the semantic matching degree between the task and the agent's capabilities, and determine whether the division of labor is reasonable.

[0221] Identify behavioral patterns: Infer advanced collaboration patterns from complex log sequences, such as conflict resolution processes, negotiation quality, and redundant behaviors, rather than simply counting events.

[0222] Causal analysis: It can infer the root cause of a problem (such as a task failure) from the context of logs, rather than just the reported phenomenon.

[0223] The second aspect is the evaluation granularity:

[0224] Traditional shortcomings: Focus primarily on task-level success rate.

[0225] The advantages of this solution are: its innovation lies in taking the overall structure and agent collaboration and communication as the evaluation entry points, covering the entire process of evaluation from the original task decomposition to sub-task scheduling, communication interaction and final output, and the indicator system is divided into multiple dimensions such as collaboration quality, communication efficiency and result consistency.

[0226] Thirdly, data sources:

[0227] Traditional shortcomings: Manually testing results or task log snippets.

[0228] Advantages of this solution: Unified collection of full-process logs (MCS scheduling combined with Agent communication and execution records).

[0229] Fourth aspect: Evaluation methods:

[0230] Traditional shortcomings: mainly rule matching or manual review.

[0231] The advantages of this solution: a hybrid mechanism combining a logical evaluation engine and an LLM intelligent evaluation engine. This overcomes the shortcomings of traditional methods that rely solely on superficial metrics such as task success rate and call frequency, enabling in-depth analysis of whether task allocation is reasonable, whether results are consistent in style, and whether there is redundancy or omission.

[0232] Fifth aspect: Feedback mechanism:

[0233] Traditional shortcomings: Decoupling of evaluation and scheduling makes it difficult to close the loop.

[0234] Advantages of this solution: Metric feedback to the scheduler enables automatic optimization. The evaluation engine is deeply integrated with the MCS scheduler, allowing key scoring metrics to be fed back to the scheduler, triggering agent configuration and task allocation optimization. This achieves a closed loop of "execution-evaluation-feedback-optimization," supporting multi-round adaptive optimization, which is impossible with traditional static evaluation systems.

[0235] Sixth aspect: Prompt vocabulary list:

[0236] Traditional shortcomings: lack of systematic support.

[0237] The advantages of this solution are: it constructs a standardized and reusable Prompt template library, which supports large models in scoring and correcting the consistency, redundancy, and content completeness of the output results of multiple agents.

[0238] The proposed multi-agent collaboration evaluation system based on large models has enormous commercial value and application prospects because it directly addresses the core pain point of current multi-agent systems (MAS) in their implementation and large-scale application: how to ensure the quality, efficiency, and reliability of complex collaboration.

[0239] Examples of the value brought by this application:

[0240] First, improve business efficiency and operational quality: Reduce manual intervention: By automating the identification and diagnosis of collaboration issues, significantly reduce the time spent on manual troubleshooting and debugging, lowering operating costs. Accelerate task completion: Identify and eliminate collaboration bottlenecks, redundant steps, and unnecessary waiting, thereby shortening task completion cycles and improving business workflow efficiency. Increase task success rate: Timely detection and remediation of collaboration defects reduce task failures or rework, improving the quality and accuracy of final deliverables. For example, in contract review, it ensures that all risk points are effectively identified and integrated, reducing legal and financial risks.

[0241] Second, reduce risks and costs: Early risk warning: Intelligently identify potential collaboration misalignments, deadlocks, or logical vulnerabilities, nipping problems in the bud and avoiding greater business losses. Reduce resource waste: Discover and eliminate duplicate work and ineffective communication between agents, optimize resource allocation, and reduce computing and communication overhead. Improve system stability: Ensure the robustness of MAS in complex and dynamic environments, reducing business interruptions caused by system crashes or abnormal behavior.

[0242] Third, accelerate product launch and innovation: Accelerate R&D iteration: Provides a fast and accurate feedback mechanism, enabling developers to validate designs, identify and resolve problems more quickly, thereby significantly shortening the MAS development cycle and accelerating product launch. Empower complex application development: Enables enterprises to develop and deploy more complex and intelligent multi-agent applications with greater confidence, expanding their business boundaries and innovation capabilities. Optimize decision support: Provides senior management with intelligent insights into system performance, risks, and optimization directions through in-depth analysis of collaboration processes, supporting more informed strategic decision-making.

[0243] Fourth, enhance customer trust and satisfaction: Improve service quality: A more efficient and accurate MAS can provide higher quality services, thereby improving customer satisfaction and loyalty. Establish industry standards: As a new paradigm for evaluating complex intelligent systems, it helps companies establish a technologically leading position in the industry.

[0244] Examples of the application prospects of this application:

[0245] First, in the field of intelligent manufacturing and industrial automation: Production line collaboration optimization: Evaluating the collaborative efficiency of intelligent agents such as robots, AGVs, and PLCs in smart factories to optimize production processes and reduce downtime. Supply chain management: Evaluating the collaborative quality of intelligent agents in order processing, inventory management, and logistics scheduling to improve supply chain resilience and responsiveness.

[0246] Second, in the financial services sector: Intelligent risk control and compliance: Evaluating the efficiency and accuracy of collaboration among multiple intelligent agents (such as data analysis agents, risk identification agents, and compliance review agents) in credit approval, anti-fraud, and compliance checks to ensure consistency and comprehensiveness in decision-making. Robo-advisory and asset management: Evaluating the quality of collaboration among multiple intelligent agents in market analysis, portfolio construction, and risk adjustment to improve the intelligence level of investment decisions.

[0247] Third, in the field of intelligent healthcare and health management: Assisted diagnosis and treatment: Evaluating the collaborative effectiveness of multiple intelligent agents (such as image recognition agents, medical record analysis agents, and medication recommendation agents) in disease diagnosis and treatment plan development to improve the accuracy and personalization of medical decisions. Hospital operations management: Evaluating the synergy of intelligent agents in scheduling, resource allocation, and patient services to optimize hospital operational efficiency.

[0248] Fourth, in the field of smart cities and public services: Intelligent traffic management: Evaluate the collaborative capabilities of intelligent agents such as traffic lights, autonomous vehicles, and emergency dispatch systems in traffic flow optimization and accident handling to improve urban traffic efficiency and safety. Emergency response and disaster management: Evaluate the synergy of multiple intelligent agents in information gathering, decision support, and resource allocation to improve the efficiency and accuracy of emergency response.

[0249] Fifth, in the field of software development and operations (DevOps / AIOps): Intelligent code generation and testing: Evaluate the collaboration quality of multiple AI agents (such as code generation, code review, and test case generation) in the software development process to improve development efficiency and code quality. AIOps collaboration analysis: Evaluate the collaboration of agents in fault detection, root cause analysis, and automated repair to improve the efficiency and reliability of system operations.

[0250] Figure 8 This is a schematic diagram of a multi-agent cooperation performance evaluation device provided in an embodiment of the present invention. Figure 8 As shown, the device includes:

[0251] The task structure index determination module 810 is used to determine task structure indexes based on task splitting information and task scheduling execution information.

[0252] The collaborative communication index determination module 820 is used to determine the collaborative communication index based on the communication information between the agents through time-series analysis and behavioral pattern recognition.

[0253] The task consistency index determination module 830 is used to determine the task consistency index based on the subtask execution information and the main task execution information of each intelligent agent.

[0254] The time resource efficiency index determination module 840 is used to determine the time resource efficiency index based on the task execution time information and the execution capability information of each intelligent agent.

[0255] The overall evaluation module 850 is used to determine the evaluation results of multi-agent cooperation performance based on the task structure index, the cooperative communication index, the task consistency index and the time resource efficiency index.

[0256] The technical solution of this embodiment improves the comprehensiveness of the evaluation results by evaluating them from multiple dimensions based on relevant information of multi-agent collaboration, thereby improving the accuracy of the evaluation results.

[0257] Optional, the task structure indicator determination module is specifically used for:

[0258] The task matching degree is determined based on the task processing capabilities of each agent and the task allocation information.

[0259] The task allocation balance is determined based on the task allocation information of each agent.

[0260] The level of agent participation is determined based on the agent information corresponding to the execution of each sub-task.

[0261] Determine task coverage based on the matching results between task breakdown information and task completion information;

[0262] The redundant task rate is determined by analyzing the proportion of the same task being repeatedly executed by different agents based on task scheduling and execution information.

[0263] Determine the rationality of the collaboration process based on task scheduling and execution information;

[0264] Determine the task decomposition completeness rate based on task breakdown information and original target task information;

[0265] Determine the information of failed tasks based on the task scheduling and execution information, and determine the task failure rate based on the information of failed tasks.

[0266] The integration quality is determined based on the original target task information and sub-task execution information;

[0267] The task structure indicators are determined based on the task matching degree, the task allocation balance, the agent participation degree, the task coverage, the redundant task rate, the rationality of the collaboration process, the task decomposition completeness rate, the task failure rate, and the integration quality.

[0268] Optional, the task structure indicator determination module includes:

[0269] The first logical evaluation unit is used to determine the corresponding first type of logical evaluation result based on the logical information of the task matching degree, the task allocation balance, the agent participation degree, the task coverage, the redundant task rate, the rationality of the collaboration process, the task decomposition completeness rate, the task failure rate, and the integration quality.

[0270] The first large model evaluation unit is used to determine the corresponding evaluation results of the first type of large language model based on the input information and prompt word information corresponding to the task matching degree, the task allocation balance, the agent participation degree, the task coverage, the redundant task rate, the rationality of the collaboration process, the task decomposition completeness rate, the task failure rate, and the integration quality.

[0271] The task structure index determination unit is used to determine the task structure index based on the first type of logic evaluation results and / or the first type of large language model evaluation results.

[0272] Optionally, the first logical evaluation unit includes a sub-unit for logical evaluation of the rationality of the collaborative process, specifically used for:

[0273] Construct a dependency graph between tasks and agents based on the task scheduling and execution information;

[0274] Perform topological sorting on the dependency graph to determine the number of loop paths and / or invalid paths;

[0275] The logical evaluation result of the collaboration process rationality is determined based on the number of loop paths and / or the number of invalid paths, and is used as the first type of logical evaluation result.

[0276] Optional, a collaborative communication metric determination module, specifically used for:

[0277] The total number of communication messages and the retry message ratio are determined based on the communication information between the agents.

[0278] The waiting time is determined based on the timing analysis results of the communication information between the agents.

[0279] The cooperative communication metrics are determined based on the total number of communication messages, the retry message ratio, and the waiting time.

[0280] Optional, the collaborative communication metrics determination module includes:

[0281] The second logic evaluation unit is used to determine the corresponding second type of logic evaluation result based on the logic information of the total number of communication messages, the retry message ratio, and the waiting time.

[0282] The second large model evaluation unit is used to determine the corresponding evaluation result of the second type of large language model based on the total number of communication messages, the retry message ratio, and the input information and prompt word information corresponding to the waiting time.

[0283] The collaborative communication index determination unit is used to determine the collaborative communication index based on the second type of logic evaluation results and / or the second type of large language model evaluation results.

[0284] Optional, the task consistency metric determination module is specifically used for:

[0285] The style consistency parameters are determined based on the output information of each agent's subtask and the output information of the main task.

[0286] The redundancy output rate is determined based on the similarity matching results between the subtask execution information and the main task execution information of each intelligent agent.

[0287] Based on the style consistency parameter and the redundancy output rate, the task consistency index is determined.

[0288] Optional, the time resource efficiency index determination module is specifically used for:

[0289] The average time of each subtask is determined based on the time information from distribution to execution of the corresponding subtask for each agent.

[0290] Based on the execution time and capability information of the corresponding subtasks of each agent, the resource redundancy rate is determined; wherein, the resource redundancy rate is used to represent the ratio of agents with the same capability being repeatedly scheduled.

[0291] The time resource efficiency index is determined based on the average time of the subtasks and the resource redundancy rate.

[0292] Optionally, the device further includes an evaluation feedback adjustment module, which, after determining the evaluation results of the multi-agent cooperation performance based on the task structure index, the cooperative communication index, the task consistency index, and the time resource efficiency index, is specifically used for:

[0293] The multi-agent configuration and task scheduling strategy are adjusted based on the evaluation results.

[0294] The multi-agent cooperation performance evaluation device provided in the embodiments of the present invention can execute the multi-agent cooperation performance evaluation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0295] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations and do not violate public order and good morals.

[0296] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0297] Figure 9 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0298] like Figure 9 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0299] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0300] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the evaluation of the performance of multi-agent cooperative methods.

[0301] In some embodiments, the evaluation of the multi-agent cooperative performance of the method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the evaluation of the multi-agent cooperative performance of the method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the evaluation of the multi-agent cooperative performance of the method by any other suitable means (e.g., by means of firmware).

[0302] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific reference products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0303] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0304] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0305] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0306] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or computing systems that include switching components (e.g., application servers), or computing systems that include front-end components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such back-end, switching, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0307] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0308] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0309] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0310] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for evaluating the performance of multi-agent cooperation, characterized in that, The method comprises: determining a task structure index according to task splitting information and task scheduling execution information; determining a cooperative communication index through timing analysis and behavior pattern recognition based on communication information between agents; determining a task consistency index according to agent subtask execution information and main task execution information; determining a time resource efficiency index according to agent execution time information and agent execution capability information; determining an evaluation result of multi-agent cooperation performance according to the task structure index, the cooperative communication index, the task consistency index, and the time resource efficiency index.

2. The method of claim 1, wherein, Determining a task structure index according to task splitting information and task scheduling execution information comprises: determining a task matching degree according to the processing task capability of each agent and the allocated task information; determining a task allocation balance according to the allocated task information of each agent; determining an agent participation degree according to the agent information corresponding to the execution of each subtask; determining a task coverage rate according to the matching result of the task splitting information and the task completion information; determining a redundant task rate according to the proportion of the same task repeatedly executed by different agents according to the task scheduling execution information; determining a cooperative process rationality according to the task scheduling execution information; determining a task disassembly completeness rate according to the task splitting information and the original target task information; determining a task failure rate according to the failure task information according to the task scheduling execution information; determining an integration quality according to the original target task information and the subtask execution information; determining a task structure index according to the task matching degree, the task allocation balance, the agent participation degree, the task coverage rate, the redundant task rate, the cooperative process rationality, the task disassembly completeness rate, the task failure rate, and the integration quality.

3. The method of claim 2, wherein, Determining a task structure index according to task splitting information and task scheduling execution information comprises: determining a corresponding first-type logical evaluation result according to the logical information of the task matching degree, the task allocation balance, the agent participation degree, the task coverage rate, the redundant task rate, the cooperative process rationality, the task disassembly completeness rate, the task failure rate, and the integration quality, respectively; determining a corresponding first-type large language model evaluation result according to the input information and prompt word information corresponding to the task matching degree, the task allocation balance, the agent participation degree, the task coverage rate, the redundant task rate, the cooperative process rationality, the task disassembly completeness rate, the task failure rate, and the integration quality, respectively; determining a task structure index according to the first-type logical evaluation result and / or the first-type large language model evaluation result.

4. The method of claim 3, wherein, Determining a corresponding first-type logical evaluation result according to the logical information of the cooperative process rationality comprises: constructing a dependency relationship graph between tasks and agents according to the task scheduling execution information; topologically sorting the dependency relationship graph to determine the number of cyclic paths and / or the number of invalid paths; determining a cooperative process rationality logical evaluation result according to the number of cyclic paths and / or the number of invalid paths as a first-type logical evaluation result.

5. The method of claim 1, wherein, According to the communication information between each agent, the timing analysis and behavior pattern recognition are performed to determine a cooperative communication index, including: According to the communication information between each agent, the total number of communication messages and the retry message ratio are determined; According to the timing analysis result of the communication information between each agent, the waiting time is determined; According to the total number of communication messages, the retry message ratio and the waiting time, the cooperative communication index is determined.

6. The method of claim 5, wherein, According to the communication information between each agent, the timing analysis and behavior pattern recognition are performed to determine a cooperative communication index, including: According to the logical information of the total number of communication messages, the retry message ratio and the waiting time, the corresponding second type of logical evaluation result is determined; According to the input information and prompt word information corresponding to the total number of communication messages, the retry message ratio and the waiting time, the corresponding second type of large language model evaluation result is determined; According to the second type of logical evaluation result and / or the second type of large language model evaluation result, the cooperative communication index is determined.

7. The method of claim 1, wherein, According to the agent sub-task execution information and the main task execution information, a task consistency index is determined, including: According to the output information of each agent sub-task and the output information of the main task, a style consistency parameter is determined; According to the similarity matching result of the agent sub-task execution information and the main task execution information, a redundant output rate is determined; According to the style consistency parameter and the redundant output rate, the task consistency index is determined.

8. The method of claim 1, wherein, According to the agent execution task time information and the agent execution capability information, a time resource efficiency index is determined, including: According to the time information of each agent corresponding sub-task from distribution to execution, the average time consumption of sub-task is determined; According to the execution time information and capability information of each agent corresponding sub-task, a resource redundancy rate is determined; wherein, the resource redundancy rate is used to represent the ratio of the same capability agent being repeatedly scheduled; According to the average time consumption of sub-task and the resource redundancy rate, the time resource efficiency index is determined.

9. The method of claim 1, wherein, After determining the evaluation result of the multi-agent cooperation performance according to the task structure index, the cooperative communication index, the task consistency index and the time resource efficiency index, the method further includes: According to the evaluation result, the multi-agent configuration and task scheduling strategy are adjusted.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the processor execute the multi-agent cooperation performance evaluation method in any one of claims 1-8.

Images