Evaluation System and Method for Unmanned Swarm Algorithms Based on Large Model Agent Skill

Abstract

This invention discloses an evaluation system and method for unmanned swarm algorithms based on a large-scale model Agent Skill. The system includes: a large-scale model core module, a Skill orchestrator, eight sub-skill modules, and a knowledge injection module. Natural language understanding and task planning are achieved through a three-layer Prompt project; a state machine is used for multi-Skill scheduling and state transition management; a three-dimensional orthogonal classification space is constructed using the algorithm classification Skill module, and automated evaluation is completed by combining simulation execution and index calculation modules; an analysis and optimization Skill module uses causal reasoning and the PROMETHEE method to achieve root cause identification and improvement scheme ranking; a ROS2 code generation Skill module outputs deployable code; and a knowledge injection module dynamically incorporates domain knowledge to improve evaluation accuracy. This invention achieves end-to-end automated evaluation, interpretable diagnosis, and one-click deployment. It is highly versatile and robust, suitable for the verification and optimization of various unmanned swarm algorithms.

Description

Technical Field

[0001] This invention relates to the field of unmanned system control and artificial intelligence, and in particular to an unmanned swarm algorithm evaluation system and method based on large model Agent Skill. Background Technology

[0002] Cooperative control of unmanned swarm systems (such as drone / unmanned vehicle formations) is a current research hotspot in the fields of artificial intelligence and robotics. The performance of swarm control algorithms directly determines the success rate, efficiency, and robustness of task execution. How to accurately, stably, and automatically evaluate and optimize unmanned swarm algorithms has always been a core challenge in engineering practice and academic research.

[0003] Currently, existing evaluation methods for unmanned swarm algorithms can be mainly divided into the following two categories: (1) Engineering evaluation method based on simulation platform: In engineering applications, especially in the development and verification of swarm algorithms, various simulation platforms are widely used for algorithm evaluation. Mainstream tools include professional swarm simulators (such as SwarmLab), general-purpose robot simulation platforms (such as Gazebo+ROS2), and deep learning simulation environments (such as AirSim / Azure). These methods offer engineering advantages such as intuitive implementation, visualization support, and repeatable execution, and can reflect the algorithm's performance in a simulated environment to a certain extent. However, their limitations are also significant: First, the evaluation process is highly fragmented, with simulation, metric calculation, performance analysis, and code deployment relying on multiple independent tools, lacking an end-to-end automated pipeline; second, the simulation code differs greatly from the actual deployed code (such as ROS2 nodes), making it difficult to directly transfer evaluation results to the physical system; finally, existing platforms lack standardized algorithm classification and metric systems, requiring users to perform extensive manual configuration and secondary development, resulting in low evaluation efficiency and difficulty in reproducing the results.

[0004] (2) Academic evaluation methods based on indicator systems and classification theory: In academic research, researchers are dedicated to constructing unified algorithm classification frameworks and quantitative evaluation metrics. For example, Olfati-Saber (IEEE TAC 2006) classified algorithms according to control objectives (formation flying, swarm flying, consistency); Ren et al. (IEEE CS 2007) analyzed algorithms from a communication topology perspective (centralized, distributed, decentralized); Yan et al. proposed a three-layer metric framework encompassing safety criticality, performance quality, and advanced analysis; and Soria et al. (IROS 2020) defined behavioral metrics such as order metrics, joint metrics, and algebraic connectivity. These methods have clear physical meanings and theoretical support, making them suitable for algorithm mechanism analysis and cross-sectional comparisons. However, its limitations are also significant: First, the classification dimension is singular, lacking a multi-dimensional orthogonal classification system covering "communication architecture - solution paradigm - control objective", resulting in huge performance differences among similar algorithms and incomparable evaluations; Second, the indicator system is not unified, with different studies using different indicator definitions and calculation methods, making cross-study comparisons difficult; Third, the evaluation results only output numerical values ​​and cannot explain the root causes of the problems, heavily relying on manual diagnosis by senior experts; Fourth, it does not provide any actionable suggestions for algorithm improvement, and the optimization process remains in an inefficient trial-and-error mode.

[0005] In recent years, large language models and their Agent Skill frameworks (such as LangChain, AutoGPT, and Claude CodeSkills) have begun to be applied to code generation and task automation, but they face serious challenges in the evaluation of unmanned swarms: general large models lack expertise in swarm control, resulting in unstable code quality; multi-skill collaboration lacks state machine management, leading to high task failure rates; LLM generation and simulation evaluation are disconnected and cannot form a closed loop; diagnosis relies on pattern matching rather than causal reasoning, resulting in inaccurate root cause identification.

[0006] In summary, existing technologies generally suffer from the following problems in the evaluation of unmanned swarm algorithms: (1) Fragmented evaluation process: The simulation, index calculation, analysis and deployment are independent of each other, lacking an end-to-end automated closed loop, resulting in low efficiency, easy errors and difficulty in reproduction; (2) Inconsistent classification and indicator system: The algorithm classification dimension is single, the indicator definition is independent, the measurement results cannot be uniformly converted across platforms and methods, and there is a lack of multi-dimensional comprehensive evaluation capability; (3) Lack of causal explanation and intelligent diagnosis: It only outputs the numerical value of the index, does not explain the root cause of the performance problem, and the diagnosis relies on expert experience, and cannot achieve automated root cause analysis; (4) Lack of improvement suggestions: No optimization schemes such as priority ranking, cost quantification and effect prediction are provided, and algorithm tuning still relies on trial and error; (5) AI-assisted capabilities are disconnected from domain knowledge: Existing general frameworks for large model agents lack domain knowledge for cluster control and reliable state management, making it difficult to handle complex tasks ranging from natural language requirements to automatic evaluation and diagnosis.

[0007] Therefore, there is an urgent need for an automated evaluation technology solution for unmanned cluster algorithms that can automate the evaluation process, make algorithm classification multidimensional orthogonal, make performance diagnosis causal inference, and provide operable optimization suggestions. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an evaluation system and method for unmanned swarm algorithms based on a large model Agent Skill. Through a Master-Agent architecture with 8 sub-Skills, three-dimensional orthogonal classification, causal reasoning, PROMETHEE multi-criteria decision-making, and a standardized index system, the system achieves fully automated evaluation of unmanned swarm algorithms from natural language input, automatic classification, simulation execution, index calculation, causal diagnosis, optimization recommendation to ROS2 code generation.

[0009] To achieve the above objectives, the present invention provides the following solution: An evaluation system for unmanned swarm algorithms based on large-scale model Agent Skill, comprising: The core module of the large model is used for natural language understanding and task planning through the Prompt project; The Skill orchestrator, connected to the core module of the large model, is used to perform multi-Skill scheduling and state transition management in a state machine-driven manner. Eight sub-skill modules are connected to the Skill orchestrator, and the eight sub-skill modules include: The Algorithm Classification Skill module is used to construct a three-dimensional orthogonal classification space based on communication architecture, solution paradigm, and control objective, perform keyword matching and confidence evaluation, and generate algorithm classification codes. The Simulation Execution Skill module is used to perform lightweight discrete unmanned swarm simulation based on the algorithm classification results, and output trajectory and status data. The Indicator Calculation Skill module is used to perform quantitative calculations, comprehensive scoring, and level determination based on a three-layer standardized indicator system and SwarmLab behavioral indicators. The Skill module is analyzed and optimized, with a built-in causal reasoning engine and PROMETHEE recommendation unit, which are used to identify the root causes of performance based on a pre-set causal graph and generate priority-ranked improvement schemes. The ROS2 code generation Skill module is used to automatically generate ROS2 node code that can be directly deployed based on algorithm features and classification results. The Experimental Design Skill module is used to design experimental schemes based on factorial design methods and configure simulation parameter combinations. The Visualization Skill module is used to generate evaluation charts and structured evaluation reports; The basic framework Skill provides four standard interface definitions for intelligent agents: observe, communicate, plan, and actuate, serving as a unified development framework for cluster control algorithms. The knowledge injection module, connected to the core module of the large model, is used to dynamically inject knowledge from the domain knowledge base into the reasoning process of the large model.

[0010] Furthermore, the causal reasoning engine is built into the analysis and optimization skill and is used to identify performance root causes based on a pre-set causal graph model. The causal reasoning uses a domain causal graph containing 13 nodes and 12 causal edges; The root cause identification process includes: adding observational evidence; performing forward chain reasoning; finding causal paths; calculating path strength; updating confidence levels; outputting root causes in descending order of confidence levels; and supporting rule reasoning based on sensor noise, computational bottlenecks, and scalability constraints.

[0011] Furthermore, the PROMETHEE recommendation unit is built into the analysis and optimization skill and is used to generate improved suggestions for priority ranking based on the PROMETHEE II multi-criteria decision-making method. The multi-criteria decision-making method includes: Step 1, Calculate the preference function:

[0012] in, , indicating the scheme and In the guidelines Differences on It is a preference threshold; The function represents the evaluation function for the j-th evaluation criterion, used to map a candidate improvement scheme to a specific score value under that criterion. a b represents two different candidate improvement schemes; Step 2, Overall Preference Index:

[0013] in, It is a standard The weights; Step 3, positive and negative flow:

[0014]

[0015] in, n This represents the total number of candidate improvement schemes participating in the PROMETHEE II ranking comparison; in the formula (n 1) Used to combine a single scheme with the remaining n The positive flow of a given option is obtained by averaging the pairwise preference indices of each option. + (a) with negative flow - (a); Step 4, Net Flow:

[0016] The improvement schemes are prioritized by sorting them from largest to smallest net flow.

[0017] Furthermore, the core module of the large model adopts a three-layer Prompt project structure, including: The role layer is used to define the identity of the Agent, the expert in evaluating unmanned cluster algorithms. The task layer is used to break down the evaluation task into steps such as intent understanding, classification, simulation, indicator calculation, diagnosis, recommendation, and report generation. The constraint layer is used to define the output format and quality requirements for 3D orthogonal classification, standardized metrics, causal inference, and Prometheus sorting.

[0018] Furthermore, the Skill orchestrator implements the following state transitions: IDLE → CLASSIFYING → SIMULATING → EVALUATING → ANALYZING → RECOMMENDING → COMPLETED; and sets an ERROR state for performing recovery retry when a task is abnormal.

[0019] Furthermore, the three-dimensional orthogonal classification dimension of the Skill module in the algorithm classification is: Communication architecture: centralized CEN, distributed DIST, decentralized DEC; Solution paradigms: Geometric Geometry (GEO), Optimization Opt (OPO), Search Src (SRC), Learning LRN (LRN), Game Theory GAM (GAM); Control objectives: formation, clustering, containment, and consistency (sync); The scores for each category are calculated by keyword matching, and the highest value is taken as the classification result. The confidence score is calculated as the score of that category / the total score. The classification coding format is: architecture-paradigm-objective.

[0020] Furthermore, the three-tiered standardized indicator system includes: Key Indicators Success rate, planning time, collision rate; Quality Indicators : Trajectory smoothness; Advanced Indicators Recovery time; The score is based on a combination of order indicators, joint indicators, algebraic connectivity, inter-agent safety, and obstacle safety. The formula for calculating the overall score is:

[0021] Level determination: A score of ≥90 is Grade A, indicating excellent and deployable; 90> ≥80 points is grade B, indicating good performance requiring minor adjustments; 80> A score of ≥70 is grade C, indicating that the score is acceptable and requires optimization; a score of >70 is grade C. A score of 60 or higher is Grade D, indicating that the score is below the target. A score of <60 indicates grade F, meaning unqualified.

[0022] Furthermore, the evaluation system also includes a fallback module, which automatically switches to keyword matching classification, static template code generation, rule engine diagnosis, and structured report output when an LLM anomaly occurs.

[0023] This invention also provides a method for evaluating unmanned cluster algorithms based on large-model agent skills, applied to the aforementioned unmanned cluster algorithm evaluation system based on large-model agent skills, comprising the following steps: The large model core module receives natural language algorithm descriptions and performs natural language understanding and task planning through the Prompt project. The algorithm classification Skill module is invoked to construct a three-dimensional orthogonal classification space based on the communication architecture, solution paradigm, and control objective. Keyword matching and confidence evaluation are performed to generate algorithm classification codes. Call the simulation execution Skill module to perform a lightweight discrete unmanned swarm simulation based on the algorithm classification results, and output trajectory and status data; The Skill module for indicator calculation is invoked, and quantitative calculation, comprehensive scoring, and level determination are completed based on a three-layer standardized indicator system and SwarmLab behavioral indicators. The Skill module is called to analyze and optimize, and the root cause of performance issues is identified based on a pre-defined cause-effect graph. The PROMETHEEII method is then used to generate an improved solution with priority ranking. The ROS2 code generation skill module is called to automatically generate ROS2 node code that can be directly deployed based on algorithm features and classification results. The visualization skill module is invoked to generate evaluation charts and structured evaluation reports, completing end-to-end automated evaluation.

[0024] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, implements the steps of the above-described evaluation method for unmanned cluster algorithms based on large model Agent Skill.

[0025] As can be seen from the above technical solutions, compared with the prior art, the unmanned cluster algorithm evaluation system and method based on large model AgentSkill provided by the present invention has the following beneficial effects: (1) Realize the full-process automation and closed-loop of unmanned cluster algorithm evaluation: Through the collaboration of the large model core, Skill orchestrator and eight sub-skill modules, the end-to-end automatic execution from natural language input, algorithm classification, simulation execution, index calculation, causal diagnosis, optimization recommendation to ROS2 code generation is completed without human intervention, which greatly improves the evaluation efficiency and reproducibility.

[0026] (2) Construct a standardized and high-precision algorithm classification and evaluation system: Based on three-dimensional orthogonal classification and domain knowledge injection, the system can accurately locate unmanned cluster algorithms, unify evaluation indicators and processes, and solve the problems of single classification, chaotic indicators and incomparability across devices in traditional methods.

[0027] (3) Possesses interpretable performance diagnosis and root cause localization capabilities: Through the built-in causal reasoning engine, it automatically traces the source of performance problems based on the domain causal graph, replacing manual expert analysis and realizing the transformation of evaluation results from "numerical output" to "cause known".

[0028] (4) Generate feasible and prioritized optimization solutions: Through the PROMETHEE II multi-criteria decision module, output improvement suggestions sorted by priority, so that algorithm optimization has a basis and a path to follow, significantly reducing debugging costs.

[0029] (5) Eliminate the technical gap between simulation and actual deployment: It can automatically generate ROS2 node code that can be directly compiled and run, realize one-click deployment of evaluation results, and solve the problems of disconnect between traditional simulation and engineering implementation and repeated development.

[0030] (6) Skill collaboration is stable and reliable, and supports exception fault tolerance: The execution process of multiple Skills is uniformly scheduled by the state machine, the task execution success rate is high, the system is robust, and it can meet the needs of continuous and batch evaluation.

[0031] (7) Lowering the barrier to entry and making it suitable for a wider range of users: Relying on large-scale natural language interaction and automated processes, non-professional users can complete high-quality algorithm evaluation without programming or cluster domain knowledge. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a framework diagram of the unmanned cluster algorithm evaluation system based on large model Agent Skill of the present invention; Figure 2 This is a causal graph model for an embodiment of the present invention, illustrating a causal graph model in the field of cluster control, comprising 13 nodes and 12 causal edges; Figure 3 This is a flowchart of the PROMETHEE II multi-criteria decision-making method according to an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] The purpose of this invention is to provide an unmanned cluster algorithm evaluation system and method based on large model Agent Skill, aiming to overcome the shortcomings of existing technologies such as fragmented evaluation process, single algorithm classification dimension, inconsistent indicator system, lack of causal explanation and intelligent diagnosis, lack of improvement suggestions, insufficient domain knowledge of large models, and high threshold for use.

[0036] Regarding question 1: How to build an end-to-end automated evaluation process for cluster algorithms? The existing technology has inconsistent data formats at different evaluation stages, complex dependencies between stages, and the evaluation process needs to support reproducibility. This invention adopts a hierarchical architecture of a master control agent and eight sub-skill modules to realize a file-based data exchange mechanism and provide standardized stage interface definitions.

[0037] Regarding question 2: How to establish a multidimensional orthogonal algorithm classification system? The existing technical dimensions need to maintain mathematical orthogonality, the classification rules need to cover the existing mainstream algorithms, and the classification results need to be mapped to the evaluation strategy. This invention constructs a three-dimensional orthogonal classification space (architecture × paradigm × target), and uses a classification algorithm based on keyword matching and confidence assessment to establish a mapping relationship between classification units and evaluation indicators.

[0038] Regarding question 3: How to implement performance diagnostics based on causal reasoning? Existing technologies require pre-defined causal knowledge in the field of cluster control, causal reasoning needs to handle uncertainty, and diagnostic results need to be interpretable and actionable. This invention constructs a domain causal graph model to implement a rule-based reasoning engine (forward / backward chain) and updates confidence levels by combining observational evidence.

[0039] Regarding question 4: How to generate suggestions for improving priority sorting? Existing technologies require comprehensive consideration of multiple decision-making criteria, and there are trade-offs between solutions. It is recommended that the solutions be adapted to the deployment context. This invention employs the PROMETHEE II multi-criteria decision-making method to construct a problem-solution knowledge base and achieve context-aware priority adjustment.

[0040] Regarding question 5: How to achieve intelligent evaluation based on large model Agent Skill? Existing large-scale technology models lack expertise in cluster control; multi-skill collaboration requires state machine management; Prompt engineering requires systematic design; and knowledge injection needs deep integration with the reasoning process. This invention relates to a hierarchical architecture of Master-Agent + 8 Sub-Skills, implementing SkillOrchestrator state machine management, and adopting a three-layer Prompt engineering strategy (role layer, task layer, constraint layer) to construct an algorithm knowledge base, an indicator knowledge base, and a causal knowledge base.

[0041] Regarding question 6: How to lower the barrier to entry for using the evaluation system? Traditional evaluation systems require specialized programming knowledge, have complex algorithm parameter configurations, and require domain experts to interpret the evaluation results; This invention provides a natural language interaction interface, automated algorithm classification and evaluation configuration, and generates interpretable diagnostic reports and improvement suggestions.

[0042] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, the present invention provides an evaluation system for unmanned cluster algorithms based on large-scale model Agent Skill, comprising: The core module of the large model is used for natural language understanding and task planning through the Prompt project; The Skill orchestrator, connected to the core module of the large model, is used to perform multi-Skill scheduling and state transition management in a state machine-driven manner. Eight sub-skill modules are connected to the Skill orchestrator, and the eight sub-skill modules include: The Algorithm Classification Skill module is used to construct a three-dimensional orthogonal classification space based on communication architecture, solution paradigm, and control objective, perform keyword matching and confidence evaluation, and generate algorithm classification codes. The Simulation Execution Skill module is used to perform lightweight discrete unmanned swarm simulation based on the algorithm classification results, and output trajectory and status data. The Indicator Calculation Skill module is used to perform quantitative calculations, comprehensive scoring, and level determination based on a three-layer standardized indicator system and SwarmLab behavioral indicators. The Skill module is analyzed and optimized, with a built-in causal reasoning engine and PROMETHEE recommendation unit (using the PROMETHEEII multi-criteria decision-making method) to identify performance root causes based on a pre-set causal graph and generate priority-ranked improvement schemes. The ROS2 code generation Skill module is used to automatically generate ROS2 node code that can be directly deployed based on algorithm features and classification results. The Experimental Design Skill module is used to design experimental schemes based on factorial design methods and configure simulation parameter combinations. The Visualization Skill module is used to generate evaluation charts and structured evaluation reports; The basic framework Skill provides four standard interface definitions for intelligent agents: observe, communicate, plan, and actuate, serving as a unified development framework for cluster control algorithms. The knowledge injection module, connected to the core module of the large model, is used to dynamically inject knowledge from the domain knowledge base into the reasoning process of the large model.

[0044] The evaluation system also includes a fallback module, which automatically switches to keyword matching classification, static template code generation, rule engine diagnosis, and structured report output when an LLM anomaly occurs.

[0045] The following is a detailed description of each module of the unmanned cluster algorithm evaluation system based on large model Agent Skill described in this invention: 1. Core Modules of the Large Model The core module of the large model adopts a three-layer Prompt project structure, including: The role layer is used to define the identity of the Agent, the expert in evaluating unmanned cluster algorithms. The task layer is used to break down the evaluation task into steps such as intent understanding, classification, simulation, indicator calculation, diagnosis, recommendation, and report generation. The constraint layer is used to define the output format and quality requirements for 3D orthogonal classification, standardized metrics, causal inference, and Prometheus sorting.

[0046] 2. Skill Arranger The Skill orchestrator implements the following state transitions: IDLE→CLASSIFYING→SIMULATING→EVALUATING→ANALYZING→RECOMMENDING→COMPLETED; It also sets the ERROR status to enable recovery and retrying when a task fails.

[0047] The status descriptions are shown in Table 1: Table 1

[0048] The workflow execution steps are shown in Table 2: Table 2

[0049] 3. Sub-skill modules (1) Algorithm Classification Skill Module The three-dimensional orthogonal classification dimension of the Skill module in the algorithm classification is: Dimension 1, Communication Architecture: Centralized CEN, Distributed DIST, Decentralized DEC; Dimension 2, solution paradigms: geometric geo, optimization opt, search src, learning lrn, game theory gam; Dimension 3, Control Objectives: Formation, Clustering, Encirclement, Consistency (sync); The scores for each category are calculated by keyword matching, and the highest value is taken as the classification result. The confidence score is calculated as the score of that category / the total score. The classification coding format is: architecture-paradigm-objective.

[0050] The three-dimensional orthogonal classification space includes 3×5×4 = 60 classification units; The classification Skill module executes the classification algorithm, employing a three-dimensional orthogonal stepwise classification strategy. It sequentially completes keyword matching, score calculation, confidence assessment, and encoding generation across three dimensions: communication architecture, solution paradigm, and control objective. The specific steps are as follows: S1, Architecture Dimension Classification Steps: First, extract feature keywords from the input algorithm description text, match them with preset keywords for centralized, distributed, and decentralized communication architecture types, and calculate the matching score for each category; select the architecture type with the highest score as the classification result for this dimension, and calculate the corresponding classification confidence level according to the method of "score of this type ÷ total score".

[0051] S2, Paradigm Dimension Classification Steps: Following the classification process and calculation rules of the architecture dimension, classification processing is performed on the solution paradigm dimension; this dimension supports five paradigm types: geometric analysis, optimization, search, learning, and game theory. The paradigm classification results and confidence levels are determined through the same keyword matching and score calculation methods.

[0052] S3, Target Dimension Classification Steps: Continue to use the same classification method as the previous two dimensions to classify the control target dimension; this dimension covers four target types: formation, cluster, encirclement, and consistency. After completing keyword matching and score calculation, output the target classification results and confidence level.

[0053] S4, Classification Code Generation Steps: Combine the classification results obtained from the above three dimensions and generate a unique classification code according to the fixed format of "architecture-paradigm-target" to uniquely identify the algorithm type; for example, the code corresponding to distributed architecture, geometric analysis paradigm, and cluster flight target is dist-geo-flock, so as to facilitate subsequent simulation configuration, indicator selection, and evaluation strategy invocation.

[0054] (2) Indicator Calculation Skill Module The three-tiered standardized indicator system, as shown in Table 3, includes: Key Indicators Success rate (success_rate), planning time (planning_time_ms), collision rate (collision_rate); Quality Indicators : trajectory smoothness; Advanced Indicators : recovery time; The score is based on a combination of order indicators, joint indicators, algebraic connectivity, inter-agent safety, and obstacle safety. The formula for calculating the overall score is:

[0055] Level determination: A score of ≥90 is Grade A, indicating excellent and deployable performance; 90> A score of ≥80 is Grade B, indicating good performance that requires minor adjustments. 80> A score of 70 or higher is grade C, indicating that the score is acceptable and requires optimization. 70> A score of 60 or higher is Grade D, indicating that the score is below the target. A score of <60 indicates grade F, meaning unqualified.

[0056] Table 3 Three-tiered standardization indicators

[0057] The scoring functions are shown in Table 4: Table 4

[0058] (3) Analyze and optimize the Skill module The causal reasoning engine, built into the analysis and optimization skill, is used to identify performance root causes based on a pre-set causal graph model. The causal reasoning employs a domain causal graph containing 13 nodes and 12 causal edges, such as... Figure 2 As shown; the descriptions of nodes and causal edges are shown in Tables 5 and 6. Table 513 Node Definitions

[0059] Table 612 Definitions of Causal Boundaries

[0060] The root cause identification process includes: Evidence addition steps: Iterate through each (variable, value) pair in the observation evidence and add the evidence to the diagnostic engine; Forward chain reasoning steps: Perform forward chain reasoning to deduce new conclusions from known facts; Causal path finding steps: Find all ancestor nodes of the problem in the causal graph, and for each ancestor node, find all causal paths from it to the problem; Causality strength calculation steps: Calculate the path strength (product of causal strengths on edges) for each causal path, and calculate the average of all path strengths for that ancestor node; Confidence update steps: If the ancestor node is in the observed evidence, the observation strength is used; otherwise, the default strength of 0.5 is used. Final confidence = evidence strength × average path strength. Root cause sorting steps: Select nodes with confidence scores greater than a threshold as root causes, and sort them in descending order of confidence score to return the root cause list.

[0061] Rule-based reasoning methods: The system implements two reasoning methods: forward chain and backward chain. The pre-defined diagnostic rules are shown in Table 7.

[0062] Table 7 Pre-set Diagnostic Rules

[0063] The PROMETHEE recommendation unit, built into the analysis and optimization skill, is used to generate improved suggestions for priority ranking based on a multi-criteria decision-making method. like Figure 3 As shown, the multi-criteria decision-making method includes: Step 1, Calculate the preference function:

[0064] in, , indicating the scheme and In the guidelines Differences on It is a preference threshold; The function represents the evaluation function of the j-th evaluation criterion, which is used to map a candidate improvement scheme to a specific score value under the criterion (such as the quantitative scores of the criteria for relevance, confidence, context matching, historical success rate, ease of implementation, etc.). a b represents two different candidate improvement schemes; g j (a) and g j (b) are the evaluation scores of scheme a and scheme b on criterion j, respectively.

[0065] Step 2, Overall Preference Index:

[0066] in, It is a standard The weights; Step 3, positive and negative flow:

[0067]

[0068] in, n This represents the total number of candidate improvement schemes participating in the PROMETHEE II ranking comparison; in the formula (n 1) Used to combine a single scheme with the remaining n The positive flow of a given option is obtained by averaging the pairwise preference indices of each option. + (a) with negative flow - (a); Step 4, Net Flow:

[0069] The improvement schemes are prioritized by sorting them from largest to smallest net flow.

[0070] The criterion weights are configured as shown in Table 8: Table 8

[0071] Context-aware adjustment strategies are shown in Table 9: Table 9

[0072] (4) Basic Framework Skill The basic intelligent agent framework defines four core interfaces, which all cluster control algorithms must implement, as shown in Table 10.

[0073] Table 10

[0074] Table 11 Built-in Algorithm Templates

[0075] The specific meanings of the letter parameters involved in each control law formula in Table 11 are as follows: ①BoidsAgent (Reynolds cluster model) formula:

[0076] Variable description: : The resultant velocity control vector of the intelligent agent; Separation force vector, used to avoid getting too close to neighbors; Alignment force vector is used to align the velocity direction with that of the neighboring vector. Cohesion vector, used to converge toward the neighboring centroid; , , : The weighting coefficients for the three behaviors of separation, alignment, and aggregation.

[0077] Constraints:

[0078] ②ConsensusAgent (Consensus Protocol) formula:

[0079] Variable description: : No. State variables (such as position or velocity) of an agent; The derivative of the state variable with respect to time (i.e., the state update rate). : No. The set of neighbors of an agent; Neighbor Intelligent Agent The corresponding state variables; The negative sign ensures that the agent's state gradually converges to the average state of its neighbors.

[0080] ③PIDAgent (PID control) formula:

[0081] Variable description: : Control output commands (such as speed or acceleration); : The deviation signal between the current state and the desired state; The integral of the deviation over time; The derivative of the deviation with respect to time; , , Proportional, integral, and derivative control gain coefficients.

[0082] ④HexagonFormationAgent (Hexagon Formation Control) formula:

[0083] Variable description: : The control velocity vector of the intelligent agent; The current position of the agent; The target position of the agent within the desired hexagonal formation; Obstacle avoidance force vector; Formation tracking control gain; Obstacle avoidance control gain.

[0084] 4. Knowledge Injection Module The domain knowledge bases used by the knowledge injection module are shown in Tables 12-14.

[0085] Table 12 Problem Definition

[0086] Table 13 Solution Definition

[0087] Table 14 Pre-built Knowledge Base

[0088] This invention establishes a knowledge injection mechanism to dynamically inject professional knowledge from the field of unmanned swarm control into the reasoning process of large models, thereby improving the accuracy and professionalism of algorithm classification, index calculation, and performance diagnosis. Specifically, the algorithm knowledge base stores algorithm classification knowledge from the swarm control field and is invoked in classification tasks; the index knowledge base stores standardized index definitions and threshold parameters and is invoked in evaluation tasks; and the causal knowledge base stores causal graph models and diagnostic rules and is invoked in diagnostic tasks.

[0089] When performing classification tasks, the knowledge injection method is as follows: first, retrieve classification knowledge related to the current input from the algorithm knowledge base, and then construct enhanced prompts based on the retrieval results, injecting content such as domain classification dimensions and typical algorithm references into the prompt information to improve the accuracy of the large model for the three-dimensional orthogonal classification of the unmanned cluster algorithm.

[0090] When performing diagnostic tasks, the knowledge injection method is as follows: first, obtain the corresponding causal graph edge definition and pre-set diagnostic rules from the causal knowledge base, and then integrate the rules and causal relationship structure into the prompts to form an enhanced reasoning context, enabling the large model to complete the identification of performance root causes and fault location based on domain knowledge.

[0091] This invention also includes a RAG enhancement mechanism to introduce external domain knowledge during large model inference to improve output accuracy and professionalism. The specific execution steps are as follows: First, the user query is converted into a vector representation to complete the query embedding; then, the top-k most relevant documents are retrieved from the vector database; finally, the retrieved document content is integrated into the large model hints to complete the context construction.

[0092] Meanwhile, this invention designs dedicated Skill implementation methods for different evaluation stages: The Algorithm Classification Skill takes algorithm description text as input and outputs classification results including communication architecture, solution paradigm, control objective, and confidence level. It primarily performs hint construction, knowledge injection, LLM inference, and result parsing. The ROS2 Generation Skill takes algorithm features and configuration information as input and outputs deployable ROS2 node code, sequentially completing feature extraction, hint construction, LLM generation, and template injection. The Analysis and Optimization Skill takes indicator data as input and outputs diagnostic results and improvement suggestions. It completes performance analysis and solution generation through causal inference, LLM interpretation, and the PROMETHEE recommendation process.

[0093] To ensure the output quality and stability for different tasks, this invention performs targeted selection and parameter configuration for large models: The algorithm classification task uses the glm-5 model with a temperature of 0.1 and a maximum number of tokens of 1024; the code generation task uses the glm-5 model with a temperature of 0.2 and a maximum number of tokens of 4096; the causal diagnosis task uses the glm-5 model with a temperature of 0.3 and a maximum number of tokens of 2048; and the report generation task uses the glm-5 model with a temperature of 0.5 and a maximum number of tokens of 4096.

[0094] In addition, the system has a robust fallback mechanism to handle large model service anomalies and ensure the continuous reliability of the evaluation process: when an LLMError occurs, the system automatically falls back to the rule engine and uses keyword matching to complete the algorithm classification; when a TimeoutError occurs, it switches to static template mode to generate code; when a JSONDecodeError occurs, it enables rule inference to complete performance diagnosis and result output, ensuring stable output of usable results under various abnormal scenarios.

[0095] Compared with existing technologies, this invention enables end-to-end automated operation of the entire unmanned cluster algorithm evaluation process, requiring no human intervention. Evaluation tasks involving 30 agents can be completed in approximately 3 seconds, significantly improving evaluation efficiency. By employing a three-dimensional orthogonal classification method for precise algorithm localization, classification accuracy is effectively improved, reaching 94%. This invention can automatically identify the root causes of performance problems, eliminating reliance on human expert analysis, with a diagnosis time of less than 1 second. It prioritizes improvement suggestions using the PROMETHEE multi-criteria decision method, outputting clear and implementable optimization solutions, achieving an 85% hit rate for Top-5 improvement suggestions. The system adopts a file-based data exchange mechanism, ensuring complete traceability of the evaluation process and 100% reproducibility of results. It can automatically generate directly deployable ROS2 code, with a code generation rate exceeding 90%. Relying on a large model to achieve natural language interaction significantly lowers the system's usage threshold, improving usability for non-expert users by 80%. Through state machine management of multi-skill collaborative execution processes, it supports task anomaly recovery, achieving a task success rate of up to 98%. By deeply integrating domain knowledge of unmanned swarms into the large model reasoning process, the accuracy of reasoning is effectively improved, increasing the diagnostic accuracy rate by 15%. Relying on the automated end-to-end evaluation process driven by the large model, the proportion of manual intervention is reduced by 85%, significantly improving the overall evaluation efficiency.

[0096] The following describes specific embodiments of the evaluation system described in this invention: Example 1: Natural Language-Based Evaluation of Boids Swarm Flight This embodiment uses the ReynoldsBoids swarm flight algorithm, described in natural language, as the evaluation object. The user requirement is to use three rules of separation, alignment, and aggregation to achieve swarm flight of 30 drones through neighbor communication.

[0097] The system first performs intent understanding and task decomposition on the input text, identifying the evaluation task, algorithm type, number of agents, and control objective. Then, it calls the algorithm classification skill to obtain a classification result of dist-geo-flock (distributed-geometric-swarm flight) with a classification confidence of 0.96. Next, it completes simulation configuration and lightweight discrete simulation with 30 agents, a simulation duration of 20 seconds, and a 100m×100m scene. Then, based on a standardized index system, it performs calculations to obtain an order index of 0.998, a joint index of 1.000, a safety index of 0.994, and an algebraic connectivity of 0.85. Finally, it generates an evaluation report with a comprehensive score of 97.1 points and a grade of A, determining that the algorithm is excellent and deployable, and simultaneously outputs a complete evaluation report, directly deployable ROS2 node code, and visualization charts.

[0098] Example 2: Formation Control Evaluation and LLM Diagnosis This embodiment uses the hexagonal formation control algorithm code as input. The system first completes algorithm classification, simulation configuration, and simulation operation according to the process in Embodiment 1, classifying the algorithm as cen-opt-form (centralized-optimized-formation). Subsequently, index calculation and performance analysis are performed, and it is found that the algorithm's success rate is only 85%, which does not meet the ideal standard. Then, the causal reasoning diagnosis module is activated to locate the two root causes: insufficient safety margin and high planning delay, with confidence levels of 89% and 72%, respectively. Based on the PROMETHEEII multi-criteria decision method, the improvement schemes are prioritized and optimization suggestions are output from high to low priority: increasing the collision radius from 2.0m to 2.5m can reduce the collision rate by 80%; optimizing the collision detection algorithm can reduce the planning time by 60%; and performing trajectory smoothing processing can improve the trajectory smoothness by 50%, providing a clear direction for algorithm debugging.

[0099] Example 3: LLM-driven ROS2 code generation This embodiment automatically generates ROS2 node code that can be directly compiled and deployed based on the algorithm class definition and natural language requirements. User requirements include subscribing to the ` / agent_states` topic to obtain neighbor states, publishing to the ` / cmd_vel` topic to output control commands, and dynamically configuring the 100Hz control frequency and parameters. Based on algorithm features and configuration information, the system generates code using ROS2 Skills. The generated code includes a main class `MyFormationAgentNode` inheriting from `rclpy.Node`, an `init()` method for initializing subscriptions, publications, and parameters, a `control_callback()` method for implementing the control loop, a `state_callback()` method for completing state updates, a `publish_command()` method for executing command publication, and the program entry function `main()`. It also provides configurable parameters such as formation spacing, control gain, and maximum speed, outputting complete and runnable ROS2 Python node code.

[0100] Example 4: Multi-turn dialogic assessment This embodiment utilizes multiple rounds of natural language interaction to evaluate and optimize an unmanned swarm algorithm. In the first round, the user requests evaluation of the consistency control algorithm, and the system queries key configuration information. In the second round, the user provides parameters for distributed communication, location consistency, 50 agents, and a 30-second simulation. The system categorizes the algorithm as dist-ana-sync and initiates the simulation. In the third round, the user reports excessive convergence time. The system uses causal analysis to pinpoint the root cause as an insufficient control gain coefficient and suggests adjusting it from 0.3 to 0.5, which improves convergence time by 40%. In the fourth round, after user confirmation, the system automatically generates optimized code and completes a re-evaluation. Comparative results show that the algorithm's convergence time is reduced from 8.5 seconds to 5.1 seconds, and the overall score improves from 92.3 to 96.8, demonstrating significant optimization.

[0101] Example 5: Demonstration of the Fallback Mechanism This embodiment verifies the system's degradation and fault tolerance capabilities when the large model service is unavailable. When the LLM service is unavailable, the system first attempts to call the large model for inference when performing algorithm classification tasks. After catching the LLMServiceUnavailable exception, it automatically switches to degradation mode and enables the rule engine, using keyword matching to complete the algorithm classification. In tasks such as code generation, causal diagnosis, and report generation, the Fallback mechanism is also triggered, using alternative solutions such as static template filling, rule inference, and structured output to replace the large model's capabilities. This ensures that even in extreme scenarios without large model support, the system can still stably output usable classification results, code, diagnostic conclusions, and evaluation reports, ensuring that the evaluation process is not interrupted.

[0102] Furthermore, this invention also provides a method for evaluating unmanned cluster algorithms based on large-model agent skills, applied to the aforementioned evaluation system for unmanned cluster algorithms based on large-model agent skills, comprising the following steps: The large model core module receives natural language algorithm descriptions and performs natural language understanding and task planning through the Prompt project. The algorithm classification Skill module is invoked to construct a three-dimensional orthogonal classification space based on the communication architecture, solution paradigm, and control objective. Keyword matching and confidence evaluation are performed to generate algorithm classification codes. Call the simulation execution Skill module to perform a lightweight discrete unmanned swarm simulation based on the algorithm classification results, and output trajectory and status data; The Skill module for indicator calculation is invoked, and quantitative calculation, comprehensive scoring, and level determination are completed based on a three-layer standardized indicator system and SwarmLab behavioral indicators. The Skill module is called to analyze and optimize, and the root cause of performance issues is identified based on a pre-defined cause-effect graph. The PROMETHEEII method is then used to generate an improved solution with priority ranking. The ROS2 code generation skill module is called to automatically generate ROS2 node code that can be directly deployed based on algorithm features and classification results. The visualization skill module is invoked to generate evaluation charts and structured evaluation reports, completing end-to-end automated evaluation.

[0103] The present invention also provides a computer-readable storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the above-described evaluation method for unmanned cluster algorithms based on large model Agent Skill.

[0104] Matters not covered in this invention are common knowledge.

[0105] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A system for evaluating unmanned swarm algorithms based on large-scale model Agent Skill, characterized in that, include: The core module of the large model is used for natural language understanding and task planning through the Prompt project; The Skill orchestrator, connected to the core module of the large model, is used to perform multi-Skill scheduling and state transition management in a state machine-driven manner. Eight sub-skill modules are connected to the Skill orchestrator, and the eight sub-skill modules include: The Algorithm Classification Skill module is used to construct a three-dimensional orthogonal classification space based on communication architecture, solution paradigm, and control objective, perform keyword matching and confidence evaluation, and generate algorithm classification codes. The Simulation Execution Skill module is used to perform lightweight discrete unmanned swarm simulation based on the algorithm classification results, and output trajectory and status data. The Indicator Calculation Skill module is used to perform quantitative calculations, comprehensive scoring, and level determination based on a three-layer standardized indicator system and SwarmLab behavioral indicators. The Skill module is analyzed and optimized, with a built-in causal reasoning engine and PROMETHEE recommendation unit, which are used to identify the root causes of performance based on a pre-built causal graph and generate priority-ranked improvement schemes. The ROS2 code generation Skill module is used to automatically generate ROS2 node code that can be directly deployed based on algorithm features and classification results. The Experimental Design Skill module is used to design experimental schemes based on factorial design methods and configure simulation parameter combinations. The Visualization Skill module is used to generate evaluation charts and structured evaluation reports; The basic framework Skill provides four standard interface definitions for intelligent agents: observe, communicate, plan, and actuate, serving as a unified development framework for cluster control algorithms. The knowledge injection module, connected to the core module of the large model, is used to dynamically inject knowledge from the domain knowledge base into the reasoning process of the large model.

2. The unmanned cluster algorithm evaluation system based on large-scale model Agent Skill as described in claim 1, characterized in that, The causal reasoning engine, built into the analysis and optimization skill, is used to identify performance root causes based on a pre-set causal graph model. The causal reasoning uses a domain causal graph containing 13 nodes and 12 causal edges; The root cause identification process includes: adding observational evidence; performing forward chain reasoning; finding causal paths; calculating path strength; updating confidence levels; outputting root causes in descending order of confidence levels; and supporting rule reasoning based on sensor noise, computational bottlenecks, and scalability constraints.

3. The unmanned cluster algorithm evaluation system based on large-scale model Agent Skill as described in claim 1, characterized in that, The PROMETHEE recommendation unit, built into the analysis and optimization skill, is used to generate improved suggestions for priority ranking based on the PROMETHEE II multi-criteria decision-making method. The multi-criteria decision-making method includes: Step 1, Calculate the preference function: in, , indicating the scheme and In the guidelines Differences on It is a preference threshold; The function represents the evaluation function for the j-th evaluation criterion, used to map a candidate improvement scheme to a specific score value under that criterion. a b represents two different candidate improvement schemes; Step 2, Overall Preference Index: in, Evaluation criteria The weights; Step 3, positive and negative flow: in, n This represents the total number of candidate improvement schemes participating in the PROMETHEE II ranking comparison; in the formula (n 1) Used to combine a single scheme with the remaining n The positive flow of a given option is obtained by averaging the pairwise preference indices of each option. + (a) with negative flow - (a); Step 4, Net Flow: The improvement schemes are prioritized by sorting them from largest to smallest net flow.

4. The unmanned cluster algorithm evaluation system based on large-scale model Agent Skill as described in claim 1, characterized in that, The core module of the large model adopts a three-layer Prompt project structure, including: The role layer is used to define the identity of the Agent, the expert in evaluating unmanned cluster algorithms. The task layer is used to break down the evaluation task into steps such as intent understanding, classification, simulation, indicator calculation, diagnosis, recommendation, and report generation. The constraint layer is used to define the output format and quality requirements for 3D orthogonal classification, standardized metrics, causal inference, and Promethee II sorting.

5. The unmanned cluster algorithm evaluation system based on large-scale model Agent Skill according to claim 1, characterized in that, The Skill orchestrator implements the following state transitions: IDLE → CLASSIFYING → SIMULATING → EVALUATING → ANALYZING → RECOMMENDING → COMPLETED; and sets the ERROR state for retrying and recovery when a task fails.

6. The unmanned cluster algorithm evaluation system based on large-scale model Agent Skill according to claim 1, characterized in that, The three-dimensional orthogonal classification dimension of the Skill module in the algorithm classification is: Communication architecture: centralized CEN, distributed DIST, decentralized DEC; Solution paradigms: Geometric Geometry (GEO), Optimization Opt (OPO), Search Src (SRC), Learning LRN (LRN), Game Theory GAM (GAM); Control objectives: formation, clustering, containment, and consistency (sync); The scores for each category are calculated by keyword matching, and the highest value is taken as the classification result. The confidence score is calculated as the score of that category / the total score. The classification coding format is: architecture-paradigm-objective.

7. The unmanned cluster algorithm evaluation system based on large-scale model Agent Skill according to claim 1, characterized in that, The three-tiered standardized indicator system includes: Key Indicators Success rate, planning time, collision rate; Quality Indicators : Trajectory smoothness; Advanced Indicators Recovery time; The score is based on a combination of order indicators, joint indicators, algebraic connectivity, inter-agent safety, and obstacle safety. The formula for calculating the overall score is: Level determination: A score of ≥90 is Grade A, indicating excellent and deployable; 90> ≥80 points is grade B, indicating good performance requiring minor adjustments; 80> A score of ≥70 is grade C, indicating that the score is acceptable and requires optimization; a score of >70 is grade C. A score of 60 or higher is Grade D, indicating that the score is below the target. A score of <60 indicates grade F, meaning unqualified.

8. The unmanned cluster algorithm evaluation system based on large-scale model Agent Skill according to claim 1, characterized in that, The evaluation system also includes a fallback module, which automatically switches to keyword matching classification, static template code generation, rule engine diagnosis, and structured report output when an LLM anomaly occurs.

9. A method for evaluating unmanned cluster algorithms based on large-scale model Agent Skill, applied to the unmanned cluster algorithm evaluation system based on large-scale model Agent Skill as described in any one of claims 1-8, characterized in that, Includes the following steps: The large model core module receives natural language algorithm descriptions and performs natural language understanding and task planning through the Prompt project. The algorithm classification Skill module is invoked to construct a three-dimensional orthogonal classification space based on the communication architecture, solution paradigm, and control objective. Keyword matching and confidence evaluation are performed to generate algorithm classification codes. Call the simulation execution Skill module to perform a lightweight discrete unmanned swarm simulation based on the algorithm classification results, and output trajectory and status data; The Skill module for indicator calculation is invoked, and quantitative calculation, comprehensive scoring, and level determination are completed based on a three-layer standardized indicator system and SwarmLab behavioral indicators. The Skill module is called to analyze and optimize, and the root cause of performance issues is identified based on a pre-defined cause-effect graph. An improved solution with priority ranking is generated using the PROMETHEE II method. The ROS2 code generation skill module is called to automatically generate ROS2 node code that can be directly deployed based on algorithm features and classification results. The visualization skill module is invoked to generate evaluation charts and structured evaluation reports, completing end-to-end automated evaluation.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the unmanned cluster algorithm evaluation method based on large model Agent Skill as described in any one of claims 1-8.

Images