Method and system for evaluating collaborative simulation efficiency of unmanned system cluster

Abstract

The invention discloses an unmanned system cluster collaborative simulation efficiency evaluation method and system, and the method comprises the steps: S1, receiving a to-be-evaluated cluster collaborative control algorithm related file and scene description information, and determining an evaluation target; s2, setting probability distribution of random parameters and triggering rules of disturbance events, and generating corresponding parameters and disturbance event configuration data; s3, starting multiple groups of simulation tasks in parallel in a computing environment based on the configuration data, injecting a disturbance event in each simulation, and recording a simulation log; s4, analyzing the simulation logs in batches, extracting key data and calculating a multi-dimensional efficiency index; and S5, carrying out statistical analysis and visualization processing on the multi-dimensional efficiency index, and automatically generating a structured evaluation report. According to the method, the technical problems of statistical significance deficiency, one-sided robustness evaluation, single evaluation dimension and the like in a traditional evaluation method are solved, and efficient reproducible efficiency evaluation is realized.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and in particular to an evaluation method and system for the collaborative simulation performance of unmanned system clusters. Background Technology

[0002] Unmanned system swarm collaborative control technology has broad application prospects in fields such as military reconnaissance, disaster relief, and environmental monitoring. Evaluating the effectiveness of swarm collaborative control algorithms is a crucial step in its practical application. Existing evaluation techniques mainly include two categories: One-Shot Deterministic Simulation (OSDS) and Manual Perturbation Scripting (MPS).

[0003] Single-scenario offline simulation is typically performed by engineers under fixed initial conditions, a defined environmental model, and preset disturbances. This involves conducting one or a small number of offline simulations (usually no more than 10) of the cluster cooperative control algorithm. The simulation platform can be based on ROS / GAZEBO, MATLAB / Simulink, or AirSim, and outputs single performance indicators such as task completion time and trajectory deviation.

[0004] To compensate for the shortcomings of single-scenario offline simulation in assessing robustness, manual script-based perturbation testing introduces scripted perturbation events (such as random node failures and packet loss) into OSDS. Testers manually or semi-automatically inject these perturbations during simulation using ad-hoc scripts. However, the lack of systematicity and limited coverage of the perturbation probability model, coupled with inconsistent script and log formats, makes experimental reproduction difficult and cross-sectional comparison challenging.

[0005] The existing technology has the following significant shortcomings: 1. Single-scene offline simulations typically involve only 5-10 simulations of a fixed scene, resulting in a very small sample size. According to the law of large numbers, the standard deviation of the estimated result is inversely proportional to the square root of the sample size, leading to an excessively wide confidence interval and insufficient statistical significance. The evaluation conclusions are easily affected by random noise and initial bias, and cannot support scientific decision-making.

[0006] 2. Manual script perturbation testing compensates for insufficient robustness assessment by manually injecting a limited number of perturbation events. However, the probability model of the perturbation lacks systematicity and has limited coverage, making it difficult to cope with continuous uncertainty and complex perturbation scenarios. Furthermore, the experimental process is fragmented and lacks parameter versioning, making it difficult to reproduce the experiment and conduct horizontal comparisons.

[0007] 3. Traditional evaluation methods have a single evaluation dimension, focusing on single indicators such as task completion time and track deviation, while ignoring the trade-off between multiple objectives such as system stability, resource consumption and safety, which can easily lead to misjudgments based on limited information.

[0008] 4. The existing evaluation process has a low degree of automation. Experiment configuration, disturbance injection, data acquisition and post-processing all rely on manual operation, which results in high human and time costs. The complexity of processing thousands of simulation data is high, which seriously hinders the rapid iteration and optimization of the algorithm.

[0009] Therefore, there is an urgent need for a technology that can overcome the above-mentioned technical deficiencies and achieve statistically reliable, robust, quantifiable, multi-dimensional coverage, and highly automated performance evaluation of unmanned system cluster collaborative control algorithms. Summary of the Invention

[0010] This application aims to address the technical problems existing in the prior art by providing a method for evaluating the collaborative simulation performance of unmanned system clusters, comprising the following steps: A method for evaluating the collaborative simulation performance of unmanned system clusters includes the following steps: S1: Receive relevant files and scenario descriptions of the collaborative control algorithm of the cluster to be evaluated, and clarify the evaluation objectives; S2: Set the probability distribution of random parameters and the triggering rules of disturbance events, and generate the corresponding parameter and disturbance event configuration data; S3: Based on the configuration data, multiple simulation tasks are launched in parallel in the computing environment. Each simulation dynamically generates the initial state and environment model according to random parameters, injects disturbance events, and records simulation logs. S4: Batch parse simulation logs, extract key data and calculate multi-dimensional performance indicators; S5: Perform statistical analysis and visualization processing on the multi-dimensional performance indicators to automatically generate a structured evaluation report.

[0011] Furthermore, the global random configuration in step S2 includes: M sets of independent parameters were generated using Latin hypercube sampling, where , where the parameter vector ,in For continuous variable dimensions, For discrete variable dimensions; Latin hypercube sampling is used to generate samples that satisfy space-filling properties, and the deviation formula is:

[0012] Where X is the set of sampling points, N is the number of sampling points, and d is the parameter dimension; Generate independent and identically distributed event sequences using a perturbation event injector. Among them, the events And serialize it into an injection file; Generate a summary of parameters and event sequences.

[0013] Further, in step S2, the random parameters include continuous variables and discrete variables, wherein: Continuous variables Support Gaussian distribution Uniform distribution truncated Gaussian distribution ; Discrete variables It supports Poisson (λ) and Emp (pk) distributions and outputs M sets of independent parameters that satisfy Latin hypercube sampling. ; Furthermore, step S3 also includes: before each simulation run, the task scheduler generates a unique hash identifier for this run and writes the identifier into the metadata of the log file.

[0014] Furthermore, the computing environment mentioned in step 3 is: based on container construction, mounting the Gazebo simulation environment and the algorithm node to be evaluated, and injecting the configuration data through environment variables; The simulation's task scheduling uses a container orchestration framework based on resource utilization metrics. The system automatically and elastically scales the number of computing resources, and the scheduling algorithm uses a min-heap priority queue to allocate computing resources.

[0015] Furthermore, the statistical calculation in step S4 also includes: Perform a two-way ANOVA to test for significant differences in algorithm performance; Sensitivity analysis was performed using the Sobol exponent method to quantify various random parameters. The weighting of the impact on the performance index y, where the Sobol exponent method is used to calculate the formula: ; Furthermore, the multi-dimensional performance indicators mentioned in step 4 include system stability indicators, task completion indicators, robustness indicators, and resource consumption indicators, wherein: The system stability index is calculated using the root mean square error:

[0016] Task completion metrics: ; Robustness metrics: including the proportion of node failures For the horizontal axis, Fitted curve with vertical axis and communication interference index; Resource consumption indicators: Average communication traffic per unit Energy consumption per unit , in: This refers to hovering power.

[0017] Furthermore, the statistical analysis described in step 5 includes calculating the sample mean, variance, and 95% confidence interval, and performing a two-way ANOVA on multiple algorithms to be evaluated; The visualization processing generates box plots, violin plots, multi-objective Pareto front plots, and 3D sensitivity surfaces, among which: Multi-objective Pareto front based on dominance relationship It is generated using the NSGA-II algorithm.

[0018] Furthermore, the structured report generated in step S5 includes: Performance confidence interval table; Robustness curve, the curve obtained through a logistic regression model A fitting is performed, where f is the node failure rate. This corresponds to the task success rate; Multi-objective Pareto front plots are used to illustrate the trade-off between task performance and resource consumption for different algorithms or parameter configurations.

[0019] This application also provides an evaluation system for the collaborative simulation performance of unmanned system swarms, applied to the evaluation method for the collaborative simulation performance of said unmanned system swarms, including a task setting unit, a global configuration unit, a simulation unit, an analysis and calculation unit, and a statistical analysis unit, wherein: The task setting unit is used to receive relevant files and scenario description information of the cluster collaborative control algorithm to be evaluated, and to clarify the evaluation objectives; The global configuration unit is used to set the probability distribution of random parameters and the triggering rules of disturbance events, and to generate corresponding parameter and disturbance event configuration data; The simulation unit is used to launch multiple sets of simulation tasks in parallel in the computing environment based on the configuration data. Each simulation dynamically generates the initial state and environment model according to random parameters, injects disturbance events, and records simulation logs. The parsing and calculation unit is used to parse simulation logs in batches, extract key data, and calculate multi-dimensional performance indicators; The statistical analysis unit is used to perform statistical analysis and visualization on the multi-dimensional performance indicators and automatically generate a structured evaluation report.

[0020] In practical applications, the system and modules disclosed in this application can be deployed on a single target server, or each module can be deployed independently on different target servers. In particular, as needed, to provide more powerful computing capabilities, modules can also be deployed on cluster target servers.

[0021] Therefore, by constructing an automated evaluation process of "task definition → global random configuration → parallel simulation → log parsing → report generation" and integrating Monte Carlo methods, containerized parallel simulation, multi-dimensional indicator systems, and systematic statistical analysis, the technical problems existing in traditional evaluation methods, such as lack of statistical significance, one-sided robustness evaluation, single evaluation dimensions, low degree of automation, and poor reproducibility, are solved. This enables a scientific, comprehensive, efficient, and reproducible performance evaluation of unmanned system cluster collaborative control algorithms.

[0022] To provide a clearer and more comprehensive understanding of this application, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating an embodiment of the method for evaluating the collaborative simulation performance of an unmanned system cluster, as described in this application. Detailed Implementation

[0025] This invention enables a comprehensive and efficient systematic evaluation of collaborative simulation of unmanned system swarms through automation. The technical solution is as follows: a method for evaluating the effectiveness of collaborative simulation of unmanned system swarms, comprising the following steps: S1: Receive relevant files and scenario descriptions of the collaborative control algorithm of the cluster to be evaluated, and clarify the evaluation objectives; S2: Set the probability distribution of random parameters and the triggering rules of disturbance events, and generate the corresponding parameter and disturbance event configuration data; S3: Based on the configuration data, multiple simulation tasks are launched in parallel in the computing environment. Each simulation dynamically generates the initial state and environment model according to random parameters, injects disturbance events, and records simulation logs. S4: Batch parse simulation logs, extract key data and calculate multi-dimensional performance indicators; S5: Perform statistical analysis and visualization processing on the multi-dimensional performance indicators to automatically generate a structured evaluation report.

[0026] Please see Figure 1 The technical solution of this application is described below with reference to various preferred embodiments. In this embodiment, the Monte Carlo performance evaluation of a 1000-drone swarm cooperative encirclement algorithm is used as the test setup condition. The test setup conditions are as follows: The hardware environment is configured as follows: The Slurm-20.11 containerized cluster consists of 8 Dell PowerEdge R750 rack nodes. Each node is configured with 2×Intel Xeon Gold 6330 (28 cores / 56 threads, 2.0 GHz base frequency), 256 GB DDR4-3200, and 2×NVIDIA A40 48 GB GPUs, interconnected via 10 GbE RoCEv2 lossless Ethernet. Software environment: The operating system was Ubuntu 22.04 LTS, the container engine was Docker 24.0, the task scheduler was Slurm-20.11, the simulation environment was ROS2-Humble+Gazebo-11+PX4-SITL v1.14, and the data processing tools were Python 3.10 (Pandas-2.0, NumPy-1.24, Seaborn, Matplotlib). The algorithm under test was the open-source CBF-DP trapping algorithm (based on the control barrier function and dynamic programming, and implemented in C++ with ROS2 nodes). Assessment Objectives: The CBF-DP algorithm is quantified in the scenario of "thousands of drones surrounding a single maneuvering target". The algorithm's overall performance and robustness boundary are compared under different parameter configurations, considering initial dispersion, wind disturbance, sensor noise, communication packet loss and random node failure.

[0027] S1: Receive relevant files and scenario descriptions of the collaborative control algorithm of the cluster to be evaluated, and clarify the evaluation objectives.

[0028] This step involves setting up the task by uploading the ROS2 node package (.deb format, containing node startup files and dependent libraries) for the CBF-DP algorithm and the Gazebo scene description file (.world format, defining the task airspace as 116.3°–116.4°E, 39.9°–40.0°N, altitude 80 m–120 m, target as a single UAV maneuvering at 8 m / s with no fixed flight path) through the system GUI. An evaluation metric set is then selected from the metric template library. Mission Completion Rate: Success Rate of Encirclement and Capture (Key Indicators), Encirclement Time (Auxiliary indicators).

[0029] System stability: Root mean square of radial error in formation (Core Indicators), Convergence Time (Auxiliary indicator)

[0030] Robustness: Failure tolerance curve (core indicator), communication interference index (auxiliary indicator).

[0031] Resource consumption: Average communication traffic per unit (Supporting indicators), Energy consumption per unit (Core metrics), CPU utilization (auxiliary metrics).

[0032] At the same time, the core objectives of the assessment are clearly defined: 1. Quantify the robustness boundary of the algorithm under the combined disturbance of "wind disturbance + communication packet loss + node failure"; 2. Find the optimal parameter configuration for "encirclement success rate - energy consumption" (e.g., prediction time domain).

[0033] This step avoids ambiguity in the evaluation direction by clearly defining the algorithm to be evaluated, the scenario boundaries, and the core metrics. Invalid data collection should be avoided. Ensure that subsequent steps (such as parameter configuration and data collection) revolve around the "robustness of the trapping algorithm and resource trade-offs." For example, the selection of metrics directly determines the scope of ROS information collection (collecting only data from...). , (Instead of collecting all data, information such as odom and power_status is collected to reduce data redundancy).

[0034] S2: Set the probability distribution of random parameters and the triggering rules of disturbance events, and generate the corresponding parameter and disturbance event configuration data.

[0035] This step of the setup and configuration includes: parameter space setting, specific parameter configuration for continuous variables, specific parameter configuration for discrete variables, and disturbance event configuration, as detailed below: S21 parameter space setting: 12 random parameters (8 continuous variables + 4 discrete variables) are set, generated by Latin cube sampling. Group parameters, the sampling process minimizes "centralization" " Deviation" ensures space filling; the deviation formula and the meaning of the symbols are as follows:

[0036] The meanings of each symbol are as follows: Centralization Bias measures the uniformity of sampling points in the parameter space. The smaller the value (0.03 in this case), the more uniform the sampling and the stronger the representativeness of the sample. : Sample point set (2000 sets in this case) ); Total number of samples (this time) ); Parameter dimensions (this time) (8 continuous + 4 discrete) : The integration variable, representing any point in the parameter space; : The coordinate values ​​in each dimension are used to calculate the sampling density under an ideal uniform distribution.

[0037] By setting the parameter space and quantifying the sampling uniformity, we can ensure that 2000 sets of samples cover the entire range of parameters from "initial position to wind disturbance to noise to failure rate", which avoids the defects of "local density and global sparsity" in traditional random sampling.

[0038] S22: Specific parameter configuration for continuous variables (truncating the normal distribution to avoid extreme values ​​from deviating from the actual scenario).

[0039] Initial horizontal position: m (airspace center) m (dispersion range), cutoff ±180 m (airspace boundary); high: m (task height) m, cut off 35–65 m; Initial velocity: m / s (cruising speed) m / s, cutoff 10–20 m / s; Wind speed in the wind field: m / s (normal wind speed) m / s (gust fluctuation), cutoff 0–8 m / s; GPS white noise: m (actual GPS accuracy range); IMU gyroscope zero bias: rad / s (typical IMU error); Communication delay: exponential distribution Exp( ms (5G-SA link latency characteristics); Packet loss rate: Beta(2,98) (mean 2%, maximum 8%, consistent with actual communication link characteristics); The specific parameter configuration for the discrete variables in S23.

[0040] Number of obstacles: Poisson distribution (An average of 8 obstacles to simulate a complex environment); Failure Node Ratio: Empirical Distribution (Covering low / medium / high failure scenarios); Disturbance type: Empirical distribution Sensor noise, communication packet loss, node failure, and combined disturbances (Each accounts for 2596, providing balanced coverage of disturbed scenarios).

[0041] S24: Disturbance event configuration.

[0042] The disturbance event is transmitted through the event space. Configure each disturbance event ,in For the trigger time, For duration, The trigger probability is satisfied. , The event attribute tuples are perturbation events that generate independent and identically distributed event sequences through a non-homogeneous Poisson process. And serialize it into an injection file.

[0043] Generate independent and identically distributed event sequences based on non-homogeneous Poisson processes. Define three types of disturbance events (covering common faults in unmanned systems): Event 1 (Instantaneous node drop): ); Event 2 (Communication Silence): ); Event 3 (Power Downgrade): ).

[0044] Event parameter meanings: (Triggering time) (Duration) (Trigger probability) (Event attributes, including the target object and fault type).

[0045] By using a non-homogeneous Poisson process, the triggering time and the target of the disturbance event are ensured to be random and independent, avoiding the limitations of traditional manual scripts with "fixed time and fixed nodes", and more closely reflecting the randomness of actual faults.

[0046] Finally, configure version management by managing each group. Generate SHA-256 digest (e.g.) Stored in Git-like content-addressable store (CAS), returning experimental batch IDB=2024052001.

[0047] S3: Based on the configuration data, multiple simulation tasks are launched in parallel in the computing environment. Each simulation dynamically generates the initial state and environment model according to random parameters, injects disturbance events, and records simulation logs.

[0048] The specific simulation configuration and deployment are as follows: Task scheduling: Parameter batch files are read using, for example, a Slurm container or a Kubernetes orchestration framework, based on resource utilization metrics. Dynamically scalable computing resources, including: The number of CPU cores requested for the current task. This represents the number of idle CPU cores in the cluster.

[0049] This initial (With a large number of idle cores), 448 parallel containers (8 nodes × 56 threads) are launched.

[0050] Scheduling algorithm: Employs a "minimum heap priority queue" to prioritize resource allocation to high-priority tasks with "complex disturbances" (ensuring that complex scenarios are completed first).

[0051] Simulation environment deployment: Each node (Pod) runs a ROS2-Humble container, pre-installed with Gazebo-11 and PX4-SITL, and injects configuration data via environment variables (e.g., export PARAMS=). This avoids inconsistencies in the environment caused by modifications to files within the container. Generation of identifiers and log storage: Generate a unique run ID (e.g., $r_1 = 550e8400-e29b-41d4-a716-446655440000)), write the log metadata.

[0052] The log storage path is, for example, / data / $r_m / rosbag.db3, mounted to MinIO distributed object storage (supporting 72 hours of hot storage + long-term cold storage). A single log is about 2.2 GB (180 s simulation, recording 27 ROS topics, bandwidth ≈12 MB / s), and the total data volume is 4.4 TB. Simulation parameters: The duration of a single simulation is 180 seconds (covering the entire process of "takeoff-encirclement-hold"), the step size is t = 0.01 seconds (meeting the PX4 flight controller control frequency requirements), and the random seed uses SHA-256 (run sequence number + timestamp) modulo 1. This ensures that the random flows in 2000 simulations are completely independent.

[0053] This embodiment uses 448 parallel containers to compress the simulation time of 2000 times from "single-thread 2000 × 3 min = 100 h" to 3.2 h, improving efficiency by more than 30 times and solving the problem of long simulation cycles in traditional single-machine simulation.

[0054] In addition, containerized deployment ensures that all simulation environments are consistent (ROS version and dependency libraries are exactly the same), avoiding "result deviations caused by environment differences".

[0055] This embodiment achieves fault isolation through distributed log storage. In this experiment, three containers crashed, but the log integrity of the other 445 tasks was not affected, thus solving the defect of traditional local storage where a single failure results in total loss.

[0056] S4: Batch parse simulation logs, extract key data and calculate multi-dimensional performance indicators.

[0057] Log analysis: Based on the API, 2000 rosbag files were parsed in batches, with multi-threaded parallel parsing enabled (e.g., 64 threads), taking 42 minutes (RAID-5 NVMe storage, sequential read speed 3.8 GB / s), extracting key data (such as drone position xi(t), speed vi(t), and target position xtarget(t) every 0.01 s). Multi-dimensional indicator calculation (automatically executed according to the template library): Task completion metrics: Encirclement success rate: Psuccess = number of successful runs / M, where "success" is defined as "the target is continuously contained within a circle with a radius of 50m for ≥ 10 s". In this instance, Psuccess = 87.4%. Encirclement time Tc: The time from the start of the simulation to the first satisfaction of the "encirclement condition". If unsuccessful, it is recorded as 180 s. The average value of this simulation is Tc = 92.3 s. System stability metrics (root mean square error):

[0058] in: The root mean square error of the formation measures the stability of the cluster formation; the smaller the value, the more stable the formation. N: Number of drones (N = 1000 in this case); T: Simulation duration (T = 180 s in this case); : t represents the formation error of the i-th UAV at time t, which in this case is "the horizontal distance from the UAV to the projection point of the ideal encirclement circle"; By eliminating instantaneous error fluctuations through integration, the long-term shape stability is quantified, and RMSEr = 3.7 m in this case.

[0059] Convergence time T95: The first time when εr,i(t) enters the "±5% steady state band" (i.e., |εr,i(t) - εr| ≤ 0.05εr) and lasts for ≥5 s. In this case, T95 = 28.5 s.

[0060] Robustness index (logistic regression fitted curve):

[0061] Where: S(f): the success rate of the task when the failure rate is f, f: the node failure rate (f ∈ [0,0.15]), β0, β1: logistic regression model parameters (β0=2.8, β1=-18.5 in this case), β0 is the intercept (S(0)≈98.1% when f=0), and β1 is the slope (the negative sign indicates that S(f) decreases as f increases).

[0062] The robustness index formula fits the nonlinear relationship between "failure rate - success rate" and quantifies the robustness boundary. In this embodiment, the measured value is f90=0.08 (S(f)=90% when 8% of nodes fail) and the critical failure rate (S(f)=50%) is 17.3%.

[0063] Communication interference index: That is, for every 1% increase in packet loss rate, the corresponding The percentage decrease was 1.2% this time.

[0064] Resource consumption indicators: Average communication traffic per unit: Bavg = (Converted to Mbps), this time the Bavg = 4.2 Mbps; Energy consumption per unit:

[0065] in: Total energy consumption of a single drone (unit: kWh).

[0066] Hovering power (this time) W, DJI Matrice 300 parameters).

[0067] : Drag power, ( kg / m For air density, m For windward area, The drag coefficient, for Time of the first (The speed of the drone).

[0068] Sensitivity analysis (Sobol index):

[0069] in: : No. The first-order Sobol exponent of the parameter ranges from [0, 1]. A larger value indicates that the parameter has a greater impact on the index. The greater the impact; : No. A random parameter (such as wind speed, packet loss rate); :remove Other parameter sets besides; Evaluation indicators (for this case) ); :variance, Given hour The expected condition.

[0070] By quantifying the weight of each parameter's impact on the success rate, this embodiment measures the "wind speed". Packet loss rate "These are key influencing factors, providing direction for subsequent algorithm optimization (such as prioritizing the optimization of wind disturbance resistance strategies).

[0071] The actual test results show that this step, through multi-threaded parsing, reduced the processing time of 4.4TB logs from "5 hours with a single thread" to 42 minutes, improving efficiency by 7 times.

[0072] This application's embodiments utilize multi-dimensional indicators covering "task-stability-robustness-resources," overcoming the shortcomings of traditional methods that focus only on a single indicator. Furthermore, by using the Sobol index to clearly define key parameters, it avoids blind parameter tuning, reducing the subsequent "prediction time domain steps from 10 steps to 6 steps," achieving precise optimization.

[0073] S5: Perform statistical analysis and visualization processing on the multi-dimensional performance indicators to automatically generate a structured evaluation report.

[0074] Basic statistics include: calculating the mean of indicators. ,variance 95% confidence interval (1.96 is the Z-value corresponding to the 95% confidence level), this time The 95% confidence interval is [86.0%, 88.8%], with a relative error of < 1%.

[0075] This embodiment is based on two-way ANOVA: it analyzes "prediction time domain (A: 5 / 10 / 15 steps)" and "failure rate (B: 5% / 10% / 15%)", and the model is as follows:

[0076] in: : The index value of the i-th level A, the j-th level B, and the k-th repetition; Overall mean; : The main effect of factor A at level i; The main effect of factor B at level j; The interaction effect between A and B; Random error (following the rules of random error) ).

[0077] The experimental results show that the main effect in the prediction time domain is significant ( The interaction effect was not significant. This indicates that the impact of the prediction time domain on performance is independent of the failure rate.

[0078] Finally, visualization processing is performed, including: Generate a box plot of RMSE_r (showing the quartiles of the error distribution) and a violin plot of T_c (showing the error probability density). Generate a Pareto front plot of "energy consumption - capture time" (based on NSGA-II algorithm, 200 generations, population 100), with the following dominance relationships: The optimal trade-off point is clearly defined as "10 steps for predicting the time domain".

[0079] Generate a 3D sensitive surface (horizontal axis: wind speed, vertical axis: packet loss rate, color: This visually demonstrates the combined effect of the two key parameters.

[0080] Finally, a report is generated: for example, an HTML report can be generated using the Jinja2 template engine, or it can be in other formats such as PDF, including: Experiment metadata (container image SHA-256:4f3a…e7 / c1, Git commit:5a3b…d2e8, list of run IDs); Statistical results table (mean, variance, 95% confidence interval); Visualization charts (box plots, Pareto fronts, robust curves); Optimization suggestions (such as "reducing the prediction time domain to 6 steps reduces energy consumption by 19% while only reducing the success rate by 2.1%"). Digital signature (to ensure the report has not been tampered with).

[0081] Therefore, the technical effects achieved by this application are as follows: 1. Significantly improved statistical confidence: Through 2000 low-biased samples, the core indicators... The relative error of the 95% confidence interval is <196, which is 15 times narrower than the traditional "10 samples" (relative error >15%), and the evaluation conclusion can support scientific decision-making.

[0082] 2. Precise robustness quantification: The robustness boundary of the algorithm is clearly defined by fitting the curve through logistic regression, which solves the problem that traditional MPS can only qualitatively describe robustness and provides a safety threshold for actual deployment (such as recommending that the failure rate should not exceed 8%).

[0083] 3. Significantly improved evaluation efficiency: The entire process takes 4.6 hours (including 30 minutes for configuration, 3.2 hours for simulation, 42 minutes for analysis, and 35 minutes for statistics), which is 90% shorter than the traditional "manual configuration + single-machine simulation" solution (approximately 2 days = 48 hours). The manpower input is reduced from 8 person-hours to 0.5 person-hours, reducing labor costs by 70%.

[0084] 4. Comprehensive multi-dimensional evaluation: The four-dimensional indicators cover "task - stability - robustness - resources", avoiding the overgeneralization of traditional methods. For example, if it is found that "predicting the time domain in 15 steps improves the success rate by 2%, but increases energy consumption by 25%", then this configuration will be excluded.

[0085] 5. High reproducibility: By configuring the digest and unique run ID using SHA-256, a third party can reproduce the experiment 2000 times on the same hardware. The difference is less than 0.3%, which solves the problem of "reproduction error > 10%" in traditional experiments and supports horizontal comparison of algorithms.

[0086] Based on the above embodiments, this application also provides an evaluation system for the collaborative simulation performance of unmanned system swarms, applied to the evaluation method for the collaborative simulation performance of the unmanned system swarms, including a task setting unit, a global configuration unit, a simulation unit, an analysis and calculation unit, and a statistical analysis unit, wherein: The task setting unit is used to receive relevant files and scenario description information of the cluster collaborative control algorithm to be evaluated, and to clarify the evaluation objectives; The global configuration unit is used to set the probability distribution of random parameters and the triggering rules of disturbance events, and to generate corresponding parameter and disturbance event configuration data; The simulation unit is used to launch multiple sets of simulation tasks in parallel in the computing environment based on the configuration data. Each simulation dynamically generates the initial state and environment model according to random parameters, injects disturbance events, and records simulation logs. The parsing and calculation unit is used to parse simulation logs in batches, extract key data, and calculate multi-dimensional performance indicators; The statistical analysis unit is used to perform statistical analysis and visualization on the multi-dimensional performance indicators and automatically generate a structured evaluation report.

[0087] The application embodiment also provides a storage medium storing a computer program, which, when executed by a processor, performs the evaluation method for the collaborative simulation performance of unmanned system clusters as described above.

[0088] This application also provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the evaluation method for the collaborative simulation performance of unmanned system clusters as described above.

[0089] It should be noted that those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, which may include, but is not limited to, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0090] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating the collaborative simulation performance of unmanned system clusters, characterized by comprising the following steps: S1: Receive relevant files and scenario descriptions of the collaborative control algorithm of the cluster to be evaluated, and clarify the evaluation objectives; S2: Set the probability distribution of random parameters and the triggering rules of disturbance events, and generate the corresponding parameter and disturbance event configuration data; S3: Based on the configuration data, multiple simulation tasks are launched in parallel in the computing environment. Each simulation dynamically generates the initial state and environment model according to random parameters, injects disturbance events, and records simulation logs. S4: Batch parse simulation logs, extract key data and calculate multi-dimensional performance indicators; S5: Perform statistical analysis and visualization processing on the multi-dimensional performance indicators to automatically generate a structured evaluation report.

2. The method for evaluating the collaborative simulation performance of unmanned system clusters as described in claim 1, characterized in that, The global random configuration in step S2 includes: M sets of independent parameters were generated using Latin hypercube sampling, where parameter vector ,in For continuous variable dimensions, For discrete variable dimensions; Latin hypercube sampling is used to generate samples that satisfy space-filling properties, and the deviation formula is: ； Where X is the set of sampling points, N is the number of sampling points, and d is the parameter dimension; Generate independent and identically distributed event sequences using a perturbation event injector. Among them, the events The data is then serialized into an injection file; a summary is generated from the parameter and event sequences.

3. The method for evaluating the collaborative simulation performance of unmanned system clusters as described in claim 1, characterized in that, In step S2, the random parameters include continuous variables and discrete variables, wherein: Continuous variables Supports Gaussian distribution Uniform distribution truncated Gaussian distribution ; Discrete variables It supports Poisson (λ) and Emp (pk) distributions and outputs M sets of independent parameters that satisfy Latin hypercube sampling. .

4. The method for evaluating the collaborative simulation performance of unmanned system clusters as described in claim 1, characterized in that, Step S3 also includes: before each simulation run, the task scheduler generates a unique hash identifier for this run and writes the identifier into the metadata of the log file.

5. The method for evaluating the collaborative simulation performance of unmanned system clusters as described in claim 1, characterized in that, The computing environment described in step 3 is: a container-based system that mounts the Gazebo simulation environment and the algorithm node to be evaluated, and injects the configuration data through environment variables; The simulation's task scheduling uses a container orchestration framework based on resource utilization metrics. The system automatically and elastically scales the number of computing resources, and the scheduling algorithm uses a min-heap priority queue to allocate computing resources.

6. The method for evaluating the collaborative simulation performance of unmanned system clusters as described in claim 1, characterized in that: The statistical calculation in step S4 further includes: Perform a two-way ANOVA to test for significant differences in algorithm performance; Sensitivity analysis was performed using the Sobol exponent method to quantify various random parameters. The weighting of the impact on the performance index y, where the Sobol exponent method is used to calculate the formula: ； in: : No. The first-order Sobol exponent with one parameter : No. A random parameter (such as wind speed, packet loss rate). : Other parameter sets, Evaluation indicators :variance, Given hour The expected condition.

7. The method for evaluating the collaborative simulation performance of unmanned system clusters as described in claim 1, characterized in that, The multi-dimensional performance indicators mentioned in step 4 include system stability indicators, task completion indicators, robustness indicators, and resource consumption indicators, among which: The system stability index is calculated using the root mean square error: ； Task completion metrics: ; Robustness metrics: including the proportion of node failures For the horizontal axis, Fitted curve with vertical axis and communication interference index; Resource consumption indicators: Average communication traffic per unit Energy consumption per unit ,in: This refers to hovering power.

8. The method for evaluating the collaborative simulation performance of unmanned system clusters as described in claim 1, characterized in that, The statistical analysis described in step 5 includes calculating the sample mean, variance, and 95% confidence interval, and performing a two-way ANOVA on multiple algorithms to be evaluated. The visualization processing generates box plots, violin plots, multi-objective Pareto front plots, and 3D sensitivity surfaces, among which: Multi-objective Pareto front based on dominance relationship It is generated using the NSGA-II algorithm.

9. The method for evaluating the collaborative simulation performance of unmanned system clusters as described in claim 1, characterized in that, The structured report generated in step S5 includes: Performance confidence interval table; Robustness curve, the curve obtained through a logistic regression model A fitting is performed, where f is the node failure rate. This corresponds to the task success rate; Multi-objective Pareto front plots are used to illustrate the trade-off between task performance and resource consumption for different algorithms or parameter configurations.

10. An evaluation system for the collaborative simulation performance of unmanned system swarms, applied to the evaluation method for the collaborative simulation performance of unmanned system swarms as described in any one of claims 1-9, characterized in that, It includes a task setting unit, a global configuration unit, a simulation unit, a parsing and calculation unit, and a statistical analysis unit, among which: The task setting unit is used to receive relevant files and scenario description information of the cluster collaborative control algorithm to be evaluated, and to clarify the evaluation objectives; The global configuration unit is used to set the probability distribution of random parameters and the triggering rules of disturbance events, and to generate corresponding parameter and disturbance event configuration data; The simulation unit is used to launch multiple sets of simulation tasks in parallel in the computing environment based on the configuration data. Each simulation dynamically generates the initial state and environment model according to random parameters, injects disturbance events, and records simulation logs. The parsing and calculation unit is used to parse simulation logs in batches, extract key data, and calculate multi-dimensional performance indicators; The statistical analysis unit is used to perform statistical analysis and visualization on the multi-dimensional performance indicators and automatically generate a structured evaluation report.

Images