Multi-agent collaborative industrial artificial intelligence algorithm evaluation system and method

Abstract

The invention discloses a multi-agent collaborative industrial artificial intelligence algorithm evaluation system and method, and aims to solve the problems of disordered evaluation process, insufficient compliance guarantee and poor system expansibility in the prior art. The system adopts a six-level layered architecture, the core is a navigation agent, and the navigation agent monitors the task state in a global data layer, performs dual verification in combination with national standard rules, and schedules data division, script writing, algorithm parameter adjustment, report generation and other special agents for sequential execution only when the state is compliant. And each agent interacts through the standardized JSON and calls a tool based on MCP packaging. According to the invention, automation, ordering and national standard compliance of the evaluation process are realized, flexible expansion is supported through loose coupling design, and comparability and traceability of the evaluation result are guaranteed.

Description

Technical Field

[0001] This invention relates to the field of industrial artificial intelligence technology, and in particular to a multi-agent collaborative industrial artificial intelligence algorithm evaluation system and method, applicable to standardized evaluation scenarios of industrial artificial intelligence algorithms such as condition monitoring, fault diagnosis, and life prediction. Background Technology

[0002] As Industry 4.0 evolves into Smart Manufacturing 2.0, Predictive Maintenance (PdM) has become a core support for manufacturing enterprises to reduce costs and increase efficiency, and algorithm evaluation is a key link in the large-scale implementation of PdM technology. The release of my country's GB / T 43555-2023 and the international IEC 63270-1:2025 standards clearly requires that algorithm evaluation cover the entire process of "data acquisition - feature extraction - model building - decision optimization," and must meet quantitative indicators such as fault prediction confidence and maintenance response time. However, in industry practice, traditional evaluation models still suffer from pain points such as cumbersome processes, inconsistent environments, and lack of standardization, urgently requiring automated, standardized, and low-barrier evaluation solutions.

[0003] Industrial enterprises face common pain points in the process of introducing, iteratively optimizing, and scaling up predictive maintenance algorithms, such as fragmented evaluation processes, high technical barriers, incomparable results, and large cost investments. Traditional "black box" blind testing and "tool patchwork" evaluation methods are difficult to meet the needs of engineering implementation.

[0004] There are currently four core technical problems in the field of industrial predictive maintenance algorithm evaluation:

[0005] 1) Traditional evaluation processes have a low degree of automation and rely heavily on manual connection of multiple independent tools (such as data preprocessing tools, scripting tools, parameter tuning tools, etc.). Data partitioning, script writing, and other steps need to be completed manually. The evaluation cycle for a single algorithm can be as long as 4-8 days, which is extremely inefficient.

[0006] 2) Poor consistency of the testing environment, tool version conflicts lead to a deviation of 18%-23% in results, and there is a lack of unified standards, making it impossible to compare results from different platforms. During the testing process, intermediate status, decision-making basis and generated data are scattered in different tools or files, lacking a unified global data management and traceability mechanism, which is not conducive to problem investigation and test reproduction.

[0007] 3) The technology threshold is high, with an average annual investment of over 800,000 yuan for enterprises, which is difficult for small and medium-sized enterprises to afford. Moreover, there is no quantitative evaluation mechanism for the results generated by intelligent agents. The traditional evaluation system architecture is tightly coupled. When adding new evaluation scenarios, indicators or tools, it is often necessary to modify the core code, resulting in high development and maintenance costs and difficulty in quickly responding to the diverse evaluation needs of industrial sites.

[0008] 4) Existing systems are not compatible with national standards, resulting in insufficient report compliance and an inability to meet the requirements of specific national standards (such as GB / T43555-2023) or international standards (such as IEC 63270-1:2025). Existing general-purpose testing systems or independent tools lack built-in national standard rule bases and compliance verification mechanisms. The compliance of the testing process and result reports heavily relies on manual review by experts, which is inefficient and prone to errors.

[0009] Algorithm testing metrics are crucial for evaluating the performance of various algorithms. By determining whether an algorithm passes these metrics, companies can obtain immediate information on its performance, stability, and generalization ability, enabling them to continuously optimize the algorithm model and adjust its internal parameters to improve reliability and stability. Predictive maintenance algorithm testing primarily targets state monitoring algorithms, fault diagnosis algorithms, and prediction algorithms. Different algorithms have various testing metrics defined to enrich the testing dimensions.

[0010] (1) Test indicators of state monitoring algorithm

[0011] State monitoring algorithms can be divided into discrimination methods based on fixed thresholds and discrimination methods based on relative thresholds.

[0012] The difference between the two discrimination methods lies in whether standard data is learned and whether the alarm threshold is customized. Compared with discrimination methods based on fixed thresholds, discrimination methods based on relative thresholds can handle scenarios that cannot be handled by current standards, and have higher flexibility and customization.

[0013] The test metrics for status monitoring algorithms include status monitoring accuracy and the rate of missed abnormal status reports.

[0014] Status identification accuracy refers to the accuracy of classifying equipment status monitoring results. By determining whether equipment is in a normal state after long-term operation, companies can shut down equipment that is malfunctioning and perform relevant maintenance in the near future.

[0015] State discrimination accuracy is calculated as the percentage of samples with correctly classified states to the total number of samples, as shown in equation (1):

[0016] Equation (1)

[0017] in, This represents the number of samples that correctly identified the state. This represents the total number of samples. The state discrimination accuracy ranges from 0 to 1, with a higher value indicating higher accuracy in state discrimination.

[0018] The false negative rate for abnormal states refers to the percentage of samples that are not correctly identified as abnormal states when monitoring equipment status and identifying abnormalities. It reflects the accuracy of the algorithm in identifying abnormal states. It is judged based on the percentage of the number of unidentified abnormalities in the sample relative to the total number of abnormalities, as shown in equation (2):

[0019] Equation (2)

[0020] in, This represents the number of abnormal samples identified. This represents the total number of abnormal samples. The false negative rate for abnormal states also ranges from 0 to 1; the smaller the value, the better the algorithm's performance in accurately identifying abnormalities.

[0021] (2) Test indicators for fault diagnosis algorithm

[0022] Fault diagnosis algorithms mainly rely on classification algorithms in machine learning as the key to identify fault types. Their test metrics mainly target indicators such as accuracy, precision, and recall. By verifying whether the algorithm meets the test metrics, testers can obtain information such as the algorithm's robustness, interpretability, and generalization ability, so as to further optimize the algorithm.

[0023] Accuracy reflects the overall performance of the algorithm and is calculated based on the percentage of correctly diagnosed samples out of the total number of diagnosed samples. The accuracy formulas for different classification algorithms are shown in equation (3):

[0024] Equation (3)

[0025] In the formula, This represents the number of samples that accurately identified category i, where n is the total number of categories in the samples, and N is the total number of samples. By examining the accuracy of the classification algorithm, we can determine whether the consistency between the algorithm's diagnostic results and the actual situation meets the prescribed standards, reflecting the algorithm's ability to closely approximate reality.

[0026] Precision rate represents the ratio of the number of positive samples correctly diagnosed by the algorithm to the total number of positive samples. The focus is on the accuracy of the samples identified as faulty in the algorithm's diagnostic results. The macro-average formula for the precision rate of different classification algorithms is shown in Equation (4).

[0027] Equation (4)

[0028] In the formula, This represents the number of samples correctly identified as category i by the algorithm. This represents the number of samples that the algorithm incorrectly identified as category i, and n represents the total number of categories of the samples.

[0029] The formula for calculating the precision rate micro-average is shown in equation (5).

[0030] Equation (5)

[0031] The accuracy ranges from 0 to 1. The higher the accuracy, the more reliable the algorithm is in fault identification.

[0032] Recall rate is also known as true positive rate, reflecting the completeness of fault samples that the algorithm can identify from all diagnostic samples. The macro-average formula for the recall rate of the n-class algorithm is shown in equation (6).

[0033] Equation (6)

[0034] in, This represents the number of samples correctly identified as category i by the algorithm. This represents the number of samples where the algorithm incorrectly identified category i as other categories, and n represents the total number of categories of the samples.

[0035] The formula for calculating the micro-average recall rate is shown in equation (7).

[0036] Equation (7)

[0037] The recall rate ranges from 0 to 1. The higher the recall rate, the more completely the diagnostic algorithm can analyze and diagnose the real fault samples in the overall sample. It is worth noting that in actual evaluation, it is often necessary to comprehensively consider accuracy, precision, and recall to test and optimize the algorithm's capabilities.

[0038] (3) Prediction algorithm test indicators

[0039] Similar to the evaluation methods for other algorithms, prediction algorithms are primarily designed for equipment lifespan prediction. Evaluation metrics are defined based on four aspects: prediction accuracy, prediction error, algorithm fit, and prediction error score. By testing the performance of prediction algorithms, testers can assess their overall ability to predict lifespan, helping companies estimate equipment operating status and make timely adjustments and maintenance to production plans and equipment.

[0040] Prediction accuracy reflects the degree to which the algorithm correctly predicts the remaining lifespan of the sample. The specific calculation formula is shown in equation (8).

[0041]

[0042] Equation (8)

[0043] In the formula, i represents the sample number, and N represents the total number of samples. express The difference between the actual remaining useful life and the predicted remaining useful life at any given time. express The actual remaining service life at any given moment. express Predicted remaining useful life at any given time.

[0044] Mean absolute error measures the degree of closeness between the predicted remaining useful life and the actual useful life. The calculation formula is shown in equation (9):

[0045] Equation (9)

[0046] The range of values ​​is Because the calculation method is relatively straightforward, this indicator has low sensitivity to outliers and can only reflect absolute deviation. It needs to be evaluated in conjunction with other indicators to summarize the overall performance of the algorithm.

[0047] The root mean square error (RMSE) places greater emphasis on larger error values ​​than the mean absolute error (MAE), meaning it is given a larger weight. If the prediction result deviates significantly from the actual situation, the RMSE value will worsen, as shown in formula (10).

[0048] Equation (10)

[0049] In the algorithm evaluation process, it is generally necessary to At a relatively low level, based on factors such as industrial scenario requirements and lifespan data range, The evaluation scope is also quite diverse.

[0050] The coefficient of determination is an important indicator used to detect the goodness of fit between the predicted service life and the actual remaining service life. It reflects the degree of data compilation that the algorithm model can interpret. The specific calculation formula is shown in (11).

[0051] Equation (11)

[0052] in, The coefficient of determination is the average of the actual remaining useful life, and its range is as follows: If the coefficient of determination is close to 1, it indicates that the algorithm has a higher degree of fit and a better predictive ability for real-world scenarios.

[0053] The prediction error score is an indicator that can determine the degree of error between the algorithm's predicted remaining service life and the actual service life. The prediction error score test uses an asymmetric function as a method to judge premature prediction (predicted service life is longer than the actual service life) and premature prediction (predicted service life is shorter than the actual service life). In industrial scenarios, in order to avoid unnecessary downtime and losses, premature prediction is generally used instead of premature prediction. Therefore, different levels of penalty factors are used in the calculation to constrain the test performance index. The specific calculation formula is shown in Equation (12).

[0054] Equation (12)

[0055] i is the sample number. The penalty factor for premature prediction is generally set to [10, 15], with a recommended value of 13. β is the penalty factor for late prediction, and its value is specified to be less than that for premature prediction. It is usually set to [7, 12], with a recommended value of 10.

[0056] The range of values ​​for SPE is: If the SPE value is close to 0, it indicates that the algorithm has low prediction error and high prediction accuracy. Furthermore, SPE detection has a certain degree of customization capability to meet different needs in industrial scenarios. For example, in the RUL prediction of aero engines, α=15 and β=7, reflecting zero tolerance for missed detections.

[0057] Therefore, there is an urgent need for an industrial algorithm evaluation solution that can automate processes, embed national standard compliance verification, and has good scalability. Summary of the Invention

[0058] This invention addresses the shortcomings of existing technologies, such as low automation, poor environmental consistency, insufficient standardization, and high operational barriers. It proposes an efficient, accurate, and compliant multi-agent collaborative industrial artificial intelligence algorithm evaluation system and method. Through a multi-agent collaborative architecture and a fully automated process, the evaluation cycle is shortened, the operational barriers are lowered, and the credibility and compliance of the evaluation results are ensured.

[0059] To achieve the above objectives, the present invention provides the following technical solution:

[0060] In a first aspect, the present invention provides a multi-agent collaborative industrial artificial intelligence algorithm evaluation system, which adopts a six-level hierarchical architecture, including:

[0061] The user layer provides multiple operation interfaces and multi-role login functions;

[0062] The interaction layer is used to parse user commands into system executable commands;

[0063] The multi-agent core layer includes a scheduling center and at least four functional agents;

[0064] Toolset layer: Encapsulates industrial testing tools based on the MCP (Model Context Protocol) framework;

[0065] The data layer uses an asynchronous database to build a global data sharing layer;

[0066] The infrastructure layer is used to enable containerized deployment and resource scheduling.

[0067] The scheduling center of the multi-agent core layer is the navigation agent, and the at least four functional agents include the data partitioning agent, the evaluation script writing agent, the algorithm parameter tuning agent, and the report generation agent.

[0068] The data layer is used to store task status and execution result data written by each intelligent agent;

[0069] The navigation agent is configured to: monitor the task status in the data layer in real time, and verify the current task status according to the preset industrial algorithm evaluation process rules and national standard compliance rules. When the status meets the standards and the national standard is compliant, the agent recommends the next matching functional agent to the user. After the functional agent executes the task and updates the data layer, the monitoring, verification and recommendation process is repeated until the evaluation process ends.

[0070] Furthermore, the preset industrial algorithm evaluation process rules include sequential execution of data partitioning, evaluation script writing, algorithm parameter tuning, and report generation steps; the national standard compliance rules include at least one of the following: data partitioning without leakage rules, evaluation indicator national standard compliance rules, and report format national standard compliance rules.

[0071] Furthermore, the data partitioning agent is configured to partition the industrial dataset without leakage based on a dual strategy of "physical components" and "time series", and to preprocess the data by filling in missing values ​​and removing outliers.

[0072] Furthermore, the evaluation script writing agent is configured to generate an executable evaluation script containing preset national standard indicator calculation logic based on the national standard indicator library; the algorithm parameter tuning agent is configured to call the hyperparameter optimization tool in the toolset layer to iteratively optimize the hyperparameters of the target industrial algorithm; and the report generation agent is configured to integrate the full-process data and generate an evaluation report containing a national standard compliance statement.

[0073] Furthermore, the industrial testing tools encapsulated in the toolset layer include at least: a hyperparameter optimization tool based on Optuna, a national standard index calculation tool, and a leak-free data partitioning tool.

[0074] Furthermore, the data stored in the data layer is in JSON format, and the data structure defines special fields required for industrial algorithm evaluation, including at least one of the following: dataset type field, national standard indicator identifier field, no-leakage division identifier field, and parameter range field.

[0075] Secondly, the present invention provides a multi-agent collaborative industrial artificial intelligence algorithm evaluation method, applied to any of the systems described above, executed by a navigation agent, and comprising the following steps:

[0076] S1: Monitor the current evaluation task status in the global data layer;

[0077] S2: Verify the current task status according to the preset industrial algorithm evaluation process rules and national standard compliance rules;

[0078] S3: If the verification passes, match and recommend the next functional agent to be executed based on the current state;

[0079] S4: After detecting that the task status in the data layer has been updated, return to step S1.

[0080] According to the multi-agent collaborative industrial artificial intelligence algorithm evaluation method of claim 7, the step S3, which involves matching the next functional agent to be executed based on the current state, specifically includes:

[0081] If the current status is that the data has been uploaded and the format complies with national standards, then match the data to divide the intelligent agent;

[0082] If the current state is that the data partitioning is complete and meets the national standard requirements for no data leakage, then the evaluation script is matched to write the intelligent agent.

[0083] If the current state is that the evaluation script has been generated and the indicator logic complies with national standards, then the matching algorithm parameter tuning agent will be used.

[0084] If the current state is that the algorithm parameter tuning is complete and the performance meets the requirements, then the matching report generates an agent.

[0085] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-mentioned multi-agent collaborative industrial artificial intelligence algorithm evaluation method.

[0086] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for evaluating industrial artificial intelligence algorithms involving multi-agent collaboration.

[0087] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0088] 1. The evaluation process has been automated and ordered: By using a guide agent to schedule the process based on the dual constraints of "state matching" and "national standard anchoring", the standard sequence of industrial algorithm evaluation steps is enforced, avoiding process errors and omissions caused by manual operation, and ensuring the rigor of the evaluation process and the reliability of the results.

[0089] 2. Built-in national standard compliance guarantee mechanism: The national standard rules (such as GB / T 43555-2023) are deeply embedded into the verification logic of the guide intelligent agent and the core tasks of each functional intelligent agent, realizing automatic compliance verification of the entire process from data division, script generation to report output, which significantly improves the industry recognition and comparability of the evaluation results.

[0090] 3. High scalability and flexibility: Utilizing a six-level loosely coupled layered architecture and standardized MCP tool encapsulation, new assessment agents only need to follow a unified interface; the guiding agent can automatically incorporate them into the process through state recognition. New assessment tools only need to be registered at the toolset layer to be invoked by all agents. This design enables the system to quickly adapt to new industrial assessment scenarios (such as adding fault location assessment) and tool requirements, with low expansion costs.

[0091] 4. Ensures traceability of data throughout the entire process: By uniformly recording the execution status, intermediate results, and national standard verification conclusions of each intelligent agent through a global data layer, a complete evaluation data chain is formed. Users can trace the input, output, and decision-making basis of any step at any time, greatly facilitating problem investigation, auditing, and evaluation reproduction. Attached Figure Description

[0092] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0093] Figure 1 The system architecture diagram of the multi-agent collaborative industrial artificial intelligence algorithm evaluation system provided in the embodiments of the present invention is shown.

[0094] Figure 2 The flowchart shows the "national standard anchoring + state matching" dual-constraint collaborative mechanism provided in the embodiments of the present invention.

[0095] Figure 3 A flowchart illustrating the specific implementation of an embodiment of the present invention. Detailed Implementation

[0096] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0097] The multi-agent collaborative industrial artificial intelligence algorithm evaluation system proposed in this invention adopts a six-level hierarchical architecture, such as... Figure 1 As shown, it includes:

[0098] The user layer provides dual entry points: a web interface and a JupyterLab programming interface, supporting login and operation for multiple roles such as administrators, algorithm engineers, and ordinary users.

[0099] The interaction layer uses Ajax asynchronous communication and natural language parsing to connect user commands with system execution, supporting both visual configuration and command-driven operation modes.

[0100] The multi-agent core layer includes a navigation agent that serves as the scheduling center, as well as at least four types of industrial evaluation-specific functional agents: data partitioning agent, evaluation script writing agent, algorithm parameter tuning agent, and report generation agent.

[0101] The toolset layer encapsulates industrial testing tools based on the MCP (Model Context Protocol) framework, including hyperparameter optimization tools, national standard indicator calculation tools, and leak-free data partitioning tools.

[0102] The data layer uses SQLite and aiosqlite to build a global data sharing layer, storing the execution results and task status data of each agent in an asynchronous and concurrent manner, and supporting data consistency verification; the infrastructure layer is based on Docker and Kubernetes to implement containerized deployment, ensuring environment consistency and supporting elastic resource scheduling.

[0103] The navigation agent is configured to: monitor the task status in the data layer in real time, and perform dual verification of the current task status according to the preset industrial algorithm evaluation process rules and national standard compliance rules; recommend the next matching functional agent to the user only when the status meets the national standard requirements; after the functional agent executes the task and updates the results and status to the data layer, repeat the above monitoring, verification and recommendation process to form a closed-loop collaboration until the final evaluation report is output.

[0104] Furthermore, the pre-defined industrial algorithm evaluation process rules define a strict sequence of "data partitioning → evaluation script writing → algorithm parameter tuning → report generation". The national standard compliance rules include rules for data partitioning without leakage, rules for national standard compliance of evaluation indicators, and rules for national standard compliance of report formats.

[0105] Furthermore, the data partitioning agent is configured to perform leak-free partitioning of industrial datasets using a dual strategy of "physical components" and "time series", and integrates preprocessing functions such as missing value imputation and outlier removal.

[0106] Furthermore, the evaluation script writing agent has a built-in national standard indicator calculation logic library, which can automatically generate executable Python scripts containing calculation code for core national standard indicators such as accuracy, RMSE, precision, and recall. The algorithm tuning agent is configured to call optimization tools in the toolset layer to iteratively optimize the hyperparameters of industrial algorithms such as CNN and XGBoost. The report generation agent is configured to integrate data from the entire process and generate a compliance evaluation report mapped to national standard clauses.

[0107] Furthermore, the data stored in the data layer adopts JSON format and defines a dedicated data structure for industrial algorithm evaluation, including dedicated fields such as dataset type, national standard indicator identifier, leak-free partition identifier, and algorithm parameter range, to ensure the accuracy and efficiency of data interaction between intelligent agents.

[0108] Figure 2 The core collaborative mechanism of this invention—a dual-constraint collaborative process of "national standard anchoring + state matching"—is demonstrated. This process is driven by a guide agent.

[0109] The navigation agent continuously monitors the task status in the data layer (e.g., "status: data_uploaded").

[0110] When a state changes, the navigation agent not only checks whether the state corresponds to a step in the expected process, but also triggers the national standard compliance verification module. For example, when the "data partitioning complete" state is detected, it verifies whether the partitioning method meets the "no leakage" requirement in the national standard.

[0111] Only when both conditions of "correct status" and "compliance with national standards" are met will the navigation agent determine and recommend the next function agent to be executed to the user (or trigger automatically) based on the preset mapping table.

[0112] The recommended / triggered functional agents obtain input data from the data layer and call the corresponding tools in the toolset layer to perform their exclusive tasks (such as the data partitioning agent performing leak-free partitioning).

[0113] After the task is completed, the functional agent writes the processing results (such as the path of the split dataset) and the new task status (such as "status: data_split_done") back to the data layer.

[0114] Once the navigation agent detects that the data layer has been updated again, it returns to step 1 and starts the next round of scheduling loops until the final evaluation report is generated, at which point the process ends.

[0115] This "dual constraint" mechanism ensures that the evaluation process is not only automated, but also compliant with industry standards and auditable.

[0116] Secondly, the present invention provides an industrial artificial intelligence algorithm evaluation method for multi-agent collaboration applied to the above-mentioned system, executed by a navigation agent, comprising the following steps:

[0117] S1: Real-time monitoring of the current evaluation task status in the global data layer;

[0118] S2: Perform dual verification of the current task status according to the preset industrial algorithm evaluation process rules and national standard compliance rules;

[0119] S3: If the verification passes, accurately match and recommend the next functional agent to be executed based on the current status; the specific matching logic includes: if the status is "data has been uploaded and the format is compliant", match the data partitioning agent; if the status is "data partitioning is complete and there is no leakage", match the evaluation script writing agent; if the status is "evaluation script is generated and the indicator logic is compliant", match the algorithm parameter tuning agent; if the status is "algorithm parameter tuning is complete and the performance meets the standard", match the report generation agent.

[0120] S4: When the task status in the data layer is detected to be updated, return to step S1 and repeat until the evaluation process ends.

[0121] Figure 3 The operation process of this system will be further explained by taking a specific implementation example—the evaluation of the CWRU rolling bearing fault diagnosis algorithm:

[0122] (1) Task creation and data upload: Users create new assessment tasks on the Web interface, upload the CWRU dataset (containing 10 fault types and 10,000 samples) and the CNN algorithm model file to be assessed. The interaction layer receives the request and initializes the task status to "data to be partitioned" in the data layer.

[0123] (2) Tour scheduling and data partitioning: When the tour guide agent detects a new task and verifies that the data format is compliant, it recommends a data partitioning agent. After user confirmation, the data partitioning agent starts and adopts a strategy that combines "by bearing number (physical component)" and "by collection time sequence" to divide the dataset into a training set (8000 records) and a test set (2000 records) to ensure no information leakage.

[0124] (3) Evaluation script writing: The intelligent agent generates a script that conforms to GB / T 43555-2023, which includes the calculation logic of four core indicators: accuracy, precision, recall, and F1 score. The user uses the script to evaluate the algorithm and outputs the evaluation result 1. The status is updated to "Data partitioning completed, verification: GB / T 43555-2023 no leakage requirement is met".

[0125] (4) Automatic Script Generation: Once the navigation agent detects that the previous step has been completed and is compliant, it recommends an agent to write the evaluation script. Based on the task type (fault diagnosis) and the selected national standard (e.g., GB / T 43555-2023), this agent selects appropriate accuracy, precision, recall, and F1 score calculation logic from the template and indicator libraries to generate a directly runnable evaluation script. Users can run this script to obtain baseline evaluation results. The status is updated to "Evaluation script generated, indicators comply with national standards".

[0126] (5) Algorithm parameter optimization: The guide agent recommends an algorithm parameter tuning agent. This agent calls the Optuna tool encapsulated in the toolset layer to perform Bayesian optimization (default 50 iterations) on hyperparameters such as the learning rate (0.001-0.01) and the number of convolutional kernels (32-128) of the CNN model. The optimized model is saved. The status is updated to "Algorithm parameter tuning completed", and the accuracy is improved from 85% to 98.5%, resulting in evaluation result 2.

[0127] (6) Compliance Report Generation: The navigation agent will then generate the final recommended report. This agent collects all intermediate data and results (partition scheme, baseline results, optimization curves, optimal parameters, and final indicators) from the data layer and automatically generates a complete evaluation report containing data, methods, results, analysis, and a compliance statement, in accordance with the report format specified in the national standard. The status will be updated to "Evaluation report generated, task completed".

[0128] Throughout the process, the inputs, outputs, statuses, and compliance conclusions at any step are recorded in real time in the data layer, forming an immutable evaluation traceability chain.

[0129] The key to this invention lies in the fact that conventional technologies such as intelligent agents, rule engines, MCP protocols, and JSON data exchange are not simply stacked together, but creatively combined and deeply customized according to the specific requirements of the industrial algorithm evaluation field for "process standardization", "result compliance" and "system scalability", thus forming a brand-new, efficient and dedicated industrial algorithm evaluation collaborative system.

[0130] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-mentioned industrial algorithm evaluation method for multi-agent collaboration.

[0131] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described industrial algorithm evaluation method for multi-agent collaboration.

[0132] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent collaborative industrial artificial intelligence algorithm evaluation system, characterized in that, It adopts a six-level layered architecture, including: The user layer provides multiple operation interfaces and multi-role login functions; The interaction layer is used to parse user commands into system executable commands; The multi-agent core layer includes a scheduling center and at least four functional agents; Toolset layer: Industrial testing tools are encapsulated based on the MCP framework; The data layer uses an asynchronous database to build a global data sharing layer; The infrastructure layer is used to enable containerized deployment and resource scheduling. The scheduling center of the multi-agent core layer is the navigation agent, and the at least four functional agents include the data partitioning agent, the evaluation script writing agent, the algorithm parameter tuning agent, and the report generation agent. The data layer is used to store task status and execution result data written by each intelligent agent; The navigation agent is configured to: monitor the task status in the data layer in real time, and verify the current task status according to the preset industrial algorithm evaluation process rules and national standard compliance rules. When the status meets the standards and the national standard is compliant, the agent recommends the next matching functional agent to the user. After the functional agent executes the task and updates the data layer, the monitoring, verification and recommendation process is repeated until the evaluation process ends.

2. The multi-agent collaborative industrial artificial intelligence algorithm evaluation system according to claim 1, characterized in that, The preset industrial algorithm evaluation process rules include sequential execution of data partitioning, evaluation script writing, algorithm parameter tuning, and report generation steps; the national standard compliance rules include at least one of the following: data partitioning without leakage rules, evaluation indicator national standard compliance rules, and report format national standard compliance rules.

3. The multi-agent collaborative industrial artificial intelligence algorithm evaluation system according to claim 1, characterized in that, The data partitioning agent is configured to partition the industrial dataset without leakage based on a dual strategy of "physical components" and "time series", and to preprocess the data by filling in missing values ​​and removing outliers.

4. The multi-agent collaborative industrial artificial intelligence algorithm evaluation system according to claim 1, characterized in that, The evaluation script writing agent is configured to generate an executable evaluation script containing preset national standard indicator calculation logic based on the national standard indicator library. The algorithm tuning agent is configured to call the hyperparameter optimization tool in the toolset layer to iteratively optimize the hyperparameters of the target industrial algorithm; the report generation agent is configured to integrate the data from the entire process and generate an evaluation report that includes a statement of compliance with national standards.

5. The multi-agent collaborative industrial artificial intelligence algorithm evaluation system according to claim 1, characterized in that, The industrial testing tools encapsulated in the toolset layer include at least: a hyperparameter optimization tool based on Optuna, a national standard index calculation tool, and a leak-free data partitioning tool.

6. The multi-agent collaborative industrial artificial intelligence algorithm evaluation system according to claim 1, characterized in that, The data stored in the data layer is in JSON format, and the data structure defines special fields required for industrial algorithm evaluation, including at least one of the following: dataset type field, national standard indicator identifier field, no leakage division identifier field, and parameter range field.

7. A method for evaluating industrial artificial intelligence algorithms involving multi-agent collaboration, characterized in that, Applied to the system according to any one of claims 1-6, executed by a navigation agent, and comprising the following steps: S1: Monitor the current evaluation task status in the global data layer; S2: Verify the current task status according to the preset industrial algorithm evaluation process rules and national standard compliance rules; S3: If the verification passes, match and recommend the next functional agent to be executed based on the current state; S4: After detecting that the task status in the data layer has been updated, return to step S1.

8. The method for evaluating multi-agent collaborative industrial artificial intelligence algorithms according to claim 7, characterized in that, In step S3, matching the next functional agent to be executed based on the current state specifically includes: If the current status is that the data has been uploaded and the format complies with national standards, then match the data to divide the intelligent agent; If the current state is that the data partitioning is complete and meets the national standard requirements for no data leakage, then the evaluation script is matched to write the intelligent agent. If the current state is that the evaluation script has been generated and the indicator logic complies with national standards, then the matching algorithm parameter tuning agent will be used. If the current state is that the algorithm parameter tuning is complete and the performance meets the requirements, then the matching report generates an agent.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the industrial artificial intelligence algorithm evaluation method for multi-agent collaboration as described in claim 7 or 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the industrial artificial intelligence algorithm evaluation method for multi-agent collaboration as described in claim 7 or 8.

Images