An intelligent power training quality evaluation method and device based on data driving and random forest fusion AHP

Abstract

This invention discloses an intelligent evaluation method and device for power training quality based on a data-driven approach and a random forest-integrated AHP (Automatic Hierarchical Method) system. It relates to the field of intelligent evaluation technology for vocational skills training. The method involves collecting multi-source raw training data and performing standardized preprocessing; labeling training quality with five levels of tags by experts; training a training quality evaluation model using a Bayesian-optimized random forest algorithm and verifying the model's accuracy; calculating the marginal contribution of each indicator using the SHAP algorithm; aggregating scores for secondary training dimensions and visualizing them; and finally, automatically outputting the student training quality grading results and a complete evaluation report. This invention integrates an AHP hierarchical architecture with automatic data weighting using random forest, significantly reducing the bias of subjective human evaluation. It can accurately locate the weak dimensions of student training, adapt to the fully automated and objective evaluation of the entire process of power virtual simulation training, and effectively solve the technical problems of strong subjectivity, low data utilization, and lack of interpretability in traditional manual evaluation.

Description

Technical Field

[0001] This invention relates to the field of intelligent assessment technology for vocational skills training, and more specifically to an intelligent assessment method and device for power training quality based on a fusion of data-driven and random forest AHP. Background Technology

[0002] Currently, electrical engineering training is characterized by strong practical application, high safety risks, and standardized processes. Traditional training quality assessment mainly relies on offline manual scoring by teachers, which has three major drawbacks: 1. High subjectivity in evaluation: The current mainstream quantitative evaluation methods are the Analytic Hierarchy Process (AHP), Delphi expert survey method, TOPSIS method, etc. The weight of the indicators depends entirely on the subjective scoring of experts. Different evaluators have different scoring standards, and the same training sample of trainees is prone to significant differences in evaluation results, resulting in insufficient evaluation credibility. 2. Low data utilization efficiency: Virtual simulation training systems can collect massive amounts of heterogeneous data from multiple dimensions, such as student operation logs, assessment scores, safety behaviors, and training duration, throughout the entire process. However, traditional manual assessments only select a small number of assessment scores as evaluation criteria, leaving a large amount of behavioral data idle and unable to uncover deeper training characteristics such as students' practical shortcomings, safety habits, and training attitudes. 3. Lack of interpretability and insufficient accuracy in assessment: Conventional linear weighted assessment models are difficult to fit the complex nonlinear relationships between training indicators, and have large prediction errors for five-level multi-class quality grading tasks; at the same time, the model output only provides the final grade and cannot pinpoint the specific dimensions and indicators that lead to low training quality, making it difficult to support targeted improvement for trainees and optimization of training management.

[0003] Existing machine learning training and evaluation schemes have two main shortcomings: pure machine learning models lack a structured hierarchical index system, and the output results have no business dimension interpretation; pure AHP hierarchical evaluation relies entirely on manual weighting and cannot achieve fully data-driven objective evaluation.

[0004] Therefore, how to integrate AHP hierarchical index architecture, random forest automatic weight learning, Bayesian hyperparameter optimization, and SHAP index contribution analysis into a unified intelligent evaluation scheme for power training is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an intelligent evaluation method and device for power training quality based on a data-driven approach and random forest-integrated AHP. It combines the advantages of hierarchical structure of AHP indicators with the advantages of automatic weighting of random forest data. The entire process relies on the simulation system to collect the original training data of students to complete the objective evaluation. Bayesian optimization is used to improve the classification accuracy of the model, and the SHAP algorithm is used to quantify the contribution of each indicator and each training dimension to the evaluation level. This enables automatic grading of training quality, visualization and location of shortcomings, reduces manual intervention, and improves the objectivity, accuracy and interpretability of the evaluation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A smart evaluation method for power training quality based on a fusion of data-driven approach and random forest with AHP (Automatic Hierarchical Methodology) includes: S1. Construct a hierarchical training quality evaluation index system, collect multi-source training raw data and complete standardized preprocessing; S2. Retain the hierarchical training quality evaluation index system and divide the standardized training raw data into training set and test set according to a fixed ratio; S3. Construct a random forest classification model, use the Bayesian optimization algorithm to globally optimize the hyperparameters of the random forest classification model, and use the training set to train a data-driven training quality evaluation model. S4. Verify the accuracy of the training quality assessment model using the test set; S5. Calculate the Shapley marginal contribution value of each third-level indicator based on the SHAP algorithm, aggregate the contribution values ​​of indicators under the same second-level dimension to obtain the dimension comprehensive score, and generate a dimension contribution visualization chart. S6. Input the standardized indicator data of the trainees to be evaluated into the completed training quality assessment model, output the training quality grading results, and generate and export a visual training assessment report by combining the dimensional contribution visualization chart.

[0007] Optionally, S1 specifically includes: S101. Establish a three-tiered training quality assessment index system. The first-level index is the training quality of students, which is further divided into 6 second-level assessment dimensions, with a total of 17 third-level quantitative assessment indicators. The third-level indicators are divided into positive indicators and negative indicators. The original data sources for each indicator include online theoretical tests, offline written tests, virtual simulation system operation logs, system security alarm records, and offline teacher manual ledger records. Data collection covers the entire training cycle, including the learning stage and the assessment stage. S102. Cleaning and processing of multi-source raw training data, including duplicate sample removal, abnormal operation data filtering, and missing value imputation; missing values ​​are processed using mean imputation. S103. Positive indicator transformation: Positive indicators retain their original values ​​directly; negative indicators are transformed differently in three categories: error step occurrence rate for ratio-type negative indicators is transformed using extreme value linear transformation; time and standard deviation-type negative indicators are transformed using target benchmarking; and frequency-type negative indicators are transformed using thresholding. S104. Perform min-max normalization to map all positively normalized index values ​​to the [0,1] interval.

[0008] Optionally, S2 specifically includes: S201. Based on the hierarchical training quality assessment index system, the hierarchical logic of the first-second-third level indicators is retained, and experts are only invited to give a five-level training quality label for the complete training sample. S202. Randomly divide all standardized sample datasets into a 7:3 ratio, with 70% of the samples used as the training set and 30% used as the independent test set.

[0009] Optionally, S3 specifically includes: S301. Using the three-level indicators of the training set as the input features of the model and the five-level training quality labels as the classification output, construct a random forest classification model. S302. Use Bayesian optimization algorithm to globally optimize the hyperparameters of random forest classification model: fit the model classification accuracy objective function with Gaussian regression surrogate model, iteratively sample and update the posterior distribution by PI improvement of probability acquisition function, and output the optimal hyperparameter combination after iterative convergence. S303. Reconstruct the random forest model using optimal hyperparameters, complete model training using the training set, automatically learn the data-driven weights of each third-level indicator, and obtain a data-driven training quality evaluation model. Optionally, S4 specifically includes: using the test set and employing six indicators—confusion matrix, macro-average accuracy, macro-average precision, macro-average recall, macro-average F1 score, and AUC value—to evaluate the performance of the training quality evaluation model.

[0010] Optionally, S5 specifically includes: S501. Introduce the Shapley value from game theory to construct the SHAP interpretation model. For any student training sample, calculate the SHAP marginal contribution value of each third-level indicator. S502. Collect all SHAP values ​​of the third-level indicators under the same second-level evaluation dimension to obtain the comprehensive contribution score of each training dimension. S503 generates radar charts and visualizations based on the comprehensive contribution scores of each training dimension, intuitively displaying the trainees' shortcomings in six dimensions of training: knowledge mastery, skill proficiency, process standardization, safety awareness, learning attitude, and operational stability.

[0011] Optionally, S6 specifically includes: S601. Input the standardized indicator data of the trainees to be evaluated into the training quality assessment model that has been trained and passed the accuracy verification, and output the final five-level training quality level. S602. Combine the contribution data of SHAP dimension, the original scores of each indicator, and the positiveized scores to generate a standardized evaluation report. The report includes the student's total score level, the sub-scores of each dimension, the identification of weak indicators, and suggestions for training optimization. The S603 front-end interface displays radar charts and heat maps to visualize the evaluation results, allowing trainees and training administrators to view and export evaluation reports online.

[0012] A smart evaluation device for power training quality based on a fusion of data-driven and random forest AHP, comprising: Data acquisition module: connects to the virtual simulation training system and offline assessment and data entry terminal; used to collect original training data from multiple sources, including students' theoretical assessments, practical operations, safety behaviors, training duration, and violation alarms, and simultaneously completes data caching and import of offline manual ledger data; Data preprocessing module: Connects to the data acquisition module, and includes a missing value imputation unit, an indicator positive conversion unit, and a min-max normalization unit; it completes data cleaning, indicator positive-negative conversion, and unit unification and standardization processing. Model training module: Connected to the data preprocessing module, it includes an AHP hierarchical dataset partitioning unit, a Bayesian optimization unit, and a random forest training unit; used to partition training / test samples, iteratively optimize the random forest hyperparameters, and train a data-driven training quality classification model. Model Validation Module: Connects to the model training module and includes a built-in confusion matrix generation unit, macro-average index calculation unit, and ROC-AUC calculation unit; inputs the test set to complete model accuracy verification, and only models that meet the accuracy standard proceed to the next evaluation process; Feature Interpretation Module: Connects to the model validation module, and includes a built-in SHAP value calculation unit, dimensional contribution aggregation unit, and visualization plotting unit; calculates the marginal contribution of each indicator, summarizes the scores of the secondary dimensions, and generates radar charts and heatmap visualization files; Evaluation Output Module: Connects to the feature interpretation module, including a front-end display sub-unit, an automatic report generation sub-unit, and a file export sub-unit; outputs the five-level grading results of student training quality, visual charts, and exportable training evaluation reports.

[0013] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method and device for intelligent evaluation of power training quality based on data-driven and random forest-integrated AHP, which has the following beneficial effects: 1. Significantly reduce subjective bias in assessments and achieve fully data-driven objective evaluation. This invention retains the AHP hierarchical indicator system only for building business hierarchical logic, completely abandoning expert manual weight assignment, and automatically learning indicator weights from massive training operation data based on the random forest algorithm; only experts need to provide the final graded labels of the samples, greatly reducing manual intervention and avoiding the subjective errors brought about by expert scoring in traditional AHP and Delphi methods. The evaluation results are all generated based on the trainees' real operation data, which significantly improves objectivity.

[0014] 2. Integrating Bayesian optimization to improve the accuracy of five-level multi-class classification assessment.

[0015] For multi-class classification tasks with a five-level quality assessment system in practical training, the Bayesian optimization global search random forest is used to find the optimal hyperparameters. Compared with grid search and random search, it requires fewer iterations and has higher optimization efficiency. Combined with the macro-average accuracy evaluation index, it is adapted to the scenario of imbalanced training sample classes. The model's prediction accuracy, F1 score, and AUC value are significantly better than the traditional linear weighted and single decision tree evaluation models, and the classification judgment error is greatly reduced.

[0016] 3. SHAP layered interpretation enables end-to-end interpretability of the evaluation process, accurately pinpointing training weaknesses.

[0017] Existing machine learning evaluation models are mostly black-box outputs, only providing the final grade. This invention uses the SHAP algorithm to quantify the marginal contribution of each tertiary indicator to the evaluation result, aggregates contribution scores according to secondary dimensions, and generates a radar chart visualization result. Training managers and trainees can intuitively identify weak dimensions such as safety awareness, skill proficiency, and process standardization, as well as the corresponding specific indicators, and carry out targeted reinforcement training, solving the pain point that evaluation results cannot be applied to guide practical training optimization.

[0018] 4. Adapts to multi-source heterogeneous data in virtual simulation training, enabling fully automated evaluation throughout the entire process.

[0019] This invention establishes a 17-item three-level indicator collection system covering the entire learning and assessment cycle, compatible with various heterogeneous data such as online simulation logs, offline written tests, and manual records; it is equipped with a complete data cleaning, forwarding, and normalization standardization process, and the device can automatically complete the entire process of data collection, model reasoning, and report generation without the need for manual statistics and calculation, which greatly reduces the labor cost of practical training assessment and supports automated batch assessment of a large number of students.

[0020] 5. Balances structured business logic with machine learning nonlinear fitting capabilities

[0021] Using AHP alone cannot handle complex nonlinear relationships between indicators, and random forest alone lacks a business-layered interpretation framework. This invention integrates the two, using AHP to construct a business-layered indicator system for power training, and random forest to mine nonlinear relationships between indicators and automatically assign weights. It takes into account the business logic of industry training and the advantages of machine learning modeling, and is suitable for the standardized and high-safety-requirement business scenarios of power and electrical training. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 The structure diagram of the training quality evaluation model based on the random forest algorithm provided by this invention is shown below. Figure 2 The method evaluation process provided by this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention discloses an intelligent evaluation method for power training quality based on a fusion of data-driven and random forest AHP, such as... Figure 1 and Figure 2 As shown, it includes: S1: Construct a hierarchical training quality evaluation index system, collect multi-source training raw data and complete standardized preprocessing; S101: Establish a three-tiered training quality assessment index system. The first-level index is the training quality of students, which is further divided into 6 second-level assessment dimensions, with a total of 17 third-level quantitative assessment indicators. The index system is detailed in Table 1. The third-level indicators are divided into positive indicators and negative indicators. The original data sources for each indicator include online theoretical tests, offline written tests, virtual simulation system operation logs, system security alarm records, and offline teacher manual ledger records. Data collection covers the entire training cycle, including the learning stage and the assessment stage. Table 1. Evaluation Index System for Practical Training Quality

[0026] S102: Cleaning and processing of multi-source raw training data, including duplicate sample removal, outlier data filtering, and missing value imputation; missing values ​​are processed using mean imputation, and the calculation formula is:

[0027] In the formula: n is the number of samples; For the first i One sample data; This represents the average sample data result.

[0028] S103: Indicator Positive Conversion Processing: (1) Positive indicators retain their original values ​​directly; positive indicators include the average score of theoretical assessment, the average score of practical assessment, the correctness rate of tool selection, the completeness of operation ticket filling, the completeness rate of operation steps, the compliance rate of wearing safety protective equipment, the execution rate of key actions of voltage testing / grounding, the total number of project completions, the frequency of repeated training, and the total learning time of the project. (2) Negative indicators are classified into three categories for differentiated positive transformation: ① For the ratio-based negative indicator, the error step occurrence rate, an extreme value linear transformation is used: positive value = 1 - error step occurrence rate; ② Negative indicators related to time and standard deviation (average project time, fluctuation coefficient of operation time, standard deviation of project score) are positiveized using the target benchmarking method. The calculation method is as follows: Positive value =

[0029] In the formula This is the original value of the indicator.

[0030] ③ Negative indicators related to frequency (number of times the intelligent learning guide is used, number of times of violation, number of times forced interruption) are positively evaluated using a threshold method. The calculation method is as follows: Positive value =

[0031] In the formula The original value of the indicator. , These are the thresholds for the number of times the test is qualified or excellent.

[0032] S104: Min-max normalization maps all positively normalized index values ​​to the [0,1] interval, eliminating differences in the dimensions of different indices. The normalization calculation formula is:

[0033] In the formula: x For sample data; and These are the minimum and maximum values ​​in the sample data, respectively.

[0034] S2: Construct an AHP hierarchical sample dataset, divide it into training and test sets, and label the evaluation tags; S201: Based on the AHP hierarchical architecture, the hierarchical logic of first-level, second-level, and third-level indicators is retained. Only experts are invited to give a five-level training quality label (excellent / good / qualified / needs improvement / unqualified) for complete training samples. No arbitrary indicator weights are manually assigned, thus eliminating subjective bias in weight assignment. S202: Randomly divide all standardized sample datasets in a 7:3 ratio, with 70% of the samples used as the training set and 30% used as the independent test set.

[0035] S3: Random Forest Training Quality Classification Model Based on Bayesian Optimization Training

[0036] S301: Using 17 normalized three-level indicators from the training set as input features for the model and five-level training quality labels as classification outputs, a random forest classification model is constructed. The random forest construction process includes bootstrap sampling, random feature selection, iterative growth of decision trees, and multi-tree ensemble voting prediction. The feature information gain is calculated as follows:

[0037] In the formula, Ent(K) is the information entropy (empirical entropy) of dataset K, which describes the degree of category confusion or uncertainty of the entire dataset before splitting. CEnt(A,K) is the conditional entropy, which represents the average remaining uncertainty of dataset K given feature A.

[0038] The voting prediction logic is as follows:

[0039] In the formula, This is the output result. It is the independent variable. It is the number of decision trees. It is the parameter vector of the b-th tree. It is the CART algorithm for constructing decision trees.

[0040] S302: The Bayesian optimization algorithm is used to globally optimize the hyperparameters of the random forest: the Gaussian regression surrogate model is used to fit the model classification accuracy objective function, the probability acquisition function is improved by PI and the posterior distribution is updated by iterative sampling, and the optimal hyperparameter combination is output after iterative convergence. S303: The random forest model is reconstructed using optimal hyperparameters. The model is trained using the training set and automatically learns the weights driven by the data of each of the three levels of indicators, without the need for manual weighting.

[0041] S4: Complete the quantitative verification of model accuracy based on the test set; The model performance is evaluated using six metrics: confusion matrix, macro average accuracy, macro average precision, macro average recall, macro average F1 score, and AUC value. The confusion matrix visualizes the matching relationship between the real labels and the model's predicted labels, and the ROC curve calculates the AUC value to characterize the overall classification performance of the model.

[0042] The formula for calculating the macro average evaluation index is:

[0043] In the formula: , , , , representing accuracy, precision, recall, and F1 score calculated by macro average, respectively; n represents the total number of sample classes; i represents the i-th class; TP represents the number of samples that were predicted as positive; TN represents the number of samples that were predicted as negative; FP represents the number of samples that were predicted as positive; and FN represents the number of samples that were predicted as negative.

[0044] S5: Calculate the contribution of single indicators and single dimensions based on the SHAP algorithm to achieve hierarchical interpretable evaluation.

[0045] S501: Introduce the Shapley value from game theory to construct the SHAP interpretation model. For any student training sample, calculate the SHAP marginal contribution value of each tertiary indicator.

[0046] In the formula: The contribution of feature i to the evaluation result is called the Shapley value; N represents the set of all features; S is any subset of features that does not contain feature i. It is the characteristic number of set S; It is the contribution of the feature set S to the model evaluation output; It is a feature set containing feature i. Contribution to the model evaluation output.

[0047] S502: Collect all SHAP values ​​of the third-level indicators under the same second-level evaluation dimension to obtain the comprehensive contribution score of each training dimension; S503: Based on contribution scores in each dimension, a radar chart visualization is generated to intuitively display the trainees' shortcomings in six dimensions of practical training: knowledge mastery, skill proficiency, process standardization, safety awareness, learning attitude, and operational stability.

[0048] S6: Automatically generate a training quality grading assessment report and output visualized results; S601: Input the standardized indicator data of the trainees to be evaluated into the trained and accuracy-verified random forest model, and output the final five-level training quality level. S602: Combine SHAP dimension contribution data, original scores of each indicator, and positiveized scores to generate a standardized evaluation report. The report includes the student's total score level, sub-scores of each dimension, weak indicator identification, and training optimization suggestions. S603: The front-end interface displays radar charts and heat maps to visualize the evaluation results, and supports trainees and training administrators to view and export evaluation reports online.

[0049] A smart evaluation device for power training quality based on a fusion of data-driven and random forest AHP, comprising: Data acquisition module: connects to the virtual simulation training system and offline assessment and data entry terminal; used to collect original training data from multiple sources, including students' theoretical assessments, practical operations, safety behaviors, training duration, and violation alarms, and simultaneously completes data caching and import of offline manual ledger data; Data preprocessing module: Connects to the data acquisition module, and includes a missing value imputation unit, an indicator positive conversion unit, and a min-max normalization unit; it completes data cleaning, indicator positive-negative conversion, and unit unification and standardization processing. Model training module: Connected to the data preprocessing module, it includes an AHP hierarchical dataset partitioning unit, a Bayesian optimization unit, and a random forest training unit; used to partition training / test samples, iteratively optimize the random forest hyperparameters, and train a data-driven training quality classification model. Model Validation Module: Connects to the model training module and includes a built-in confusion matrix generation unit, macro-average index calculation unit, and ROC-AUC calculation unit; inputs the test set to complete model accuracy verification, and only models that meet the accuracy standard proceed to the next evaluation process; Feature Interpretation Module: Connects to the model validation module, and includes a built-in SHAP value calculation unit, dimensional contribution aggregation unit, and visualization plotting unit; calculates the marginal contribution of each indicator, summarizes the scores of the secondary dimensions, and generates radar charts and heatmap visualization files; Evaluation Output Module: Connects to the feature interpretation module, including a front-end display sub-unit, an automatic report generation sub-unit, and a file export sub-unit; outputs the five-level grading results of student training quality, visual charts, and exportable training evaluation reports.

[0050] The present invention will be further explained below with specific examples.

[0051] Example 1: Implementation Steps of the Intelligent Evaluation Method for Practical Training Quality

[0052] Step 1: Establish an indicator system and collect raw training data.

[0053] Based on Table 1, a three-level indicator system was constructed. The first-level indicator is the quality of trainees' practical training, which has 6 second-level dimensions: knowledge mastery, skill proficiency, process standardization, safety awareness, learning attitude, and operational stability, totaling 17 third-level quantitative indicators.

[0054] Data collection sources: The power virtual simulation system automatically collects students' theoretical written test scores, scores of each practical project, operation time, tool selection records, safety protection wearing logs, violation alarms, forced interruption records, training times, and learning time; offline teachers supplement and enter offline practical manual records of data, and all raw data are synchronously stored in the data collection module of the evaluation device.

[0055] Step 2: Standardization and preprocessing of multi-source training data

[0056] 1. Data cleaning: Remove duplicate training operation logs, filter samples with extreme abnormal time consumption and abnormal scores caused by system failures; fill in a small number of missing indicator data with mean imputation; 2. Positive conversion of indicators: Distinguish between 10 positive indicators and 7 negative indicators, and use the extreme value method, target benchmarking method, and threshold method to complete the positive conversion of values ​​respectively; the error step occurrence rate is converted by 1-original ratio; time fluctuation indicators such as average project time and score standard deviation are converted by target benchmarking method; frequency indicators such as number of violations and number of forced interruptions are converted by threshold method. 3. Normalization: All positive indicators are substituted into the normalization calculation formula and scaled to the 0~1 range to eliminate the dimensional differences between scores, durations, frequency, and ratios.

[0057] Step 3: Dataset Splitting and Labeling

[0058] All standardized samples were compiled, and each complete sample was labeled with a five-level quality label by experts in the power training industry: Excellent, Good, Qualified, Needs Improvement, and Unqualified, without providing any indicator weights; all samples were randomly divided into training and test sets in a 7:3 ratio.

[0059] Step 4: Bayesian optimization training of the random forest classification model

[0060] A random forest model is built using 17 normalized three-level indicators as input features and five-level labels as classification output: 1. Random Forest Single Tree Construction: Samples are extracted through bootstrapping, some features are randomly selected, and decision trees are grown with information gain as the optimal splitting criterion. Multiple decision trees are generated through repeated iterations, and the classification result is finally output by voting. 2. Bayesian hyperparameter optimization: Using the model's macro-average F1 score as the optimization objective, a Gaussian process surrogate model is constructed. The PI sampling function iteratively samples and updates the posterior distribution. After 20 iterations, the optimal number of trees, single tree depth, feature sampling ratio, and other hyperparameters are output. 3. The random forest is reconstructed using the optimal hyperparameters, and the model is trained using 70% of the training set, automatically learning the weights driven by the data of each indicator.

[0061] Step 5: Test Set Model Performance Validation

[0062] Input 30% of the test set into the trained model to generate a confusion matrix. Calculate the AUC value by plotting the ROC curve using the macro-average accuracy, precision, recall, and F1 score. If the macro-average F1 score is lower than the preset threshold, re-execute the Bayesian optimization iteration parameter tuning until the model accuracy reaches the target.

[0063] Step 6: Calculation and Visual Interpretation of SHAP Metric Contribution

[0064] 1. Call the SHAP algorithm and calculate the SHAP marginal contribution value of each tertiary indicator based on the Shapley value formula; 2. Collect the SHAP values ​​of all tertiary indicators under the same dimension according to the six secondary dimensions to obtain the comprehensive contribution score of each dimension; 3. A radar chart is generated using the scores of the six dimensions as coordinate axes to intuitively display the balance and shortcomings of trainees in each dimension of their practical training.

[0065] Step 7: Automated Assessment and Report Output

[0066] The system inputs standardized indicator data of trainees to be evaluated into a validated random forest model, and outputs a five-level training quality rating. The system automatically integrates the original data of each indicator, positive scores, dimensional SHAP contribution scores, and weakness analysis to generate a complete evaluation report. The front end displays the results as radar charts and heat maps, and supports trainees and training administrators to view and export PDF evaluation reports online.

[0067] Example 2: Hardware and Software Implementation of Practical Training Quality Assessment Device

[0068] The hardware of the device includes a local server, a virtual simulation training terminal, an offline data entry terminal, and a display terminal; the software is deployed on the local server and consists of six functional modules: 1. Data Acquisition Module: Connects to the power virtual simulation training platform via API interface to capture the entire process operation log in real time; sets up a manual input port to receive offline training records from teachers and complete unified data caching; 2. Data preprocessing module: Built-in Python data processing scripts automatically perform missing value imputation, indicator positiveization, min-max normalization, and output standardized feature matrix; 3. Model Training Module: Integrates Bayesian optimization toolkit and random forest algorithm library, automatically partitions datasets, iteratively optimizes, trains classification models, and stores the optimal model file; 4. Model Validation Module: Automatically calculates the confusion matrix, macro-average, four major evaluation indicators, and AUC value, outputs a model performance validation report, and automatically triggers parameter retuning for unqualified models; 5. Feature Interpretation Module: Built-in SHAP calculation tool, automatically collects dimensional contribution values, and calls plotting tools to generate radar charts and heatmap visualization files; 6. Evaluation Output Module: Connects to the front-end web page to display the grading results and visual charts, with built-in report templates, and allows for one-click generation and export of standardized training evaluation reports.

[0069] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0070] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligently evaluating the quality of power training based on a fusion of data-driven approaches and random forests using AHP, characterized in that, include: S1. Construct a hierarchical training quality evaluation index system, collect multi-source training raw data and complete standardized preprocessing; S2. Retain the hierarchical training quality evaluation index system and divide the standardized training raw data into training set and test set according to a fixed ratio; S3. Construct a random forest classification model, use the Bayesian optimization algorithm to globally optimize the hyperparameters of the random forest classification model, and use the training set to train a data-driven training quality evaluation model. S4. Verify the accuracy of the training quality assessment model using the test set; S5. Calculate the Shapley marginal contribution value of each third-level indicator based on the SHAP algorithm, aggregate the contribution values ​​of indicators under the same second-level dimension to obtain the dimension comprehensive score, and generate a dimension contribution visualization chart. S6. Input the standardized indicator data of the trainees to be evaluated into the completed training quality assessment model, output the training quality grading results, and generate and export a visual training assessment report by combining the dimensional contribution visualization chart.

2. The intelligent evaluation method for power training quality based on a fusion of data-driven and random forest AHP as described in claim 1, characterized in that, S1 specifically includes: S101. Establish a three-tiered training quality assessment index system. The first-level index is the training quality of students, which is further divided into 6 second-level assessment dimensions, with a total of 17 third-level quantitative assessment indicators. The third-level indicators are divided into positive indicators and negative indicators. The original data sources for each indicator include online theoretical tests, offline written tests, virtual simulation system operation logs, system security alarm records, and offline teacher manual ledger records. Data collection covers the entire training cycle, including the learning stage and the assessment stage. S102. Cleaning and processing of multi-source raw training data, including duplicate sample removal, abnormal operation data filtering, and missing value imputation; missing values ​​are processed using mean imputation. S103. Positive indicator transformation: Positive indicators retain their original values ​​directly; negative indicators are transformed differently in three categories: error step occurrence rate for ratio-type negative indicators is transformed using extreme value linear transformation; time and standard deviation-type negative indicators are transformed using target benchmarking; and frequency-type negative indicators are transformed using thresholding. S104. Perform min-max normalization to map all positively normalized index values ​​to the [0,1] interval.

3. The intelligent evaluation method for power training quality based on a fusion of data-driven and random forest AHP as described in claim 1, characterized in that, S2 specifically includes: S201. Based on the hierarchical training quality assessment index system, the hierarchical logic of the first-second-third level indicators is retained, and experts are only invited to give a five-level training quality label for the complete training sample. S202. Randomly divide all standardized sample datasets into a 7:3 ratio, with 70% of the samples used as the training set and 30% used as the independent test set.

4. The intelligent evaluation method for power training quality based on a fusion of data-driven and random forest AHP as described in claim 2, characterized in that, S3 specifically includes: S301. Using the three-level indicators of the training set as the input features of the model and the five-level training quality labels as the classification output, construct a random forest classification model. S302. Use Bayesian optimization algorithm to globally optimize the hyperparameters of random forest classification model: fit the model classification accuracy objective function with Gaussian regression surrogate model, iteratively sample and update the posterior distribution by PI improvement of probability acquisition function, and output the optimal hyperparameter combination after iterative convergence. S303. The random forest model is reconstructed using the optimal hyperparameters. The model is trained using the training set and automatically learns the data-driven weights of each of the three-level indicators to obtain a data-driven training quality evaluation model.

5. The intelligent evaluation method for power training quality based on a fusion of data-driven and random forest AHP as described in claim 1, characterized in that, The S4 specifically includes: using the test set and employing six indicators—confusion matrix, macro average accuracy, macro average precision, macro average recall, macro average F1 score, and AUC value—to complete the performance evaluation of the training quality assessment model.

6. The intelligent evaluation method for power training quality based on a fusion of data-driven and random forest AHP as described in claim 1, characterized in that, S5 specifically includes: S501. Introduce the Shapley value from game theory to construct the SHAP interpretation model. For any student training sample, calculate the SHAP marginal contribution value of each third-level indicator. S502. Collect all SHAP values ​​of the third-level indicators under the same second-level evaluation dimension to obtain the comprehensive contribution score of each training dimension. S503 generates radar charts and visualizations based on the comprehensive contribution scores of each training dimension, intuitively displaying the trainees' shortcomings in six dimensions of training: knowledge mastery, skill proficiency, process standardization, safety awareness, learning attitude, and operational stability.

7. The intelligent evaluation method for power training quality based on a fusion of data-driven and random forest AHP as described in claim 1, characterized in that, S6 specifically includes: S601. Input the standardized indicator data of the trainees to be evaluated into the training quality assessment model that has been trained and passed the accuracy verification, and output the final five-level training quality level. S602. Combine the contribution data of the SHAP dimension, the original scores of each indicator, and the positiveized scores to generate a standardized evaluation report. The S603 front-end interface displays radar charts and heat maps to visualize the evaluation results, allowing trainees and training administrators to view and export evaluation reports online.

8. A smart evaluation device for power training quality based on a fusion of data-driven and random forest AHP, characterized in that, The intelligent evaluation method for power training quality based on a fusion of data-driven and random forest AHP, as described in any one of claims 1-7, includes: Data acquisition module: connects to the virtual simulation training system and offline assessment and data entry terminal; used to collect original training data from multiple sources, including students' theoretical assessments, practical operations, safety behaviors, training duration, and violation alarms, and simultaneously completes data caching and import of offline manual ledger data; Data preprocessing module: Connects to the data acquisition module, and includes a missing value imputation unit, an indicator positive conversion unit, and a min-max normalization unit; it completes data cleaning, indicator positive-negative conversion, and unit unification and standardization processing. Model training module: Connected to the data preprocessing module, it includes an AHP hierarchical dataset partitioning unit, a Bayesian optimization unit, and a random forest training unit; used to partition training / test samples, iteratively optimize the random forest hyperparameters, and train a data-driven training quality classification model. Model Validation Module: Connects to the model training module and includes a built-in confusion matrix generation unit, macro-average index calculation unit, and ROC-AUC calculation unit; inputs the test set to complete model accuracy verification, and only models that meet the accuracy standard proceed to the next evaluation process; Feature Interpretation Module: Connects to the model validation module, and includes a built-in SHAP value calculation unit, dimensional contribution aggregation unit, and visualization plotting unit; calculates the marginal contribution of each indicator, summarizes the scores of the secondary dimensions, and generates radar charts and heatmap visualization files; Evaluation Output Module: Connects to the feature interpretation module, including a front-end display sub-unit, an automatic report generation sub-unit, and a file export sub-unit; outputs the five-level grading results of student training quality, visual charts, and exportable training evaluation reports.

Images