Operation performance evaluation method, computer equipment and computer program product

Abstract

The invention is suitable for the technical field of nuclear power human factor engineering, and provides an operation performance evaluation method, computer equipment and a computer program product, and the method comprises the steps: obtaining multi-dimensional data of a target operator in a process of achieving a task target in a target scene; generating an experimental sample set, and training the basic model based on the experimental sample set to obtain a target evaluation model; and inputting the multi-dimensional data into the target evaluation model to obtain a personnel performance score, wherein the personnel performance score represents the operation performance of the target operator for realizing the task target in the target scene. According to the method and the device, the operation performance of the task target under the target scene realized by the target operator in the digital main control room can be efficiently and accurately evaluated in real time.

Description

Technical Field

[0001] This application belongs to the field of nuclear power human factors engineering technology, and in particular relates to an operational performance evaluation method, computer equipment and computer program product. Background Technology

[0002] Nuclear power plants have numerous systems and equipment, fluctuating operating conditions, and highly coupled systems with complex operational tasks. In the digital control room of a nuclear power plant, operators face an increasingly complex operating environment: the widespread use of digital instruments and control systems has led to an exponential increase in information volume; the "keyhole effect" of human-machine interfaces exacerbates the situational awareness load, causing difficulties for operators in situational cognition and potentially leading to numerous human error patterns. Operators need to accurately judge the system status and quickly implement correct mitigation measures, placing a huge workload and mental stress on them, easily inducing misjudgments and misoperations, thereby jeopardizing the safe operation of the nuclear power plant.

[0003] Insufficient understanding of operator error mechanisms and a lack of early warning systems for human error under high-intensity cognitive operations have become a technical bottleneck and research hotspot in the international development of future reactors. Furthermore, existing operator performance evaluation methods have significant limitations and cannot effectively adapt to the dynamic needs of digital control rooms. Summary of the Invention

[0004] In view of this, embodiments of this application provide an operational performance evaluation method, computer equipment, and computer program products to achieve efficient and accurate real-time evaluation of the operational performance of target operators in a target scenario within a digital control room.

[0005] A first aspect of this application provides an operational performance evaluation method applied to computer equipment in a digital control room, the method comprising: Acquire multi-dimensional data on the process by which target operators achieve task objectives in a target scenario; Generate an experimental sample set and train a base model based on the experimental sample set to obtain the target evaluation model; The multi-dimensional data is input into the target evaluation model to obtain the personnel performance score, which represents the operational performance of the target operator in achieving the task objective in the target scenario.

[0006] In one implementation of the first aspect, the multi-dimensional data includes main task measurement data, workload data, situational cognition data, and physiological data; The main task measurement data includes task completion time, operation accuracy, procedure compliance, and error steps; The workload data includes the target operator's task load index scale score for the key main tasks associated with the task target; The contextual awareness data includes the contextual awareness scoring scale scores of the target operator for the key main tasks associated with the task target; The physiological data includes electrocardiogram (ECG) data, electroencephalogram (EEG) data, and eye movement data.

[0007] In one implementation of the first aspect, generating an experimental sample set and training a base model based on the experimental sample set to obtain a target evaluation model includes: Based on the procedures for the task objectives in the experimental scenario, at least one task sequence is generated, and the task sequence includes at least one key main task. Obtain an experimental dataset, which includes multiple experimental data points. The experimental data includes multi-dimensional data on the execution of task sequences by operators in the experimental scenario to complete task objectives. The multi-dimensional data includes main task measurement data, workload data, situational cognition data, and physiological data. Determine the completion status of each key main task in the task sequence and the completion status of the task sequence itself; The experimental data corresponding to the completed task sequence are identified as experimental samples. For the sequence of tasks that are already completed The following formula is used to generate the corresponding total task score for the experimental samples:

[0008]

[0009] in, The total task score for personnel is the k-th task sequence that includes j key main tasks. The weights of task sequence k are... Let L be the score for task sequence k, and L be the total number of key main tasks in task sequence k. The task load metric score for the j-th critical primary task in task sequence k. The score for the j-th critical main task, when the j-th critical main task is in an incomplete state. The value is 0 when the j-th critical main task is in a completed state. =1, Let be the weight of the j-th key main task; The total task score of the personnel is used as the true value label of the experimental sample, and the experimental sample is added to the experimental sample set; The target evaluation model is obtained by training the basic model based on the experimental sample set.

[0010] In one implementation of the first aspect, the key main task includes at least one key sub-task, and the key sub-task includes at least one key step; The method further includes: If the completion time of all key subtasks in the key main task is within the time limit of the key subtask, the key main task is confirmed to be completed. If the completion time of any key subtask in the key main task is not within the time limit of the key subtask, the key main task is confirmed to be incomplete. If all the key main tasks in the task sequence are in a completed state, the task sequence is confirmed to be in a completed state. If any key main task in the task sequence is incomplete, the task sequence is confirmed to be incomplete.

[0011] In one implementation of the first aspect, the step of training a base model based on the experimental sample set to obtain a target evaluation model includes: Configure an attention head for each dimension of the multi-dimensional data and construct the basic model using a multi-head attention mechanism; A training set is partitioned from the experimental sample set, and the base model is iteratively trained adversarially based on the following formula and the training set:

[0012] in, For input features, For adversarial examples, For the amplitude of the disturbance, This is a sign function used to control the direction of the disturbance. loss function For input features The gradient; Where N is the number of samples in the training set. It is the true value of the i-th sample in the training set. It is the base model prediction value of the i-th sample in the training set; When the value of the loss function converges, the final base model is output as the target evaluation model.

[0013] In one implementation of the first aspect, before acquiring multi-dimensional data on the process of the target operator achieving the task objective in the target scenario, the method further includes: Obtain multidimensional performance features to be identified ,in This represents the performance feature to be identified in the nth dimension; The multidimensional performance features to be identified are extended into a polynomial-generated extended feature vector that includes higher-order terms and interaction terms. ; The polynomial-generated extended feature vectors are used as inputs to construct the following ridge regression model:

[0014] in, To predict personnel performance scores, This is a vector of regression coefficients; The loss function of the ridge regression model described above is: ; in, For actual personnel performance scores, Mean squared error is used to measure the difference between the predicted value and the actual value. This is an L2 regularization term used to suppress model complexity; Multiple key features are determined from the multidimensional performance features to be identified based on the absolute values ​​of the regression coefficients, and the key features are divided into multidimensional data.

[0015] In one implementation of the first aspect, after inputting the multi-dimensional data into the target evaluation model to obtain the personnel performance score, the method further includes: Calculate the quartiles of the same batch of data, where the quartiles include the upper quartile, the median, and the lower quartile. Based on the relationship between the personnel performance score and the quartiles, the performance level corresponding to the personnel performance score is determined, and the performance levels from high to low are Level 1, Level 2, Level 3, and Level 4. If the operation accuracy rate is lower than the preset accuracy rate, the performance level will be downgraded by one level. If the task completion time exceeds the task completion time limit by a preset proportion, the performance level will be downgraded by one level.

[0016] In one implementation of the first aspect, after inputting the multi-dimensional data into the target evaluation model to obtain the personnel performance score, the method further includes: Each feature is calculated based on the following marginal utility calculation formula. The marginal contribution is calculated and interpretability analysis results are generated: ; in, Represents the complete feature set of the model; express It does not contain any subset of feature i; This means that only a subset of samples x are retained. The values ​​of the corresponding features; This means that the model uses only the features of a subset S to correspond to the predicted values ​​of x; Representation of features Average contribution to performance forecast output.

[0017] A second aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0018] A third aspect of this application provides a computer program product, including a computer program that, when run, causes the method described in the first aspect to be executed.

[0019] The beneficial effect of the first aspect of the embodiments of this application is: by acquiring multi-dimensional data of the target operator in the process of achieving the task objective in the target scenario, generating an experimental sample set, and training a basic model based on the experimental sample set to obtain a target evaluation model, and then inputting the multi-dimensional data into the target evaluation model to obtain a personnel performance score, the personnel performance score representing the target operator, thereby realizing the real-time, efficient and accurate evaluation of the operational performance of the target operator in achieving the task objective in the target scenario in the digital control room.

[0020] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram illustrating the implementation process of the operational performance evaluation method provided in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the implementation process of the operational performance evaluation method provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the implementation process of the operational performance evaluation method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the framework of the target evaluation model provided in the embodiments of this application; Figure 5This is a schematic diagram illustrating the implementation process of the operational performance evaluation method provided in the embodiments of this application; Figure 6 This is a schematic diagram illustrating the implementation process of the operational performance evaluation method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the computer device provided in an embodiment of this application; Figure 8 This is a schematic diagram of a computer program product provided in an embodiment of this application. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0024] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0025] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0027] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0029] This application provides an operational performance evaluation method for real-time, efficient, and accurate assessment of the operational performance of target operators in a digital control room as they achieve task objectives within a target scenario. The method acquires multi-dimensional data from the target operators during the process of achieving task objectives in a target scenario, generates an experimental sample set, trains a base model based on the experimental sample set to obtain a target evaluation model, and then inputs the multi-dimensional data into the target evaluation model to obtain a personnel performance score. This personnel performance score represents the target operator, enabling real-time, efficient, and accurate assessment of their operational performance in a digital control room as they achieve task objectives within the target scenario.

[0030] In nuclear power settings, under both routine operation and accident conditions, personnel need to continuously process and understand equipment status information and operating parameters. "Performance" refers to the degree to which personnel perform activities to achieve goals. Therefore, personnel need to engage in significant mental and even physical activity to maintain a good performance level. "Performance" and "performance level" are common terms in human factors engineering. In some embodiments of this application, operational performance corresponds to the reliability of the operator's actions in achieving task objectives.

[0031] The operational performance evaluation method provided in this application can be applied to computer devices such as tablet computers, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and servers. This application does not impose any restrictions on the specific type of computer device.

[0032] like Figure 1 As shown in the illustration, this application provides an operational performance evaluation method applied to computer equipment in a digital control room. The method includes: Step S1: Obtain multi-dimensional data on the process by which the target operator achieves the task objective in the target scenario.

[0033] In this application, the digital control room refers to the nuclear power plant's digital control room. The target scenarios are nuclear power-related operational scenarios within the nuclear power plant's digital control room, including reactor operation monitoring, online equipment monitoring, radiation protection and environmental monitoring, emergency fault handling, and emergency response scenarios in the event of major accidents. All these scenarios require performance evaluation of the operators performing the tasks to ensure the safety and reliability of daily operations and emergency responses during accidents. The target operator is the nuclear power plant's digital control room operator.

[0034] In the application, based on the procedures for the task objectives under the target scenario, at least one task sequence is generated. The task sequence includes at least one key main task, each key main task includes at least one key sub-task, and each key sub-task includes at least one key step. A step is a specific, actionable sub-objective, categorized into three types: detection, action, and consequence. It represents the operations that operators must perform to achieve the task objectives.

[0035] In application, critical primary tasks and critical sub-tasks refer to the control, operation, and intervention activities on safety-related equipment and systems that must be personally and directly performed by operators through a digital human-machine interface (including necessary interface management operations) to maintain or restore the unit's safe state under all operating conditions in a nuclear power scenario. These tasks typically do not include purely informational monitoring, inspection, recording, and communication, but rather refer to the final safety actions that result in substantial changes to equipment status or process parameters and have a direct impact on the nuclear safety barrier. Critical primary tasks and critical sub-tasks are jointly determined by operations personnel and human factors experts.

[0036] Step S2: Generate an experimental sample set and train a base model based on the experimental sample set to obtain the target evaluation model.

[0037] Step S3: Input the multi-dimensional data into the target evaluation model to obtain the personnel performance score, which represents the operational performance of the target operator in achieving the task objective in the target scenario.

[0038] In the application, before step S3, the completion status of the task sequence corresponding to the task objective is determined, and step S3 is entered if the task sequence is completed.

[0039] In applications, multi-dimensional data is normalized before being input into the target evaluation model, for example, by using Z-score normalization.

[0040] In one embodiment, the multidimensional data includes primary task measurement data, workload data, situational awareness data, and physiological data.

[0041] The main task measurement data includes task completion time, operation accuracy, procedure compliance, and error steps.

[0042] In application, task completion time, operation accuracy, procedure compliance, and erroneous steps are obtained or calculated from the operation logs in the digital control room. Operation accuracy is calculated as 1 - (number of erroneous steps) / (total number of operation steps). Procedure compliance is the proportion of all operation steps that conform to the procedures corresponding to the task objectives.

[0043] The workload data includes the target operator's score on the task load index scale for the key main tasks associated with the task target.

[0044] In the application, after the task objective is achieved and the corresponding task sequence is completed, the operator fills out a Task Load Index (TLX) scale for each key main task associated with the task objective. The TLX scale includes six core dimensions: mental demand, physical demand, time pressure, operational performance, self-efficacy, and frustration. Each dimension has a score range of 0-10, with 10 being the highest and 0 the lowest. A two-stage weighted scoring method is used to perform 15 pairwise comparisons across the six dimensions to determine the weight of each dimension. The total TLX score is calculated based on the scores and weights, as shown in the following formula: ; in, The task load metric for a critical task is scored, with a score range of 0-10. This represents the weight of the i-th dimension, which can be obtained through a pairwise comparison method: compare 15 pairs of the 6 dimensions, and the subject selects the dimension that has a greater impact on their perceived load in each pair. The number of times each dimension is selected is the weight. This represents the score for each dimension.

[0045] The contextual awareness data includes the contextual awareness rating scale scores of the target operator for the key primary tasks associated with the task objective.

[0046] In application, once the task objective is achieved and the corresponding task sequence is completed, the operator fills out a Situation Awareness Rating Technique (SART) scale for each key main task associated with the task objective.

[0047] The physiological data includes electrocardiogram (ECG) data, electroencephalogram (EEG) data, and eye movement data.

[0048] In the application, electrocardiogram (ECG) data is collected using an ECG acquisition device, electroencephalogram (EEG) data is collected using an EEG acquisition device, and eye movement data is collected using an eye tracker. ECG data includes instantaneous heart rate, heart rate variability, and the standard deviation of the RR interval; EEG data includes alpha waves, beta waves, theta waves, and gamma waves; and eye movement data includes blink frequency, blink count, total fixation time, maximum fixation time, and minimum fixation time.

[0049] like Figure 2 As shown, in one embodiment, step S2, generating an experimental sample set and training a base model based on the experimental sample set to obtain a target evaluation model, includes: Step S21: Based on the procedure of the task objectives in the experimental scenario, generate at least one task sequence, the task sequence including at least one key main task.

[0050] In applications, a task objective may have multiple implementation processes. Therefore, one or more task sequences are generated based on the procedures for a task objective. Each task sequence corresponds to a unique consequence: success or failure. Each task sequence includes at least one key master task, which is the task that must be executed for the success of that task sequence.

[0051] Step S22: Obtain the experimental dataset, which includes multiple experimental data. The experimental data includes multi-dimensional data of the operator executing the task sequence during the process of completing the task objective in the experimental scenario. The multi-dimensional data includes main task measurement data, workload data, situational cognition data, and physiological data.

[0052] In applications, experimental data includes multi-dimensional data acquired from the nuclear power plant's digital instrumentation and control system or full-range operator training simulators, showing operators executing task sequences during the process of achieving task objectives in experimental scenarios. This multi-dimensional data includes primary task measurement data, workload data, situational awareness data, and physiological data.

[0053] In this application, after collecting multi-dimensional data, high-pass filtering is used for data preprocessing to remove noise components from the physiological data. The filtering operation is performed using the open-source scientific computing library SciPy in a Python environment, with a cutoff frequency of 0.5Hz and a stopband cutoff frequency of 50Hz, outputting the noise-suppressed ECG signal. A time-aware interpolation method is used to align the timestamps of different dimensions of data, extracting the original timestamps for each dimension, using the time axis of the main task measurement data as a reference. For non-reference dimensions, a weighted average is calculated based on the proportion of the time difference between the target time point and the preceding and following original timestamps to obtain the interpolated data for the target point. Finally, the timestamp-aligned multi-dimensional data is output.

[0054] Step S23: Determine the completion status of each key main task in the task sequence and the completion status of the task sequence.

[0055] Step S24: Determine the experimental data corresponding to the completed task sequence as the experimental sample.

[0056] In practice, performance evaluation is only performed on operations corresponding to task sequences that are already completed; operations corresponding to task sequences that are not yet completed are deemed unreliable. Therefore, experimental data corresponding to task sequences that are already completed are used as experimental samples, while experimental data corresponding to task sequences that are not yet completed are not used as experimental samples.

[0057] Step S25: For the task sequence that is already in the completed state. The following formula is used to generate the corresponding total task score for the experimental samples:

[0058]

[0059] in, The total task score for personnel is the k-th task sequence that includes j key main tasks. The weights of task sequence k are... Let L be the score for task sequence k, and L be the total number of key main tasks in task sequence k. The task load metric score for the j-th critical primary task in task sequence k. The score for the j-th critical main task, when the j-th critical main task is in an incomplete state. The value is 0 when the j-th critical main task is in a completed state. =1, Let be the weight of the j-th key main task.

[0060] In application, the weights of key primary tasks in the task sequence under the target scenario are determined by combining the experience of expert instructors and operator procedure manuals. Task sequence weight Perform the assignment.

[0061] In applications, task sequences weight It includes five levels, for example, 1, 0.75, 0.5, 0.25, and 0, representing the task sequence. The degree of adaptability to the task objectives in the target scenario. =1 indicates that the sequence is very appropriate; =0.75 indicates that the sequence is appropriate; =0.5 indicates that the sequence is somewhat disordered; =0.25 indicates that the sequence is inappropriate; =0 indicates that the sequence is very inappropriate.

[0062] Step S26: Use the total score of the personnel task as the true value label of the experimental sample, and add the experimental sample to the experimental sample set.

[0063] Step S27: Train the basic model based on the experimental sample set to obtain the target evaluation model.

[0064] like Figure 3 As shown, in one embodiment, the key main task includes at least one key sub-task, and the key sub-task includes at least one key step; The method further includes: Step S41: If the completion time of all key sub-tasks in the key main task is within the time limit of the key sub-task, then the key main task is confirmed to be in a completed state.

[0065] Step S42: If the completion time of any key subtask in the key main task is not within the time limit of the key subtask, the key main task is confirmed to be incomplete.

[0066] Step S43: If all key main tasks in the task sequence are in a completed state, confirm that the task sequence is in a completed state.

[0067] Step S44: If any key main task in the task sequence is incomplete, confirm that the task sequence is incomplete.

[0068] In application, the completion status of a key primary task and the task sequence is determined based on the success criteria of the key primary task. If all key subtasks of a key primary task meet the task success criteria and their completion times are less than their respective time limits, the key primary task is considered complete. If all key primary tasks in the task sequence meet the task success criteria and are all in a completed state, and the total completion time of the key primary tasks (i.e., the completion time of the task sequence) is less than the time limit set for the task objective, the task objective is considered complete.

[0069] In one embodiment, step S27, training a base model based on the experimental sample set to obtain a target evaluation model, includes: Step S271: Configure an attention head for each dimension of the multi-dimensional data and construct a basic model using a multi-head attention mechanism.

[0070] Step S272: Divide the experimental sample set into a training set and perform iterative adversarial training on the base model based on the following formula and the training set:

[0071] in, For input features, For adversarial examples, For the amplitude of the disturbance, This is a sign function used to control the direction of the disturbance. loss function For input features The gradient; Where N is the number of samples in the training set. It is the true value of the i-th sample in the training set. It is the base model prediction value of the i-th sample in the training set.

[0072] Step S273: When the value of the loss function converges, the final base model is output as the target evaluation model.

[0073] In the application, a training set of sample data, including main task measurement data, workload data, situational cognition data and physiological data, is used as input. The total task score of the personnel corresponding to the sample is used as the true value label of the sample data. The personnel performance score is used as the predicted value of the model. The base model is trained so that the predicted value of the sample data predicted by the base model is as close as possible to the true value label.

[0074] In application, the calculation of personnel performance scores in this application is a continuous value regression problem, with personnel performance scores distributed in the range of 0-2 points.

[0075] like Figure 4 As shown, an attention head is configured for each dimension of the multi-dimensional data, and a multi-head attention mechanism is used to build the basic model. The attention head captures the interactions between features through an attention mechanism; the calculation formula for the attention mechanism is shown below: ; in, For the output of the i-th attention head, , , Let be the query matrix, key matrix, and value matrix of the i-th attention head, respectively. Key vector Dimensions (such as) It is n× The matrix (where n is the sequence length), the scaling factor is used to avoid If the dot product is too large, the softmax gradient will vanish.

[0076] Based on the outputs of multiple single attention heads, a multi-attention mechanism is used to capture the relationships between dimensions. The multi-head attention fusion calculation formula is as follows: ; in, The output of the i-th attention head (corresponding to the single attention head formula) That is, the feature representation computed independently by each attention head. This means concatenating the outputs of each attention head together. This represents the linear transformation matrix after fusion.

[0077] In one embodiment, the experimental sample set is divided into a training set, a test set, and a validation set in a certain ratio (e.g., 7:1.5:1.5). The number of iterations, the initial learning rate, and the batch size are set. Cosine annealing learning rate scheduling is used to accelerate model convergence. The formula for calculating cosine annealing is as follows: ; in, This represents the learning rate at step t of the training process. This represents the maximum value of the learning rate. The minimum learning rate is represented by t, where t is the current iteration number, and T is the length of a cosine period. In this embodiment, the iteration number t = 100, T = 20, and the initial learning rate is set to... , , .

[0078] In one embodiment, adversarial training is employed to enhance model robustness. The adversarial training method used in this application is the Fast Gradient Signed Method (FGSM), and the adversarial training formula is as follows:

[0079] in, For input features, For adversarial examples, For the amplitude of the disturbance, This is a sign function used to control the direction of the disturbance. loss function For input features The gradient.

[0080] The goal of model iterative optimization is to minimize the residual between the predicted and the true values. In this embodiment, the mean squared error (MSE) is used as the loss function, and its calculation formula is as follows: ; Where n is the number of samples, It is the true value of the i-th sample. It is the model prediction value for the i-th sample.

[0081] The model is considered to have converged and training is complete when the loss value of MSE is stable or decreases by less than 2% during iterative optimization. The final base model is then output as the target evaluation model.

[0082] In application, the mean absolute error (MAE), mean squared error (MSE), and coefficient of determination (R²) are used to evaluate the model training results. The specific calculation formulas are as follows: ; ; ; Where n is the number of samples, It is the true value of the i-th sample. This is the model prediction value for the i-th sample. MAE measures the average error of the model's predictions; MSE measures the average squared error of the model's predictions and is more sensitive to outliers; R² measures the model's ability to explain data variation.

[0083] like Figure 5 As shown, in one embodiment, before step S1, which involves acquiring multi-dimensional data on the process by which the target operator achieves the task objective in the target scenario, the method further includes: Step S01: Obtain multidimensional performance features to be identified ,in This represents the performance feature to be identified in the nth dimension; Step S02: Expand the multidimensional performance features to be identified into a polynomial-generated extended feature vector that includes higher-order terms and interaction terms. ; Step S03: Use the polynomial-generated extended feature vector as input to ridge regression to construct the following ridge regression model:

[0084] in, To predict personnel performance scores, This is a vector of regression coefficients; The loss function of the ridge regression model described above is: ; in, For actual personnel performance scores, Mean squared error is used to measure the difference between the predicted value and the actual value. This is an L2 regularization term used to suppress model complexity.

[0085] Step S04: Determine multiple key features from the multidimensional performance features to be identified based on the absolute value of the regression coefficients, and divide the key features into multidimensional data.

[0086] In application, the larger the absolute value of the regression coefficient, the stronger the correlation between the corresponding performance feature to be identified and the personnel performance score. Multiple important features are selected from the multidimensional performance features to be identified using "polynomial generation + ridge regression". Then, "measurability, stability, and sensitivity" are combined to determine multiple key features from the important features, and these key features are divided into multidimensional data.

[0087] Table 1 shows the set of digital control room operator performance indicators (performance-related multi-dimensional data and key features) determined from multi-dimensional performance characteristics to be identified in the embodiments of this application.

[0088]

[0089] like Figure 6 As shown, in one embodiment, after inputting the multi-dimensional data into the target evaluation model to obtain the personnel performance score, step S3 further includes: Step S51: Calculate the quartiles of the same batch of data, including the upper quartile, the median, and the lower quartile.

[0090] Step S52: Based on the relationship between the personnel performance score and the quartiles, determine the performance level corresponding to the personnel performance score. The performance levels are divided into Level 1, Level 2, Level 3, and Level 4 performance levels from high to low.

[0091] Step S53: If the operation accuracy rate is less than the preset accuracy rate, the performance level is downgraded by one level.

[0092] Step S54: If the task completion time exceeds the task completion time limit by a preset proportion, the performance level is downgraded by one level.

[0093] In the application, after obtaining the personnel performance scores, the lower quartile Q1, median Q2, and upper quartile Q3 corresponding to the task objective in the target scenario are obtained. Q1, Q2, and Q3 correspond to the lower limits of the top 25%, top 50%, and top 75% of the performance scores of the experimental operators (15-20 or more) who completed the task objective in the target scenario, respectively. If there is no performance score data for the experimental operators who completed the task objective in the target scenario, the lower limits of the top 25%, top 50%, and top 75% of the performance scores of operators who completed the task objective in the target scenario in this and historical periods are used. The performance level of the digital control room operators is divided into four intervals according to the relationship between personnel performance scores and quartiles: excellent, good, satisfactory, and poor, as shown in Table 2, the operator performance level matrix.

[0094] Table 2 Example of Operator Performance Level Matrix

[0095] In the application, the error rate ER of the operator in completing the task objective in the target scenario is extracted, where 1 - error rate is the operation accuracy rate, and TT is the task completion time. To avoid situations where "the error rate is low but the task times out, or the task is completed quickly but the error rate is high," this application introduces a "time efficiency completion index." ": .

[0096] Where TT is the actual value of the task completion time; This refers to the task completion time limit in the task guidelines. Based on the requirements of the target scenario and the recommendations of nuclear power plant operator training instructors, operators, and human factors safety engineers, the preset accuracy rate, task completion time limit, and preset percentage are determined. The preset accuracy rate can be 97%, 95%, or 90%, and the preset percentage can be 10% or 20%. Based on the operational accuracy rate and time efficiency completion indicators, it is determined whether to downgrade the performance level corresponding to the personnel's performance score.

[0097] In one implementation, if TT is greater than If TT is not greater than 1, the performance level will be downgraded to a worse level. Then, assess whether the operation accuracy rate is less than the preset accuracy rate, and whether the task completion time exceeds the task completion time limit by a preset proportion (i.e., the time efficiency completion index). (Is it greater than the preset ratio?) If TT is not greater than... However, if the operational accuracy rate is lower than the preset accuracy rate, the performance level corresponding to the employee's performance score will be downgraded by one level. If TT is not greater than... However, if the task completion time exceeds the task completion time limit by a preset proportion, the performance level corresponding to the personnel's performance score will be downgraded by one level. The lower limit of the downgraded performance level is the deterioration level. Through the above method, based on the model output, the model output is reasonably corrected according to the operator's observable data to ensure that the model output performance accurately reflects the operator's performance level.

[0098] In one embodiment, after inputting the multi-dimensional data into the target evaluation model to obtain the personnel performance score, step S3 further includes: Step S61: Calculate each feature based on the following marginal effect calculation formula. The marginal contribution is calculated and interpretability analysis results are generated: ; in, Represents the complete feature set of the model; express It does not contain any subset of feature i; This means that only a subset of samples x are retained. The values ​​of the corresponding features; This means that the model uses only the features of a subset S to correspond to the predicted values ​​of x; Representation of features Average contribution to performance forecast output.

[0099] like Figure 4 As shown in the embodiments of this application, the target evaluation model also includes SHAP interpretability analysis. After the target evaluation model outputs the personnel performance score, the model is called to calculate the SHAP value of each feature dimension. The higher the SHAP value, the higher the importance and contribution of that feature to the prediction result or the global prediction result. The interpretability analysis is compiled into a report and output to the user along with the personnel performance score. Specifically, the Deep SHAP interpreter is initialized in advance, including modality (dimension) level analysis and feature level analysis. In modality level analysis, other modalities are kept unchanged, and the mean of the corresponding modality data in the background data is used to replace it. The SHAP value of the corresponding modality is calculated, and this process is repeated to obtain the modality level SHAP score and ranking. In feature level analysis, the feature to be analyzed is replaced with the mean of the corresponding feature in the background data, keeping the other features unchanged. The SHAP score of the feature is calculated, and finally the SHAP score of each feature is summarized and ranked. The modality level analysis and feature level analysis results are output in the form of a graph and output to the user along with the personnel performance score.

[0100] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0101] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 7 As shown, the computer device 7 of this embodiment includes: at least one processor 70 ( Figure 7 (Only one is shown in the diagram) a processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, wherein the processor 70 executes the computer program 72 to implement the steps in any of the above-described embodiments of the operational performance evaluation methods.

[0102] The computer device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that... Figure 7 The computer device 7 is merely an example and does not constitute a limitation on the computer device 7. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0103] The processor 70 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0104] In some embodiments, the memory 71 may be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 71 may be an external storage device of the computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 7. Furthermore, the memory 71 may include both internal and external storage units of the computer device 7. The memory 71 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0105] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0108] like Figure 8 As shown, this application embodiment provides a computer program product 80, including a computer program 72. When the computer program 72 is run, the steps in the above-described operation performance evaluation method embodiments are executed.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / computer equipment, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0110] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] In the embodiments provided in this application, it should be understood that the disclosed computer devices and methods can be implemented in other ways. For example, the computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for evaluating operational performance, characterized in that, The method, applied to computer equipment in a digital control room, includes: Acquire multi-dimensional data on the process by which target operators achieve task objectives in a target scenario; Generate an experimental sample set and train a base model based on the experimental sample set to obtain the target evaluation model; The multi-dimensional data is input into the target evaluation model to obtain the personnel performance score, which represents the operational performance of the target operator in achieving the task objective in the target scenario.

2. The operational performance evaluation method as described in claim 1, characterized in that, The multi-dimensional data includes main task measurement data, workload data, situational cognition data, and physiological data; The main task measurement data includes task completion time, operation accuracy, procedure compliance, and error steps; The workload data includes the target operator's task load index scale score for the key main tasks associated with the task target; The contextual awareness data includes the contextual awareness scoring scale scores of the target operator for the key main tasks associated with the task target; The physiological data includes electrocardiogram (ECG) data, electroencephalogram (EEG) data, and eye movement data.

3. The operational performance evaluation method as described in claim 1, characterized in that, The process of generating an experimental sample set and training a base model based on the experimental sample set to obtain the target evaluation model includes: Based on the procedures for the task objectives in the experimental scenario, at least one task sequence is generated, and the task sequence includes at least one key main task. Obtain an experimental dataset, which includes multiple experimental data points. The experimental data includes multi-dimensional data on the execution of task sequences by operators in the experimental scenario to complete task objectives. The multi-dimensional data includes main task measurement data, workload data, situational cognition data, and physiological data. Determine the completion status of each key main task in the task sequence and the completion status of the task sequence itself; The experimental data corresponding to the completed task sequence are identified as experimental samples. For the sequence of tasks in the completed state The following formula is used to generate the corresponding total task score for the experimental samples: in, The total task score for personnel is the k-th task sequence that includes j key main tasks. The weights of task sequence k are... Let L be the score for task sequence k, and L be the total number of key main tasks in task sequence k. The task load metric score for the j-th critical primary task in task sequence k. The score for the j-th critical main task, when the j-th critical main task is in an incomplete state. The value is 0 when the j-th critical main task is in a completed state. =1, Let be the weight of the j-th key main task; The total task score of the personnel is used as the true value label of the experimental sample, and the experimental sample is added to the experimental sample set; The target evaluation model is obtained by training the basic model based on the experimental sample set.

4. The operational performance evaluation method as described in claim 3, characterized in that, The key main task includes at least one key sub-task, and the key sub-task includes at least one key step. The method further includes: If the completion time of all key subtasks in the key main task is within the time limit of the key subtask, the key main task is confirmed to be in a completed state. If the completion time of any key subtask in the key main task is not within the time limit of the key subtask, the key main task is confirmed to be incomplete. If all the key main tasks in the task sequence are in a completed state, the task sequence is confirmed to be in a completed state. If any key main task in the task sequence is incomplete, the task sequence is confirmed to be incomplete.

5. The operational performance evaluation method as described in claim 3, characterized in that, The process of training the base model based on the experimental sample set to obtain the target evaluation model includes: Configure an attention head for each dimension of the multi-dimensional data and construct the basic model using a multi-head attention mechanism; A training set is partitioned from the experimental sample set, and the base model is iteratively trained adversarially based on the following formula and the training set: in, As input features, For adversarial examples, For the amplitude of the disturbance, This is a sign function used to control the direction of the disturbance. loss function For input features The gradient; Where N is the number of samples in the training set. It is the true value of the i-th sample in the training set. It is the base model prediction value of the i-th sample in the training set; When the value of the loss function converges, the final base model is output as the target evaluation model.

6. The operational performance evaluation method as described in claim 1, characterized in that, Before acquiring multi-dimensional data on the process of the target operator achieving the task objective in the target scenario, the process also includes: Obtain multidimensional performance features to be identified ,in This represents the performance feature to be identified in the nth dimension; The multidimensional performance features to be identified are extended into a polynomial-generated extended feature vector that includes higher-order terms and interaction terms. ; The polynomial-generated extended feature vectors are used as inputs to construct the following ridge regression model: in, To predict personnel performance scores, This is a vector of regression coefficients; The loss function of the ridge regression model described above is: ; in, For actual personnel performance scores, Mean squared error is used to measure the difference between the predicted value and the actual value. This is an L2 regularization term used to suppress model complexity; Multiple key features are determined from the multidimensional performance features to be identified based on the absolute values ​​of the regression coefficients, and the key features are divided into multidimensional data.

7. The operational performance evaluation method as described in claim 2, characterized in that, After inputting the multi-dimensional data into the target evaluation model to obtain the personnel performance score, the method further includes: Calculate the quartiles of the same batch of data, where the quartiles include the upper quartile, the median, and the lower quartile. Based on the relationship between the personnel performance score and the quartiles, the performance level corresponding to the personnel performance score is determined, and the performance levels from high to low are Level 1, Level 2, Level 3, and Level 4. If the operation accuracy rate is lower than the preset accuracy rate, the performance level will be downgraded by one level. If the task completion time exceeds the task completion time limit by a preset proportion, the performance level will be downgraded by one level.

8. The operational performance evaluation method as described in any one of claims 1 to 7, characterized in that, After inputting the multi-dimensional data into the target evaluation model to obtain the personnel performance score, the method further includes: Each feature is calculated based on the following marginal utility calculation formula. The marginal contribution is calculated and interpretability analysis results are generated: ； in, Represents the complete feature set of the model; express It does not contain any subset of feature i; This means that only a subset of samples x are retained. The values ​​of the corresponding features; This means that the model uses only the features of a subset S to correspond to the predicted values ​​of x; Representation of features Average contribution to performance forecast output.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method described in any one of claims 1 to 8 to be performed.

Images