Practical training efficiency evaluation method based on multi-agent collaborative analysis and related equipment

Abstract

The application discloses a practical training efficiency evaluation method based on multi-agent collaborative analysis and related equipment, and relates to the technical field of data processing. The method comprises the following steps: obtaining various event streams in a practical training process based on a pre-implanted probe; using a preset multi-agent collaborative analysis strategy to process the various event streams respectively by using a plurality of preset agents to obtain a result score, a process score and a specification score; and processing the result score, the process score and the specification score by using a preset practical training efficiency evaluation model to generate a practical training efficiency evaluation result. The application collects process event streams by using a probe, generates a result score, a process score and a specification score respectively by using a plurality of agents, and fuses the evaluation. The debugging process is included in the evaluation index, the evaluation is more objective and comprehensive, and the comprehensive practical ability of a practical training person can be truly reflected.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a training effectiveness evaluation method and related equipment based on multi-agent collaborative analysis. Background Technology

[0002] With the rapid development of information technology and the continuous upgrading of industrial structure, complex skill-based positions, such as software engineering, embedded development, and artificial intelligence, are placing unprecedented demands on the practical abilities and professional qualities of talents. To bridge the gap between school education and the needs of enterprises, various vocational schools and online training platforms are widely adopting practical training based on real projects or simulation environments to improve students' hands-on skills and engineering practice levels.

[0003] However, most existing practical training evaluation technologies focus on judging the final results of practical training, while neglecting the detailed and quantitative analysis of students' practical process, resulting in significant deficiencies in the objectivity, comprehensiveness and predictive effectiveness of the evaluation results. Summary of the Invention

[0004] The main purpose of this application is to provide a training effectiveness evaluation method and related equipment based on multi-agent collaborative analysis, aiming to solve the technical problem of how to improve the accuracy of training effectiveness evaluation.

[0005] To achieve the above objectives, this application proposes a training effectiveness evaluation method based on multi-agent collaborative analysis. The training effectiveness evaluation method based on multi-agent collaborative analysis includes: Based on pre-embedded probes, the event flow of various events during the training process is acquired; Based on a preset multi-agent collaborative analysis strategy, multiple preset agents are used to process the event streams respectively to obtain result score, process score and standard score. The process score is generated based on debugging entropy and convergence index. Debugging entropy represents the degree of confusion of students in the code debugging process, and convergence index represents whether students tend to solve the problem in the code debugging process. The result score, process score, and standard score are processed using a preset training effectiveness evaluation model to generate training effectiveness evaluation results.

[0006] In one embodiment, the preset plurality of intelligent agents includes a result analysis intelligent agent, a process analysis intelligent agent, and a specification check intelligent agent. The step of using the preset plurality of intelligent agents to process the event streams based on a preset multi-agent collaborative analysis strategy to obtain result scores, process scores, and specification scores further includes: The result analysis agent is used to analyze the event streams to perform functional testing and static checks, and generate result scores. The process analysis agent is used to analyze the event streams to determine the debugging entropy and convergence index, and to generate process scores. The specification checking agent is used to analyze the event streams to check code comment rate, naming conventions, and structural clarity, and to generate specification scores.

[0007] In one embodiment, the step of using a process analysis agent to analyze the event streams to determine the debugging entropy and convergence index, and generating process scores, further includes: Based on the aforementioned event streams, a sequence of behaviors is generated; Based on the behavior sequence, determine the actual correction vector; Based on the actual correction vector, the debugging entropy is determined, and based on the preset ideal correction vector and the actual correction vector, the convergence index is determined. Based on the convergence index and the debugging entropy, a process score is generated.

[0008] In one embodiment, the step of determining the actual correction vector based on the behavior sequence further includes: Perform an error event extraction operation on the behavior sequence to construct an error event sequence; Based on any two adjacent error events in the error event sequence, the location offset, error type change, and severity change are determined. Based on the position offset, the error type change, and the severity change, an actual correction vector is constructed.

[0009] In one embodiment, before the step of determining the debugging entropy based on the actual correction vector, the method further includes: The K-means clustering algorithm is used to divide the modified vector space into K states, where the value of K is predetermined. Based on the K states, a clustering operation is performed on the sample correction vector to obtain K cluster centers, where each cluster center represents a debugging state.

[0010] In one embodiment, the step of determining the debugging entropy based on the actual correction vector further includes: Multiple actual correction vectors are combined in chronological order to construct a debugging trajectory; Each actual correction vector in the debugging trajectory is mapped to the nearest cluster center among the K cluster centers to obtain the mapping result; Based on the mapping result, the Shannon entropy of the state distribution is calculated, and the directional entropy is calculated based on the directional category of the pre-defined correction vector. Based on the Shannon entropy and the directional entropy, the debugging entropy is determined.

[0011] In one embodiment, the step of acquiring various event flows during the training process based on pre-implanted probes further includes: Based on students' historical training data, use the Project Response Theory model or the Deep Knowledge Tracking model to estimate students' current abilities. Based on the student's current ability, the perturbation injection agent is used to determine the information gain of each perturbation in the preset perturbation library, and the perturbation with the largest information gain is set as the target perturbation. The target disturbance is injected into the training environment to generate a fault event; Based on pre-embedded probes, the event streams for handling the fault events during the training process are obtained.

[0012] In one embodiment, the step of processing the result score, the process score, and the standardization score using a preset training effectiveness evaluation model to generate training effectiveness evaluation results further includes: Based on the aforementioned event streams, the robust tensor within the disturbance response segment is determined using a robust tensor quantization model to obtain a robust comprehensive score. The robustness score, result score, process score, and standardization score are processed using a preset training effectiveness evaluation model to generate training effectiveness evaluation results.

[0013] Furthermore, to achieve the above objectives, this application also proposes a training effectiveness evaluation device based on multi-agent collaborative analysis, wherein the training effectiveness evaluation device based on multi-agent collaborative analysis includes: The acquisition module is used to acquire various event streams during the training process based on pre-embedded probes; The data processing module is used to process the event streams using multiple preset agents based on a preset multi-agent collaborative analysis strategy, and obtain result score, process score and standard score. The process score is generated based on debugging entropy and convergence index. Debugging entropy represents the degree of confusion of students in the code debugging process, and convergence index represents whether students tend to solve the problem in the code debugging process. The generation module is used to process the result score, the process score and the standard score using a preset training effectiveness evaluation model to generate training effectiveness evaluation results.

[0014] Furthermore, to achieve the above objectives, this application also proposes a training effectiveness evaluation device based on multi-agent collaborative analysis. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the training effectiveness evaluation method based on multi-agent collaborative analysis as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the training effectiveness evaluation method based on multi-agent collaborative analysis as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the training effectiveness evaluation method based on multi-agent collaborative analysis as described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: This application discloses a training effectiveness evaluation method and related equipment based on multi-agent collaborative analysis, relating to the field of data processing technology. Compared to existing training evaluation technologies that mostly focus on judging the final training results while neglecting the refined and quantitative analysis of students' practical processes, resulting in significant deficiencies in objectivity, comprehensiveness, and predictive effectiveness, this application first acquires various event flows during the training process based on pre-embedded probes. Then, based on a preset multi-agent collaborative analysis strategy, multiple preset agents process each event flow to obtain result scores, process scores, and standardization scores. The process score is generated based on debugging entropy and convergence index. Debugging entropy characterizes the degree of confusion during code debugging, and convergence index characterizes whether students tend to solve problems during code debugging. Finally, a preset training effectiveness evaluation model is used to process the result scores, process scores, and standardization scores to generate training effectiveness evaluation results.

[0018] Understandably, this application collects process event flows through probes, generates result scores, process scores, and specification scores separately through multiple agents, and integrates them for evaluation. It incorporates the debugging process into the evaluation indicators, making the assessment more objective and comprehensive, and truly reflecting the trainee's comprehensive practical ability. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an embodiment of the training effectiveness evaluation method based on multi-agent collaborative analysis in this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the training effectiveness evaluation method based on multi-agent collaborative analysis in this application. Figure 3 This is a schematic diagram of the module structure of the training effectiveness evaluation device based on multi-agent collaborative analysis according to an embodiment of this application; Figure 4 This is a schematic diagram of the hardware operating environment involved in the training effectiveness evaluation method based on multi-agent collaborative analysis in the embodiments of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution in this application's embodiments is: In this embodiment, for ease of description, the following description uses a training effectiveness evaluation device based on multi-agent collaborative analysis as the execution subject.

[0026] Due to the rapid development of information technology and the continuous upgrading of industrial structure, complex skill-based positions, such as software engineering, embedded development, and artificial intelligence, place unprecedented demands on the practical abilities and professional qualities of talents. To bridge the gap between school education and the needs of enterprises, various vocational schools and online training platforms widely adopt practical training based on real projects or simulation environments to improve students' hands-on skills and engineering practice levels.

[0027] However, most existing practical training evaluation technologies focus on judging the final results of practical training, while neglecting the detailed and quantitative analysis of students' practical process, resulting in significant deficiencies in the objectivity, comprehensiveness and predictive effectiveness of the evaluation results.

[0028] This application provides a solution in which: first, based on pre-embedded probes, various event flows during the training process are acquired; then, based on a preset multi-agent collaborative analysis strategy, multiple preset agents are used to process each event flow to obtain result scores, process scores, and standardization scores. The process scores are generated based on debugging entropy and convergence index. The debugging entropy characterizes the degree of confusion of students during code debugging, and the convergence index characterizes whether students tend to solve problems during code debugging; finally, a preset training effectiveness evaluation model is used to process the result scores, process scores, and standardization scores to generate training effectiveness evaluation results.

[0029] Understandably, this application collects process event flows through probes, generates result scores, process scores, and specification scores separately through multiple agents, and integrates them for evaluation. It incorporates the debugging process into the evaluation indicators, making the assessment more objective and comprehensive, and truly reflecting the trainee's comprehensive practical ability.

[0030] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, or a training effectiveness evaluation device based on multi-agent collaborative analysis, etc. The following description uses a training effectiveness evaluation device based on multi-agent collaborative analysis as an example to illustrate this embodiment and the subsequent embodiments.

[0031] Based on this, the embodiments of this application provide a training effectiveness evaluation method based on multi-agent collaborative analysis, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the training effectiveness evaluation method based on multi-agent collaborative analysis of this application.

[0032] In this embodiment, the training effectiveness evaluation method based on multi-agent collaborative analysis includes steps S10 to S30: Step S10: Based on the pre-implanted probes, acquire the event streams during the training process; It should be noted that a probe refers to a lightweight data acquisition module pre-integrated into an online training platform or code editor, used to capture user actions.

[0033] It should be noted that each event stream refers to an operation sequence record with millisecond-level timestamps. In this application, a "probe" is embedded in the front end of the IDE or training platform to record the following event streams with millisecond-level timestamps: Tcode: Code edits (number of lines diffed, location of changes). Tcompile: Compilation action and results (success / failure, error code, error line number, error type). Trun: Execution results and output. Tdoc: Documentation / help viewing behavior.

[0034] In this embodiment, at the start of the training, the probe automatically begins listening, recording the trainee's keyboard input, code modification differences, compilation trigger actions, compilation result feedback, program output, and browsing of the built-in help documentation throughout the process. These actions are encapsulated into an ordered data stream containing timestamps, action types, and status information.

[0035] Understandably, the purpose of this step is to build the data foundation for subsequent analysis.

[0036] Specifically, the step of acquiring various event flows during the training process based on pre-implanted probes further includes steps S11 to S14: Step S11: Based on students' historical training data, estimate students' current abilities using Project Response Theory (PRT) or Deep Knowledge Tracking (DTT) models. It should be noted that the Item Response Theory (IRT) model is a measurement model based on latent trait theory. It maps students' abilities to continuous numerical values ​​(such as ability value θ) through students' historical answer or operation data (accuracy rate, time taken, etc.). It is often used in adaptive assessment to dynamically estimate the level of test subjects.

[0037] Deep Knowledge Tracking (DKT) model is a time-series prediction model based on recurrent neural networks (RNN). It uses students' historical practice sequences (question-answer pairs) to predict their mastery probability of knowledge points and can output a capability state vector that changes over time.

[0038] Students' current abilities ( ) refers to the quantitative estimate of a student's comprehensive programming / debugging ability at time t. It can be a scalar (IRT) or a high-dimensional vector (DKT) representing their prior knowledge level.

[0039] In this embodiment, the system extracts the student's past training task completion records (including compilation pass rate, bug fixing time, test case pass rate, etc.) from the database, inputs them into a pre-trained IRT or DKT model, and the model performs forward inference calculations to output the ability estimate at the current time t. This serves as the basis for subsequent perturbation selection decisions.

[0040] Understandably, this step provides a personalized prior knowledge benchmark for perturbation injection, enabling subsequent perturbation selection to be "personalized," avoiding injecting perturbations that are too difficult or too easy for all students, thereby achieving adaptive assessment.

[0041] Step S12: Based on the student's current ability, use the perturbation injection agent to determine the information gain of each perturbation in the preset perturbation library, and set the perturbation with the largest information gain as the target perturbation; It should be noted that the Perturbation Injection Agent (PIA) is an automated module in the system responsible for strategically selecting and executing fault injection. It has the ability to access the perturbation library, call the capability estimation interface, calculate information gain, and trigger injection actions.

[0042] The perturbation library refers to a pre-built collection of faults, where each perturbation contains a five-tuple (type, level, timing, scope, reversibility) covering types such as missing dependencies, data pollution, interface exceptions, implicit bugs, and environmental jitter.

[0043] Information gain refers to the expected reduction in the uncertainty of the system's assessment of a student's ability after injecting a perturbation d_i, given the student's current ability θ_t. It is denoted as I(d_i|θ_t). The larger the gain, the more effectively the perturbation can "detect" the student's unknown ability boundaries.

[0044] In this embodiment, the perturbation injection agent traverses all candidate perturbations d_i in the perturbation library. For each perturbation, it calculates its conditional mutual information I(d_i|θ_t) by combining the current capability θ_t and the historical empirical conditional probability distribution. Then, it compares all candidate values ​​and selects the perturbation d with the largest information gain. =arg max_i I(d_i|θ_t), mark it as the target perturbation and prepare it for injection.

[0045] Understandably, this step ensures that each perturbation injection is the "optimal evaluation action," enabling the limited number of training perturbations to obtain the maximum diagnostic information and avoiding the loss of evaluation efficiency caused by blind or fixed-sequence injections.

[0046] Step S13: Inject the target disturbance into the training environment to generate a fault event; It should be noted that the training environment refers to the actual operating space where students write, compile, run, and debug code, which usually includes layers such as IDE, compiler, container / virtual machine, file system, and network proxy.

[0047] Injection refers to the technical action of actively implanting a pre-set fault into a specific level of the training environment during its operation, such as modifying the content of a target file, tampering with environment variables, or intercepting and rewriting API response data.

[0048] A failure event refers to a specific abnormal state or error that is triggered in the training environment after the injection action takes effect (such as compilation error, runtime crash, or dependency missing message). This event will serve as a real-world problem that students will need to face and solve next.

[0049] In this embodiment, the system performs specific injection operations based on the injection level and triggering method specified by the target perturbation. For example, if the perturbation type is "dependency missing," then specific dependency library files in the project are temporarily removed or renamed; if it is "interface exception," then the specified API is intercepted through a network proxy and an incorrect response is returned. After the injection is completed, the training environment presents an abnormal state for students to perceive and handle.

[0050] Understandably, this step upgrades the originally static and undisturbed practical training task into a dynamic test scenario with fault pressure, requiring students to mobilize their troubleshooting abilities to actively cope with the situation, thereby creating real triggering conditions for subsequent evaluation of its engineering robustness.

[0051] Step S14: Based on the pre-implanted probes, acquire the event streams of each event that handles the fault events during the training process.

[0052] In this embodiment, after the target disturbance is successfully injected (i.e., the fault event has been generated), the system starts or continues to run the probe pre-embedded in the front end of the IDE and training platform. The probe records data of the entire process from the student's perception of the fault to the attempt to solve it with millisecond precision, including but not limited to: code editing differences (modification of line numbers and content), compilation trigger actions and error message feedback, program execution output, document viewing behavior, etc., and encapsulates these timestamped records into a structured event stream sequence.

[0053] Understandably, this step precisely captures complete behavioral data within the "disturbance response segment," providing a time-aligned input source for subsequent calculations of robustness tensors (such as detection delay and positioning accuracy) and debugging entropy, ensuring that process evaluation can accurately pinpoint the student's reaction speed and operational effectiveness in dealing with specific faults.

[0054] Step S20: Based on a preset multi-agent collaborative analysis strategy, multiple preset agents are used to process the event streams respectively to obtain result score, process score and standard score. The process score is generated based on debugging entropy and convergence index. Debugging entropy represents the degree of confusion of students in the code debugging process, and convergence index represents whether students tend to solve the problem in the code debugging process. It should be noted that the multi-agent collaborative analysis strategy refers to a collaborative working mechanism that enables multiple virtual analysis units with different professional evaluation logics to work in parallel and integrate their output results.

[0055] Process score is a numerical indicator that specifically measures the trainee's logical thinking and debugging efficiency during the code writing process.

[0056] Debugging entropy is a quantitative value calculated by measuring the degree of disorder in the distribution and direction of code correction vectors. It is used to measure the divergence and randomness of debugging behavior.

[0057] The convergence index is a metric derived by calculating the similarity between the actual corrected path and the ideal convergence direction, used to determine whether the behavior tends to solve the problem.

[0058] In this embodiment, the system executes three analysis tasks in parallel based on a collaborative analysis strategy. The first agent receives the final submitted code entity, performs functional testing and static structure scanning, and outputs a score. The second agent parses the event stream in step S10, extracts the changes in error attributes before and after each modification to construct a correction vector sequence; then, it calculates the debugging entropy and convergence index in the sequence, and fits them to a process score based on a preset nonlinear mapping function. The lower the debugging entropy and the closer the convergence index is to the positive extremum, the higher the process score. The third agent performs pattern matching on the code text according to a preset coding standard rule base and outputs a standardization score.

[0059] Understandably, the purpose of this step is to achieve multi-dimensional quantitative evaluation. By introducing a process sub-mechanism, it effectively distinguishes between "solving the problem correctly on the first try" and "successfully succeeding after a large number of blind attempts," thus solving the problems of distorted evaluation results and inability to identify false mastery in related technologies, and enabling the quantification of implicit professional qualities such as error-correction ability.

[0060] Specifically, the preset multiple intelligent agents include a result analysis intelligent agent, a process analysis intelligent agent, and a specification check intelligent agent. The step of using the preset multiple intelligent agents to process the event streams based on the preset multi-agent collaborative analysis strategy to obtain the result score, process score, and specification score further includes steps S21 to S23: Step S21: Use the result analysis agent to analyze the event streams of each item to perform functional testing and static checks, and generate result scores; It should be noted that functional testing refers to a dynamic verification process that involves injecting a pre-designed set of input data into the code submitted by the trainee and comparing the actual output with the expected output to verify the correctness of the code logic.

[0061] Static analysis refers to the process of detecting unused variables, potential logical branch defects, or resource leakage risks by analyzing the syntax structure, control flow, and data flow of code without actually running it.

[0062] In this embodiment, the result analysis agent extracts a complete snapshot of the trainee's final submitted code from the event streams obtained in step S10. First, the agent initiates the functional testing process, sequentially loading multiple preset test cases, passing the input parameters of each test case to the code execution environment, capturing the actual output value of the code, and precisely comparing it with the expected output value corresponding to that test case. A functional test score is generated based on the proportion of test cases that pass. Subsequently, the agent initiates the static analysis process, traversing the code's abstract syntax tree, performing pattern matching based on a predefined set of code quality rules, identifying syntactic irregularities, potential defects, and modules exceeding complexity limits in the code, and deducting corresponding points based on the number and severity of the issues. Finally, the functional test score and the static analysis score are weighted and calculated to generate the final score.

[0063] Understandably, the purpose of this step is to objectively evaluate the final output of the training. By combining dynamic functional verification with static quality scanning, the result score can not only reflect whether the code can run correctly, but also characterize the robustness and maintainability of its internal structure. Compared with existing solutions that rely solely on the pass rate of test cases, the evaluation conclusion is more comprehensive and rigorous.

[0064] Step S22: Use the process analysis agent to analyze the event streams to determine the debugging entropy and convergence index, and generate process scores; In this embodiment, the process analysis agent takes the event stream generated in step S10 as input. First, it performs error event filtering and extraction, identifying all time points of compilation failure or runtime anomalies, and constructs an error event sequence based on timestamps. For each pair of adjacent error events in the error event sequence, the agent calculates the line number offset ratio at the code modification location, the degree of difference in error type encoding, and the change in error severity level, encapsulating these three values ​​into an actual correction vector. All actual correction vectors are arranged in their generation sequence to form a complete debugging trajectory. Subsequently, the agent performs distance matching between each actual correction vector in the debugging trajectory and multiple pre-trained cluster centers, assigning it to the most similar debugging state category to obtain a state sequence. Simultaneously, it determines whether the change direction of each actual correction vector belongs to the category of tending towards resolution, deviating from resolution, or oscillation in place, obtaining a direction sequence. The agent calculates the information entropy of the probability of each state in the state sequence and the information entropy of the probability of each direction category in the direction sequence, weights and sums them, and combines them with a compensation term calculated based on the convergence index to obtain the debugging entropy. Simultaneously, the agent calculates the cosine of the angle between each actual correction vector and the preset ideal convergence direction vector in the debugging trajectory, and takes the arithmetic mean of all cosine values ​​to obtain the convergence index. Finally, the agent maps the debugging entropy value and the convergence index to process scores.

[0065] Understandably, the purpose of this step is to quantitatively model and score the efficiency and logical thinking of the training process. For the first time, it transforms the trainees' error-correction behavior patterns from qualitative observation into quantitatively measurable debugging entropy and convergence index indicators. This allows the system to accurately distinguish between two fundamentally different behavioral processes: "systematic reasoning error correction" and "random blind trial and error." This solves the problem of distorted evaluations due to neglecting process data in related technologies, resulting in high scores but low performance.

[0066] Step S23: Use the specification checking agent to analyze the event streams of each item to check the code comment rate, naming conventions and structural clarity, and generate specification scores.

[0067] It should be noted that the code comment rate refers to the proportion of the number of lines or characters of comment text in the source code to the total number of lines or characters of code. It is a basic indicator used to measure the self-interpretability and maintainability of code.

[0068] Naming conventions refer to whether the naming format of identifiers conforms to the pre-defined camelCase, underscore, or specific prefix and suffix rules.

[0069] Structural clarity refers to whether the organization of code logic blocks is reasonable, including the length control of function bodies, the depth limit of nested loops, and the simplicity of inter-module dependencies.

[0070] In this embodiment, the code text finally submitted in the event stream of step S10 is used as the analysis object by the code specification checking agent. The agent first performs a lexical scan on the code text, separating valid code lines from comment lines, and calculates the code comment rate through counting and ratio operations. Then, the agent extracts all variable names, function names, class names, and other identifiers from the code, matching them one by one against a preset naming style regular expression rule set, and counting the number of identifiers that violate naming conventions. Simultaneously, the agent parses the abstract syntax tree structure of the code, calculates the number of statements in each function body, the nesting depth of each loop structure, and the complexity of import dependencies between source files, comprehensively evaluating the structural clarity. Finally, the comment rate compliance, naming convention compliance, and structural clarity scores are aggregated and calculated according to preset weights to generate a specification score.

[0071] Understandably, the purpose of this step is to independently assess the engineering and maintainability level of the code, making the implicit requirements of enterprises for code quality regarding teamwork and long-term maintenance explicit into quantifiable standard indicators. This guides trainees to develop coding habits that conform to industry standards while focusing on functional implementation, making the training evaluation system closer to the actual requirements of real positions for the engineering literacy of practitioners.

[0072] Step S30: Use a preset training effectiveness evaluation model to process the result score, the process score, and the standard score to generate training effectiveness evaluation results.

[0073] Specifically, the step of processing the result score, the process score, and the standardization score using a preset training effectiveness evaluation model to generate training effectiveness evaluation results further includes steps S31-S32: Step S31: Based on the event streams, use the robust tensor quantization model to determine the robust tensor within the disturbance response segment to obtain the robustness comprehensive score. Step S32: Use a preset training effectiveness evaluation model to process the robustness comprehensive score, the result score, the process score, and the standardization score to generate training effectiveness evaluation results.

[0074] It should be noted that the robust tensor quantization model is a computational model specifically designed to measure students' resilience and recovery capabilities when faced with injected disturbances (faults). This model takes the time-series event flow within the disturbance response segment as input, outputs a four-dimensional tensor, and weights and aggregates each dimension into a single score.

[0075] The disturbance response period refers to the period from the moment the target disturbance is successfully injected into the training environment (t_injection) until one of the following three states is achieved: (1) the student restores the task to normal operation (successful response); (2) the preset recovery time window T_max is exceeded (response timeout); (3) the system automatically rolls back the environment (response failure). This time period is the evaluation window corresponding to this disturbance.

[0076] In this application, the robustness tensor is denoted as a quadruple R = (τ_detect, ρ_locate, η_recover, κ_overcorrect), with the following four components: τ_detect (detection delay): The time difference between the moment the perturbation is injected and the moment the student first produces an observable response (such as compiling, running, or modifying code); ρ_locate (Location Precision): The reciprocal of the distance between the student's initial code modification location and the actual perturbation injection location, based on the Abstract Syntax Tree (AST) structure. The larger the value, the more accurate the location. η_recover (Recovery Success Rate): Within a limited time window T_max, the proportion of disturbances that allow the student to restore the task to normal operation out of the total number of injected disturbances; κ_overcorrect (overcorrection coefficient): During the recovery process, the ratio of the amount of modification to the perturbation-independent code region to the amount of modification to the perturbation-dependent code region on the Program Dependency Graph (PDG) reflects whether there is "blindly modifying" behavior.

[0077] The robustness composite score refers to a score ranging from 0 to 100 obtained by linearly weighting and aggregating the above four-dimensional tensors according to preset weights. The calculation formula is as follows: =100·(0.20· +0.30·ρ_locate+0.35·η_recover+0.15·(1 κ_overcorrect)), where This is the normalized value of τ_detect.

[0078] It should be noted that the preset training effectiveness evaluation model refers to a weighted fusion algorithm or machine learning evaluation network with pre-set weight allocation parameters.

[0079] In this embodiment, the system first segments the event stream obtained in step S10 according to the timestamp of the perturbation injection, identifying the perturbation response segment corresponding to the current perturbation. Then, it extracts key events within this time period: the first reaction time, the location of the first modified code, the number of successful recovery attempts, and the range of code regions involved in each modification. Next, using AST difference comparison and program dependency graph traversal algorithms, it calculates four components: τ_detect, ρ_locate, η_recover, and κ_overcorrect. Finally, it substitutes each component into the aforementioned weighted formula to obtain the student's comprehensive robustness score against the current perturbation.

[0080] Understandably, this step quantifies students' reaction speed, problem location accuracy, recovery effectiveness, and behavioral purposefulness in the dynamic fault handling process into comparable scores, filling the quantitative gap of the implicit quality of "adaptability and troubleshooting ability" in traditional evaluation, and providing an independent engineering robustness dimension for the final performance evaluation.

[0081] Furthermore, the system inputs the robustness composite score calculated in step S31, along with the result score, process score, and specification score generated by the multiple agents, into a preset weighted fusion model. This model uses five pre-configured weights (w_1~w_5, with the robustness score having the highest weight, defaulting to w_1~w_5). We perform a weighted summation of (=0.30) to obtain the total effectiveness index E, which is the final evaluation result of the training effectiveness.

[0082] It should be noted that in this application, the training effectiveness evaluation result refers to the effectiveness index E:

[0083]

[0084] in: Normalize the entropy value to [0, 1], with default weights: , Indicates debugging entropy The theoretical / statistical lower limit is a pre-set benchmark value or the minimum debugging entropy obtained statistically based on historical training samples (or training data from the same batch), used to characterize the entropy level when the trainee's debugging process is most organized and closest to the ideal trajectory; Indicates debugging entropy The theoretical / statistical upper limit is a pre-set saturation value or the maximum debugging entropy obtained statistically based on historical training samples (or training data from the same batch), used to characterize the entropy level when the debugging process is most chaotic and the actions are least directional. Will Linear mapping to the interval [0, 1]; when ≤ or ≥ When the boundary value is reached, it is truncated to 0 or 1.

[0085] In addition, it should be noted that during the students' practical training, teachers or staff can monitor the training situation in real time based on the sliding window. By monitoring the debugging entropy value in the sliding window in real time, abnormal states can be detected, and if an abnormal state occurs, human intervention can be carried out.

[0086] Furthermore, after obtaining the training effectiveness evaluation results, the job matching degree can be calculated based on the preset job weight template and the differentiated robustness requirements of different jobs:

[0087] Where j is the job index, representing the j-th target job (e.g., front-end development, back-end development, testing, algorithm, operation and maintenance, etc.) registered in the preset job weight template; i is the capability index, representing the i-th capability indicator participating in the job matching calculation. The capability indicators are derived from the aforementioned robustness comprehensive score, result score, process score (debugging entropy, convergence index), standardization score (annotation rate, naming standardization, structural clarity) and their sub-items, and are uniformly normalized to the same dimension. This represents the trainee's score on the i-th competency indicator; This represents the weight coefficient assigned to the i-th competency indicator by the j-th position in the preset position weight template, satisfying... =1, 0≤ ≤1, pre-configured by domain experts or based on the competency map of the target position, to depict the differences in the emphasis of various competency requirements for different positions. This is the matching score between the trainee and the j-th job position.

[0088] After the job matching degree is calculated, a radar chart is generated based on the calculation results. The radar chart shows the matching status of students in various job dimensions, which helps to confirm the students' subsequent training direction and employment recommendation direction.

[0089] This application discloses a training effectiveness evaluation method and related equipment based on multi-agent collaborative analysis, relating to the field of data processing technology. Compared to existing training evaluation technologies that mostly focus on judging the final training results while neglecting the refined and quantitative analysis of students' practical processes, resulting in significant deficiencies in objectivity, comprehensiveness, and predictive effectiveness, this application first acquires various event flows during the training process based on pre-embedded probes. Then, based on a preset multi-agent collaborative analysis strategy, multiple preset agents process each event flow to obtain result scores, process scores, and standardization scores. The process score is generated based on debugging entropy and convergence index. Debugging entropy characterizes the degree of confusion during code debugging, and convergence index characterizes whether students tend to solve problems during code debugging. Finally, a preset training effectiveness evaluation model is used to process the result scores, process scores, and standardization scores to generate training effectiveness evaluation results.

[0090] Understandably, this application collects process event flows through probes, generates result scores, process scores, and specification scores separately through multiple agents, and integrates them for evaluation. It incorporates the debugging process into the evaluation indicators, making the assessment more objective and comprehensive, and truly reflecting the trainee's comprehensive practical ability.

[0091] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The step of using a process analysis agent to analyze the event streams to determine the debugging entropy and convergence index, and to generate process scores, further includes steps A10 to A40: Step A10: Generate a sequence of actions based on the event streams. It should be noted that, in this application, the sequence of actions is as follows:

[0092] In this embodiment, the process analysis agent performs structured reorganization on the raw event stream collected in step S10. The agent traverses each record in the event stream, aligning and merging scattered keyboard input events, compilation action events, execution result events, and document viewing events according to their timestamps. Redundant noise data irrelevant to the debugging process is removed, and state snapshots and triggered error feedback information before and after each code change are extracted. Ultimately, a logically continuous sequence of behaviors is formed, with debugging actions as nodes and state changes as edges.

[0093] Understandably, the purpose of this step is to perform semantic extraction and noise reduction on the original event stream, transforming the massive amount of raw data that was originally mixed with irrelevant operations into high-quality analytical material containing only key nodes for error correction. This provides a data foundation with a clear structure and semantics for the accurate calculation of subsequent correction vectors, and avoids interference from irrelevant operations on behavioral pattern analysis.

[0094] Step A20: Determine the actual correction vector based on the behavior sequence; Specifically, the step of determining the actual correction vector based on the behavior sequence further includes steps A21 to A23: Step A21: Perform an error event extraction operation on the behavior sequence to construct an error event sequence; It should be noted that, in this application, the error event sequence is as follows:

[0095] Each error event Includes: timestamp Error location (File name + line number), error type (Syntax errors, logical errors, runtime errors, etc., using a predefined error type encoding table), error severity (Levels 1-5, based on whether compilation is blocked, the scope of impact, etc.)

[0096] Step A22: Based on any two adjacent error events in the error event sequence, determine the position offset, error type change, and severity change; Step A23: Construct the actual correction vector based on the position offset, the error type change, and the severity change.

[0097] In this embodiment, the process analysis agent parses the behavior sequence generated in step A10, identifies all event nodes marked as compilation errors or runtime anomalies, and constructs an error event sequence according to the timestamp order. For each pair of adjacent error events in the error event sequence, the agent performs the following three calculations: First, extract the line number of the code error location corresponding to each of the two error events, calculate the absolute value of the difference between the two line numbers, and divide the difference by the total number of lines of code in the training project to obtain the normalized position offset; Second, map the error types of the two error events to the corresponding binary codes in the preset error type encoding table, calculate the Hamming distance between the two binary codes, and divide the distance by the total number of bits in the encoding to obtain the normalized error type change; Third, extract the severity level values ​​corresponding to each of the two error events, calculate the difference between the latter level and the former level, and divide the difference by the maximum severity level difference constant to obtain the normalized severity change. The agent combines the above three normalized values ​​into a triplet as the actual correction vector representing the attribute of the code modification behavior. Repeat the above operation for all adjacent error event pairs in the error event sequence to obtain the complete set of actual correction vectors.

[0098] It should be noted that in this application, for two adjacent error events... and Calculate the correction vector:

[0099] Position offset:

[0100] in, The total number of lines of code, normalized to [0, 1].

[0101] Error type changes:

[0102] Changes in severity:

[0103] Additionally, it should be noted that in this application, the position offset, error type change, and severity change all need to be normalized to [0, 1].

[0104] Understandably, by transforming discrete error events into continuous behavioral change metrics and defining three normalized dimensions—location offset, type change, and severity change—every modification attempt during code debugging is mapped to a comparable and computable vector space for the first time. This provides a mathematical basis for horizontal comparison and vertical tracking of the debugging behaviors of different trainees in different tasks.

[0105] Step A30: Based on the actual correction vector, determine the debugging entropy; based on the preset ideal correction vector and the actual correction vector, determine the convergence index. Specifically, before the step of determining the debugging entropy based on the actual correction vector, steps A31 to A32 are also included: Step A31: Use the K-means clustering algorithm to divide the modified vector space into K states, where the value of K is preset. Step A32: Based on the K states, perform a clustering operation on the sample correction vector to obtain K cluster centers, where each cluster center represents a debugging state.

[0106] It should be noted that K cluster centers refer to Each center represents a debug state. For example, the state semantics are explained as follows: In-situ fine-tuning (location and type remain unchanged, but severity decreases) is an effective correction. The location changes but the error type remains the same, indicating a need for troubleshooting and identification. The error type has changed and its severity has increased, meaning a new error has been introduced. Significant changes across all dimensions, i.e., random trial and error.

[0107] Specifically, the step of determining the debugging entropy based on the actual correction vector further includes steps A33 to A36: Step A33: Combine multiple actual correction vectors in chronological order to construct the debugging trajectory; It should be noted that in this application, the actual correction vectors are arranged in chronological order to form the debugging trajectory:

[0108] Step A34: Map each actual correction vector in the debugging trajectory to the nearest cluster center among the K cluster centers to obtain the mapping result; In this embodiment, each correction vector in the debugging trajectory is mapped to the nearest cluster center, and the mapping result is as follows:

[0109] Step A35: Based on the mapping result, calculate the Shannon entropy of the state distribution, and calculate the directional entropy based on the directional category of the pre-defined correction vector; It should be noted that the Shannon entropy used to calculate the state distribution refers to: in, For state Frequency of occurrence.

[0110] Step A36: Determine the debugging entropy based on the Shannon entropy and the directional entropy.

[0111] It should be noted that the K-means clustering algorithm is an unsupervised machine learning method. Its goal is to automatically divide a set of multidimensional data points into a pre-specified number of groups, so that the data points in the same group are as close as possible in spatial distance, and the data points in different groups are as far apart as possible.

[0112] The modified vector space refers to the space of all possible actual modified vectors. The three-dimensional mathematical space is constructed, with its three coordinate axes corresponding to the dimensions of position offset, error type change, and severity change, respectively.

[0113] The value of K refers to the pre-defined number of cluster groups, which determines the granularity of the modified vector space division.

[0114] Sample correction vectors refer to a set of correction vector instances collected and pre-labeled from a large amount of historical training data, used to train clustering models, and serve as learning materials for determining the location of cluster centers.

[0115] Clustering operation refers to the iterative calculation process of the K-means algorithm on sample data. Each iteration includes assigning each sample point to the state of the nearest cluster center, and recalculating the geometric mean of all sample points in each state based on the assignment results to update the cluster center position, until the cluster center position no longer changes significantly.

[0116] Cluster centers are a set of representative three-dimensional coordinate points determined in the modified vector space after clustering operations. Each coordinate point is the arithmetic mean of the modified vectors of all samples in its corresponding state region.

[0117] Mapping refers to the process of calculating the Euclidean distance between each actual correction vector in the debugging trajectory and the K cluster centers one by one, and marking the actual correction vector as the state identifier of the cluster center with the shortest distance to it.

[0118] The nearest cluster center is the cluster center in the modified vector space that has the smallest Euclidean distance to the actual modified vector to be classified.

[0119] The mapping result refers to a sequence of state identifiers arranged in chronological order, which is obtained by converting the debugging trajectory. Each element in the sequence corresponds to the debugging state category to which the actual correction vector at the same position in the original debugging trajectory belongs.

[0120] The Shannon entropy of state distribution refers to the numerical value obtained by calculating the dispersion of the frequency of occurrence of each debugging state category in the mapping result sequence using the Shannon entropy formula in information theory. It is used to measure the disorder and unpredictability of trainees switching between different troubleshooting behavior modes.

[0121] The pre-defined direction category of the correction vector refers to three types of error correction trends that are artificially divided based on the numerical sign and magnitude of the change in severity in the actual correction vector: trending towards resolution, deviating from resolution, and oscillating in place.

[0122] Orientation entropy is a numerical value calculated by the frequency of occurrence of the orientation category of each actual correction vector in the debugging trajectory. It is used to measure the degree of fluctuation of the trainee's troubleshooting behavior in the macro trend and its consistency with the goal.

[0123] It should be noted that the ideal correction vector is a predefined reference vector that represents the optimal error correction behavior. The position offset of this vector is zero, the error type change is zero, and the severity change is a negative extreme value pointing to a decrease. It represents the ideal situation of fixing the current error in place without introducing new errors.

[0124] It should be noted that this step includes two parallel computational branches. The first branch is used to determine the debugging entropy: the process analysis agent arranges all the actual correction vectors generated in step A20 in timestamp order to form a complete debugging trajectory; it calculates the Euclidean distance between each actual correction vector in the debugging trajectory and multiple cluster centers obtained through pre-training clustering algorithms, and classifies them into the debugging state represented by the nearest cluster center to obtain a state sequence; at the same time, it judges the sign and magnitude of the change in severity in each actual correction vector, and classifies it into one of three categories: tending towards the solution direction, deviating from the solution direction, or oscillating in place, to obtain a direction sequence; it calculates the information entropy of the frequency of occurrence of each state in the state sequence and the information entropy of the frequency of occurrence of each direction category in the direction sequence, and combines the weighted sum of the two with the compensation term calculated based on the convergence index to obtain the debugging entropy. The second branch is used to determine the convergence index: The process analysis agent obtains each actual correction vector in the debugging trajectory one by one, and calculates the cosine similarity between the actual correction vector and the preset ideal correction vector. The cosine similarity of all the angles is accumulated and divided by the total number of actual correction vectors to obtain the arithmetic mean value between negative one and positive one. This value is the convergence index.

[0125] In this embodiment, when calculating the direction entropy, the direction category of the correction vector is first defined: Towards a solution (d=1): or( and and ).

[0126] Deviation solution (d=2): .

[0127] Oscillation in place (d=3): Other cases.

[0128] Furthermore, calculate the directional entropy:

[0129] After obtaining the directional entropy and Shannon entropy, calculate the debugging entropy:

[0130] in: This is the weighting coefficient, with a default value of 0. , The convergence index is denoted as .

[0131] In addition, the convergence index is calculated in this application as follows: First, define the ideal correction vector. This indicates in-place modification, unchanged type, and decreased severity. The convergence index is then:

[0132] in, The closer the value is to 1, the more the debugging tends to converge.

[0133] Understandably, in this embodiment, the debugging entropy integrates the state dispersion and directional disorder of the trainee during the troubleshooting process into a scalar index, providing an objective basis for evaluating the logicality of thinking. The lower the debugging entropy value, the clearer and more convergent the troubleshooting path. At the same time, the convergence index provides a trend measure for judging whether the trainee is effectively approaching the problem solution by comparing the consistency between the actual correction path and the ideal convergence direction. The closer the convergence index is to positive one, the more goal-oriented the troubleshooting strategy is.

[0134] Step A40: Generate process score based on the convergence index and the debugging entropy.

[0135] It should be noted that in this implementation, the process analysis agent obtains the debugging entropy value and convergence index value calculated in step A30. The agent transforms and fuses the two values ​​according to a pre-defined mapping function. The debugging entropy value, after inverse normalization, is converted into an entropy score where lower entropy values ​​result in higher scores. The convergence index, after linear scaling, is converted into a convergence score where stronger convergence trends result in higher scores. The agent then weights and sums the entropy score and convergence score according to preset weighting coefficients to obtain the final process score. A higher process score indicates higher error-correction efficiency, stronger logical thinking, and a behavior pattern closer to the ideal state during the completion of the training task.

[0136] Understandably, the purpose of this step is to aggregate multi-dimensional process quantitative indicators into a single evaluation score. By organically integrating debugging entropy and convergence index into a process score, the originally abstract behavioral pattern analysis conclusions can be presented in an intuitive score form, which is convenient for subsequent weighted fusion calculation with result score and standardization score. At the same time, it also enables trainees and teachers to directly understand and compare the process evaluation results, providing an operable quantitative basis for teaching feedback and personalized guidance.

[0137] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the training effectiveness evaluation method based on multi-agent collaborative analysis of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0138] This application also provides a training effectiveness evaluation device based on multi-agent collaborative analysis. Please refer to [reference needed]. Figure 3 The training effectiveness evaluation device based on multi-agent collaborative analysis includes: Acquisition module 10, the acquisition module is used to acquire various event streams during the training process based on pre-embedded probes; Data processing module 20 is used to process the event streams using multiple preset agents based on a preset multi-agent collaborative analysis strategy, and obtain result score, process score and standard score. The process score is generated based on debugging entropy and convergence index. Debugging entropy represents the degree of confusion of students in the code debugging process, and convergence index represents whether students tend to solve the problem in the code debugging process. The generation module 30 is used to process the result score, the process score and the standard score using a preset training effectiveness evaluation model to generate training effectiveness evaluation results.

[0139] The training performance evaluation device based on multi-agent collaborative analysis provided in this application adopts the training performance evaluation method based on multi-agent collaborative analysis in the above embodiments, and can solve the technical problem of training performance evaluation based on multi-agent collaborative analysis. Compared with related technologies, the beneficial effects of the training performance evaluation device based on multi-agent collaborative analysis provided in this application are the same as the beneficial effects of the training performance evaluation method based on multi-agent collaborative analysis provided in the above embodiments, and other technical features in the training performance evaluation device based on multi-agent collaborative analysis are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0140] This application provides a training performance evaluation device based on multi-agent collaborative analysis. The training performance evaluation device based on multi-agent collaborative analysis includes: at least one processor; and a memory communicatively connected to at least one processor; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to execute the training performance evaluation method based on multi-agent collaborative analysis in the above embodiment 1.

[0141] The following is for reference. Figure 4 This document illustrates a structural diagram of a training performance evaluation device based on multi-agent collaborative analysis suitable for implementing embodiments of this application. The training performance evaluation device based on multi-agent collaborative analysis in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The training performance evaluation device based on multi-agent collaborative analysis shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0142] like Figure 4As shown, the training performance evaluation device based on multi-agent collaborative analysis may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the training performance evaluation device based on multi-agent collaborative analysis. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the training performance evaluation device based on multi-agent collaborative analysis to exchange data with other devices wirelessly or via wired communication. Although the figure shows a training performance evaluation device based on multi-agent collaborative analysis with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0143] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0144] The training performance evaluation device based on multi-agent collaborative analysis provided in this application adopts the training performance evaluation method based on multi-agent collaborative analysis in the above embodiments, and can solve the technical problem of training performance evaluation based on multi-agent collaborative analysis. Compared with related technologies, the beneficial effects of the training performance evaluation device based on multi-agent collaborative analysis provided in this application are the same as the beneficial effects of the training performance evaluation method based on multi-agent collaborative analysis provided in the above embodiments, and other technical features in this training performance evaluation device based on multi-agent collaborative analysis are the same as the features disclosed in the method of the previous embodiment, and will not be repeated here.

[0145] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0146] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0147] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the training performance evaluation method based on multi-agent cooperative analysis in the above embodiments.

[0148] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0149] The aforementioned computer-readable storage medium may be included in a training effectiveness evaluation device based on multi-agent collaborative analysis; or it may exist independently and not be assembled into a training effectiveness evaluation device based on multi-agent collaborative analysis.

[0150] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the training effectiveness evaluation device based on multi-agent collaborative analysis, cause the training effectiveness evaluation device based on multi-agent collaborative analysis to: Based on pre-embedded probes, the event flow of various events during the training process is acquired; Based on a preset multi-agent collaborative analysis strategy, multiple preset agents are used to process the event streams respectively to obtain result score, process score and standard score. The process score is generated based on debugging entropy and convergence index. Debugging entropy represents the degree of confusion of students in the code debugging process, and convergence index represents whether students tend to solve the problem in the code debugging process. The result score, process score, and standard score are processed using a preset training effectiveness evaluation model to generate training effectiveness evaluation results.

[0151] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0153] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0154] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described training effectiveness evaluation method based on multi-agent cooperative analysis, thereby solving the technical problem of training effectiveness evaluation based on multi-agent cooperative analysis. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the training effectiveness evaluation method based on multi-agent cooperative analysis provided in the above embodiments, and will not be elaborated upon here.

[0155] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the training effectiveness evaluation method based on multi-agent collaborative analysis as described above.

[0156] The computer program product provided in this application can solve the technical problem of evaluating training effectiveness based on multi-agent collaborative analysis. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the training effectiveness evaluation method based on multi-agent collaborative analysis provided in the above embodiments, and will not be repeated here.

[0157] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A training effectiveness evaluation method based on multi-agent collaborative analysis, characterized in that, The training effectiveness evaluation method based on multi-agent collaborative analysis includes: Based on pre-embedded probes, the event flow of various events during the training process is acquired; Based on a preset multi-agent collaborative analysis strategy, multiple preset agents are used to process the event streams respectively to obtain result score, process score and standard score. The process score is generated based on debugging entropy and convergence index. Debugging entropy represents the degree of confusion of students in the code debugging process, and convergence index represents whether students tend to solve the problem in the code debugging process. The result score, process score, and standard score are processed using a preset training effectiveness evaluation model to generate training effectiveness evaluation results.

2. The training effectiveness evaluation method based on multi-agent collaborative analysis as described in claim 1, characterized in that, The preset multiple intelligent agents include a result analysis intelligent agent, a process analysis intelligent agent, and a specification check intelligent agent. The step of using the preset multiple intelligent agents to process the event streams based on the preset multi-agent collaborative analysis strategy to obtain result scores, process scores, and specification scores further includes: The result analysis agent is used to analyze the event streams to perform functional testing and static checks, and generate result scores. The process analysis agent is used to analyze the event streams to determine the debugging entropy and convergence index, and to generate process scores. The specification checking agent is used to analyze the event streams to check code comment rate, naming conventions, and structural clarity, and to generate specification scores.

3. The training effectiveness evaluation method based on multi-agent collaborative analysis as described in claim 2, characterized in that, The step of using the process analysis agent to analyze the event streams to determine the debugging entropy and convergence index, and to generate process scores, further includes: Based on the aforementioned event streams, a sequence of behaviors is generated; Based on the behavior sequence, determine the actual correction vector; Based on the actual correction vector, the debugging entropy is determined, and based on the preset ideal correction vector and the actual correction vector, the convergence index is determined. Based on the convergence index and the debugging entropy, a process score is generated.

4. The training effectiveness evaluation method based on multi-agent collaborative analysis as described in claim 3, characterized in that, The step of determining the actual correction vector based on the behavior sequence further includes: Perform an error event extraction operation on the behavior sequence to construct an error event sequence; Based on any two adjacent error events in the error event sequence, the location offset, error type change, and severity change are determined. Based on the position offset, the error type change, and the severity change, an actual correction vector is constructed.

5. The training effectiveness evaluation method based on multi-agent collaborative analysis as described in claim 3, characterized in that, Before the step of determining the debugging entropy based on the actual correction vector, the method further includes: The K-means clustering algorithm is used to divide the modified vector space into K states, where the value of K is predetermined. Based on the K states, a clustering operation is performed on the sample correction vector to obtain K cluster centers, where each cluster center represents a debugging state.

6. The training effectiveness evaluation method based on multi-agent collaborative analysis as described in claim 4, characterized in that, The step of determining the debugging entropy based on the actual correction vector further includes: Multiple actual correction vectors are combined in chronological order to construct a debugging trajectory; Each actual correction vector in the debugging trajectory is mapped to the nearest cluster center among the K cluster centers to obtain the mapping result; Based on the mapping result, the Shannon entropy of the state distribution is calculated, and the directional entropy is calculated based on the directional category of the pre-defined correction vector. Based on the Shannon entropy and the directional entropy, the debugging entropy is determined.

7. The training effectiveness evaluation method based on multi-agent collaborative analysis as described in claim 1, characterized in that, The step of acquiring various event flows during the training process based on pre-implanted probes further includes: Based on students' historical training data, use the Project Response Theory model or the Deep Knowledge Tracking model to estimate students' current abilities. Based on the student's current ability, the perturbation injection agent is used to determine the information gain of each perturbation in the preset perturbation library, and the perturbation with the largest information gain is set as the target perturbation. The target disturbance is injected into the training environment to generate a fault event; Based on pre-embedded probes, the event streams for handling the fault events during the training process are obtained.

8. The training effectiveness evaluation method based on multi-agent collaborative analysis as described in claim 1, characterized in that, The step of processing the result score, process score, and standardization score using a preset training effectiveness evaluation model to generate training effectiveness evaluation results further includes: Based on the aforementioned event streams, the robust tensor within the disturbance response segment is determined using a robust tensor quantization model to obtain a robust comprehensive score. The robustness score, result score, process score, and standardization score are processed using a preset training effectiveness evaluation model to generate training effectiveness evaluation results.

9. A training effectiveness evaluation device based on multi-agent collaborative analysis, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the training effectiveness evaluation method based on multi-agent collaborative analysis as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the training effectiveness evaluation method based on multi-agent collaborative analysis as described in any one of claims 1 to 8.

Images