A simulation assessment system and assessment method integrating theoretical knowledge and course training

Abstract

The present application relates to the field of educational assessment simulation technology, and specifically to a simulation assessment system and assessment method that integrates theoretical knowledge and course training, the method comprising: generating a standard feature sequence based on the operation steps of a physical model in a simulation assessment task; analyzing the marking results of the user's theoretical answers, and comparing the differences between the user's operation sequence and the standard feature sequence, and combining the distribution and discreteness of the elements in the user's operation time series to obtain the user's first score; analyzing the correlation between the user's operation sequence and the user's operation time series, and combining the randomness of the distribution of the elements in the user's operation time series and the first score to obtain the user's second score; and obtaining the user's simulation assessment result based on the numerical value of the user's second score. The present application aims to improve the limitations of the single assessment evaluation dimension and improve the accuracy of the assessment results.

Description

Technical Field

[0001] The present application relates to the field of educational assessment simulation technology, and specifically to a simulation assessment system and assessment method that integrates theoretical knowledge and course training. Background Art

[0002] With the deepening of the digital transformation of education and the reform of core competency assessment, traditional education assessment has long faced the dilemma of "focusing on knowledge memorization and neglecting practical application". Standardized tests are difficult to comprehensively assess students' high-order thinking and problem-solving abilities. Since the 21st century, the maturity of technologies such as virtual reality, big data analysis, and artificial intelligence has provided technical support for building an immersive and intelligent assessment system.

[0003] Current simulation assessment systems generally adopt a result-oriented evaluation model, which only records the final score or the correctness of the operation results, while completely ignoring the key behavioral data of the user during the operation. This evaluation method has the limitation of a single evaluation dimension, which causes the single assessment result of complex assessment scenarios to deviate from the actual skill level. In addition, traditional systems lack the ability to accurately trace incorrect steps and can only feedback "failed the assessment" but cannot point out the specific defective steps and error types, resulting in unclear defect location, making users unclear about the direction of improvement and wasting training resources on repetitive trial and error. Summary of the Invention

[0004] In view of the above, it is necessary to provide a simulation assessment system and assessment method that integrates theoretical knowledge and course training to solve the above problems.

[0005] The first aspect of the present application provides a simulation assessment method that integrates theoretical knowledge and course training, the method comprising:

[0006] Mark the correctness of the theoretical answers of users in each step of the simulation test;

[0007] Analyze the marking results of the user's theoretical answers and compare the differences between the standard operation steps of the physical model in the simulation assessment task and the operation sequence of the physical model by the user. Combined with the time distribution of the user to complete each step and the degree of dispersion of the completion time, the user's first score is obtained;

[0008] Analyze the correlation between the user's operation sequence and the time it takes for the user to complete each step in the simulation assessment, and combine the randomness of the time distribution of the user's completion of each step and the first score to obtain the user's second score;

[0009] Based on the numerical value of the user's second score, the user's simulation assessment result is obtained.

[0010] The standard operating steps for the physical model are determined by the pressing sequence of the sensors in the physical model.

[0011] The correctness of the theoretical answer results of each step of the simulated assessment is marked as follows:

[0012] When the user's theoretical answer to each step is correct, the corresponding marking result is the first preset value; otherwise, the marking result is the second preset value; wherein the first preset value is not equal to the second preset value.

[0013] The first score of the user is obtained as follows:

[0014] Generate a standard feature sequence based on the standard operating steps of the physical model in the simulation assessment task;

[0015] According to the user's operation sequence in the simulation assessment, the user operation sequence is obtained;

[0016] According to the time it takes for users to complete each step in the simulation assessment, the user operation time series is obtained;

[0017] The first result is obtained by forward fusion of the proportion of incorrect label values of the user's theoretical answer results and the difference between the user's operation sequence and the standard feature sequence;

[0018] Calculating the sum of the element mean and the element dispersion of the user operation time series; recording the ratio of the element mean to the sum as the second result;

[0019] The negative correlation mapping of the first result is positively fused with the second result to obtain a first score of the user.

[0020] The difference between the user operation sequence and the standard feature sequence is determined by the edit distance between the two sequences.

[0021] The element dispersion is determined by the standard deviation of all elements in the user operation time series.

[0022] The step of obtaining the user's second score is as follows:

[0023] Calculate the correlation between the user operation sequence and the user operation time series, and perform forward fusion with the sample entropy of the user operation time series to obtain the third result;

[0024] The negative correlation mapping of the third result is positively fused with the first score to obtain a second score of the user operation; wherein the second score is negatively correlated with the positive fusion result and positively correlated with the first score.

[0025] The correlation between the user operation sequence and the user operation time series is determined by a mutual information analysis algorithm.

[0026] The process of obtaining the user's simulated assessment result based on the numerical value of the user's second score includes:

[0027] A threshold range is set, and the user's simulation assessment result is obtained based on the threshold range where the user's second score is located; the simulation assessment result includes: excellent, good, qualified, and unqualified.

[0028] In a second aspect, an embodiment of the present application further provides a simulation assessment system that integrates theoretical knowledge and course training, and implements any one of the simulation assessment methods that integrate theoretical knowledge and course training. The system includes:

[0029] Model configuration and sensor integration module, used to obtain standard operating procedures for physical models in simulation assessment tasks;

[0030] The operation behavior dynamic collection module is used to mark the correctness of the theoretical answer results of the user in each step of the simulation test; obtain the operation sequence of the user in the simulation test; and obtain the time it takes for the user to complete each step;

[0031] The step and time quantitative analysis module is used to analyze the marking results of the user's theoretical answers and compare the differences between the standard operation steps of the physical model in the simulation assessment task and the user's operation sequence on the physical model. The user's first score is obtained by combining the time distribution of the user to complete each step and the degree of dispersion of the completion time;

[0032] A dynamic coupling risk assessment module is used to analyze the correlation between the user's operation sequence and the time it takes for the user to complete each step in the simulation assessment, and to obtain the user's second score by combining the randomness of the time distribution of the user's completion of each step and the first score;

[0033] The multi-level judgment and feedback optimization module is used to obtain the user's simulation assessment result based on the numerical value of the user's second score.

[0034] This application has at least the following beneficial effects:

[0035] 1. Traditional assessments ignore the hidden impact of step sequence errors and time fluctuations on security. Based on the analysis of the differences between user operation sequences and standard feature sequences, as well as the characteristic analysis of user operation time series, this method reflects the standardization of step sequences and the stability of time management, solves the safety hazards caused by incorrect step sequences, eliminates the speculative behavior of users sacrificing stability for speed, and avoids the one-sidedness of single-dimensional scoring.

[0036] 2. To address the problem that existing technologies ignore the dynamic coupling risk of step sequence and time fluctuations, this method reflects the strength of the association between step errors and time confusion through the correlation between user operation sequences and user operation time sequences and the randomness of the distribution of user operation time elements, thereby eliminating the impact of "compliance cheating" based solely on the single dimension of steps or time.

[0037] 3. Set qualified thresholds, good thresholds, and excellent thresholds based on task types to accurately evaluate users' operation scores, solving the one-sidedness of traditional evaluations. Generate an evaluation report that includes weak link positioning and improvement suggestions, forming an "evaluation-feedback-improvement" closed loop, quantifying user skill shortcomings. Through precise defect positioning and closed-loop feedback, reduce repetitive trial and error, shorten the skill compliance cycle, and build a quantifiable, traceable, and optimizable skill evaluation system, which significantly improves training effectiveness and actual operation safety, and provides a more comprehensive solution for standardized training in high-risk industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of the steps of a simulation assessment method that integrates theoretical knowledge and course training is provided in one embodiment of the present application;

[0039] Figure 2 A block diagram of a simulation assessment system that integrates theoretical knowledge and course training is provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the description of the embodiments of this application, words such as "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "or," and "for example" is intended to present the relevant concepts in a concrete manner.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the art of this application. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0042] It should also be noted that the terms "first" and "second" in this application and the accompanying drawings are used to distinguish similar objects, rather than to describe a specific order or sequence. The methods disclosed in the embodiments of this application or the methods shown in the flowcharts include one or more steps for implementing the methods. Without departing from the scope of protection of this application, the order of executing multiple steps can be interchanged with each other, and some steps can also be deleted.

[0043] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0044] The following describes in detail a specific scheme of a simulation assessment system and assessment method that integrates theoretical knowledge and course training provided by this application with reference to the accompanying drawings.

[0045] See also Figure 1 , which shows a flowchart of the steps of a simulation assessment method that integrates theoretical knowledge and course training provided by an embodiment of the present application. The method is as follows Figure 2 The simulation assessment system shown here integrates theoretical knowledge and course training. The system includes: a model configuration and sensor integration module, an operation behavior dynamic acquisition module, a step and time quantitative analysis module, a dynamic coupling risk assessment module, and a multi-level judgment and feedback optimization module.

[0046] The model configuration and sensor integration module is used to obtain the standard operating procedures for the physical model in the simulation assessment task.

[0047] The model utilizes a modular design, encompassing common physical components such as detachable connectors, sensor interfaces, and simulation tools. It also supports the replacement of functional modules such as the robotic arm, electronic components, and fluidic devices based on training needs. The model integrates multiple sensor types, including motion sensors and pressure sensors in this embodiment, to collect real-time data on user operation trajectory, force, and accuracy. All pressure sensors are numbered, and a standard pressure sensor pressing sequence is established based on the specific assessment tasks of the model. This determined pressing sequence is converted into a standard feature sequence, which uniquely identifies the correct pressing sequence.

[0048] It's important to note that when building a physical model, it's necessary to base it on relevant theoretical knowledge, such as mechanical design principles, electronic circuit theory, and fluid mechanics principles, to ensure the model's rationality and feasibility. For example, when designing a robotic arm module, mechanical structure design and kinematics theory must be applied to ensure the arm's flexible and accurate movements. When designing an electronic component module, the operating principles of electronic circuits and the laws of signal transmission must be followed to ensure the proper operation of the electronic components and accurate data collection.

[0049] The operation behavior dynamic collection module is used to mark the correctness of the theoretical answer results of the user in each step of the simulation assessment; obtain the operation sequence of the user in the simulation assessment; and obtain the time it takes for the user to complete each step.

[0050] In the constructed physical model, pressure sensors are deployed at each key operation node (the part where the examinee must perform an operation). When the user touches the node (for example, pressing a fixture into place), the sensor triggers a signal, and the system automatically records the completion of the step and numbers the identified key operation nodes for subsequent identification and processing.

[0051] Before each operation step, the user needs to answer theoretical questions related to the current step, such as mechanical principles, circuit theory, etc., and the system records the correctness of the answer; if the user's answer is correct, it is marked as the first preset value, and if the answer is wrong, it is marked as the second preset value; the marked results of the user's theoretical answers in all steps are combined into a user answer sequence; in this embodiment, the first preset value is 1, and the second preset value is 0.

[0052] The system automatically records the user's pressure sensor pressing steps during the simulated assessment, generating a user operation sequence. It also records the time it takes for the user to perform each step and sorts the steps in order to obtain a user operation time sequence. It should be understood that the user operation sequence and the elements with the same sequence number in the user operation time sequence have a one-to-one correspondence, so the user operation sequence and the user operation time sequence have the same number of elements.

[0053] The step and time quantitative analysis module is used to analyze the marking results of the user's theoretical answers, and compare the differences between the standard operation steps of the physical model in the simulation assessment task and the user's operation sequence on the physical model. The user's first score is obtained by combining the time distribution of the user to complete each step and the discrete degree of the completion time.

[0054] Because the existing simulation assessment system only relies on the final score for evaluation and ignores the key details of the user's operation process, the evaluation results are one-sided and cannot truly reflect the user's comprehensive skill level. If only the correctness of the results is focused on, the user may complete the operation in the wrong order. Although the final score is qualified, the actual operation may cause equipment damage or safety hazards due to the reversal of steps. Secondly, there is a lack of quantitative analysis of time management. Users may frequently experience fluctuations in time consumption due to the pursuit of speed. Such unstable operations cannot be recognized by traditional systems, resulting in superficial training effects and difficulty in improving the standardization and reliability of actual operations.

[0055] To solve the above problem, the proportion of incorrect label values of the user's theoretical answer results is forward fused with the difference between the user operation sequence and the standard feature sequence to obtain the first result; the sum of the element mean and the element dispersion of the user operation time series is calculated; the ratio of the element mean to the sum is recorded as the second result; the negative correlation mapping of the first result is forward fused with the second result to obtain the user's first score.

[0056] In this embodiment, the proportion of incorrect marking values of the user's theoretical answer results is specifically the ratio of the number of steps marked as 0 in the user's answer sequence to the total number of steps, denoted as H; the difference between the user operation sequence and the standard feature sequence is calculated using the edit distance, denoted as D; the element dispersion of the user operation time series is calculated using the standard deviation, denoted as , the element mean of the user operation time series is recorded as The user's first score is recorded as A, and the specific calculation formula is: , where e is a natural constant.

[0057] It should be noted that the edit distance between the user operation sequence and the standard feature sequence represents the minimum number of edit operations between the user operation sequence and the standard feature sequence, and is used to quantify the degree of sequential deviation of the operation steps. The larger the value, the more omissions, redundancies, or sequence errors the user has, which is more likely to cause safety hazards or operation failures. The mean of the user operation time series reflects the average time consumption of the operation. The sum of the mean and standard deviation reflects the user's comprehensive performance between speed and stability by combining the "absolute time consumption" of the time cost with the "fluctuation risk", thus characterizing the comprehensive time cost of the operation. The closer the value is to 1, the more stable the user's operation time management is. The closer the value is to 0, the more drastic the user's operation time fluctuations are and the worse the stability is.

[0058] The first score combines the standardization of steps and the quality of time management to quantify the overall performance of user operations. The larger the A value, the better the user performance.

[0059] The dynamic coupling risk assessment module is used to analyze the correlation between the user's operation sequence and the time it takes for the user to complete each step in the simulation assessment, and combine the randomness of the time distribution for the user to complete each step and the first score to obtain the user's second score.

[0060] In this application, the user's first score only uses single-dimensional weighting characteristics and static evaluation logic, which cannot solve the problem of the lack of dynamic correlation between the user's operation sequence and the user's operation time series. As a result, the evaluation results are difficult to truly reflect the complexity and multifaceted nature of the user's skill level. There is a dynamic correlation between the sequence of steps and time management: for example, if the user spends too much time on a key step, it may obscure the correctness of the sequence, while the wrong step may cause the subsequent operation time to be compressed, exacerbating the overall risk. Therefore, analyzing the user operation sequence or the user operation time series separately cannot fully capture the complexity of user behavior, and a joint analysis of the two is required.

[0061] Based on the above analysis, the second score of the user operation is calculated, specifically: the correlation between the user operation sequence and the user operation time series is calculated, and forward fused with the sample entropy of the user operation time series to obtain a third result; the negative correlation mapping of the third result is forward fused with the first score to obtain the second score of the user operation; wherein, the second score is negatively correlated with the forward fusion result and positively correlated with the first score.

[0062] In this embodiment, the specific process of obtaining the correlation between the two sequences is as follows: taking the user operation sequence and the user operation time series as input, using the mutual information analysis algorithm to calculate the mutual information score between the two sequences, denoted as S. The mutual information quantifies the dependency between the operation steps and time fluctuations. The higher the mutual information value, the stronger the correlation between step errors and operation time confusion; then taking the user operation time series as input, calculate the sample entropy E of the user operation time series.

[0063] In this embodiment, the forward fusion of multiple variables adopts the multiplication calculation method, and the second score of the user operation is recorded as B. The formula is: ; where e represents a natural constant.

[0064] The larger the value of the mutual information score between the user operation sequence and the user operation time series, the stronger the correlation between step errors and operation time fluctuations. For example, step errors lead to subsequent time compression, or time confusion causes step omissions. There is a negative impact of dynamic coupling between steps and time management in user operations. That is, incorrect operations and time loss aggravate each other, and the user's operation standardization is damaged due to the strong correlation and confusion between steps and time. The larger the sample entropy of the user operation time series, the more irregular the user operation time series is, and the user's time management has uncontrollable randomness, which may lead to chaotic time allocation due to step errors or unskilled operation. The comprehensive step sequence correctness and time stability indicate that the user's step sequence is correct and time management is stable. The larger the value, the more the user's operation complies with the standard and the time allocation is reasonable, and the basic skills meet the standards.

[0065] The dynamic coupling evaluation index comprehensively reflects the standardization of steps and the associated risks between steps and the time spent. The larger the value, the correct the user's operation steps are, and there is no strong correlation between steps and time. This indicates that the user's operation is standardized, stable, and low-risk, and the security is stronger. It avoids "cheating" in a single dimension and truly reflects the comprehensive ability in complex operation scenarios.

[0066] The multi-level judgment and feedback optimization module is used to obtain the user's simulation assessment result based on the numerical value of the user's second score.

[0067] Based on the user's second score obtained by the dynamic coupling risk assessment module, the simulation assessment system implements the assessment judgment through the following process.

[0068] 1. Threshold setting: Set the passing threshold for different assessment tasks , good threshold and excellent threshold , where the qualified threshold The value range is 0.5~0.7, which means that the user operation meets the basic safety standards; the good threshold The value range of is 0.7~0.85, which means that the user's requirement for operation standardization is significantly in line with the requirements; the excellent threshold The value range of is 0.85~0.9, which means that the user's representative operation is highly accurate. In this embodiment, 、 and The values of are 0.5, 0.7 and 0.85 respectively.

[0069] 2. Multi-dimensional classification: When the second score B of the user operation is greater than or equal to the excellent threshold When the user's operation steps are completely standardized, time management is highly stable, and the risk is extremely low, the user's assessment is judged as "excellent"; when the second score B of the user's operation is greater than or equal to the good threshold and less than the excellent threshold , it means that the user's operation steps have slight step deviations or time fluctuations, but overall they meet the safety regulations, and the user's assessment is judged as "good"; when the second score B of the user's operation is greater than or equal to the qualified threshold and is less than the good threshold , it means that the user has met the basic requirements, but needs to strengthen the step sequence or time control training, and the user's assessment is judged as "qualified"; when the second score B of the user's operation is less than the qualified threshold When the user is erroneous in their operation steps or has serious time out of control, there are significant safety hazards and they need to be retrained. The user's assessment will be judged as "unqualified".

[0070] 3. Feedback report generation: The system automatically generates a detailed assessment report containing the following content: Based on the two steps above, it clearly marks whether the assessment was passed and the level. Combined with the specific values of S (correlation between steps and time spent), E (time randomness), and A (step and time stability), it points out the user's weaknesses in terms of incorrect step sequence, time fluctuations, dynamic coupling risks, etc., and recommends targeted training courses based on the analysis results.

[0071] Through the dynamic coupling evaluation of the second score B, the simulation assessment system not only quantifies whether the user meets the standards, but also more accurately locates skill deficiencies, realizes the "assessment-feedback-improvement" closed loop, and ensures a deep connection between training effects and actual operational safety.

[0072] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

[0073] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the basic features of the present application. Therefore, from any point of view, the above embodiments of the present application should be regarded as exemplary and non-restrictive; modifications to the technical solutions described in the above embodiments, or equivalent replacement of some of the technical features therein, do not deviate from the essence of the corresponding technical solutions within the scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application.

Claims

1. A simulation assessment method that integrates theoretical knowledge and course training, characterized by: The method includes: Mark the correctness of the theoretical answers of users in each step of the simulation test; Generate a standard feature sequence based on the standard operation steps of the physical model in the simulation assessment task; obtain a user operation sequence based on the user's operation sequence in the simulation assessment; obtain a user operation time series based on the time it takes the user to complete each step in the simulation assessment; forwardly fuse the proportion of incorrect label values of the user's theoretical answer results with the difference between the user operation sequence and the standard feature sequence to obtain a first result; calculate the sum of the element mean and the element dispersion of the user operation time series; record the ratio of the element mean to the sum as a second result; forwardly fuse the negative correlation map of the first result with the second result to obtain the user's first score; Calculating the correlation between the user operation sequence and the user operation time series, and performing forward fusion with the sample entropy of the user operation time series to obtain a third result; performing forward fusion on the negative correlation map of the third result and the first score to obtain a second score of the user operation; wherein the second score is negatively correlated with the forward fusion result and positively correlated with the first score; Based on the numerical value of the user's second score, the user's simulation assessment result is obtained.

2. A simulation assessment method integrating theoretical knowledge and course training as claimed in claim 1, characterized in that: The standard operating procedure for the mock-up is determined by the order in which the sensors in the mock-up are pressed.

3. A simulation assessment method integrating theoretical knowledge and course training as claimed in claim 1, characterized in that: The correctness of the theoretical answer results of the users in each step of the simulation test is marked as follows: When the user's theoretical answer to each step is correct, the corresponding marking result is the first preset value; otherwise, the marking result is the second preset value; wherein the first preset value is not equal to the second preset value.

4. A simulation assessment method integrating theoretical knowledge and course training as claimed in claim 1, characterized in that: The difference between the user operation sequence and the standard feature sequence is determined by the edit distance between the two sequences.

5. A simulation assessment method integrating theoretical knowledge and course training as claimed in claim 1, characterized in that: The element dispersion is determined by the standard deviation of all elements in the user operation time series.

6. A simulation assessment method integrating theoretical knowledge and course training as claimed in claim 1, characterized in that: The correlation between the user operation sequence and the user operation time series is determined by a mutual information analysis algorithm.

7. A simulation assessment method integrating theoretical knowledge and course training as claimed in claim 1, characterized in that: The process of obtaining the user's simulated assessment result based on the numerical value of the user's second score includes: A threshold range is set, and the user's simulation assessment result is obtained based on the threshold range where the user's second score is located; the simulation assessment result includes: excellent, good, qualified, and unqualified.

8. A simulation assessment system integrating theoretical knowledge and course training, which implements a simulation assessment method integrating theoretical knowledge and course training as described in any one of claims 1 to 7, characterized in that: The system comprises: Model configuration and sensor integration module, used to obtain standard operating procedures for physical models in simulation assessment tasks; The operation behavior dynamic collection module is used to mark the correctness of the theoretical answer results of the user in each step of the simulation test; obtain the operation sequence of the user in the simulation test; and obtain the time it takes for the user to complete each step; The step and time quantitative analysis module is used to analyze the marking results of the user's theoretical answers and compare the differences between the standard operation steps of the physical model in the simulation assessment task and the user's operation sequence on the physical model. The user's first score is obtained by combining the time distribution of the user to complete each step and the degree of dispersion of the completion time; A dynamic coupling risk assessment module is used to analyze the correlation between the user's operation sequence and the time it takes for the user to complete each step in the simulation assessment, and to obtain the user's second score by combining the randomness of the time distribution of the user's completion of each step and the first score; The multi-level judgment and feedback optimization module is used to obtain the user's simulation assessment result based on the numerical value of the user's second score.

Images