A multi-modal interactive teaching simulation training system

Abstract

The application relates to the technical field of intelligent teaching simulation training, and discloses a multi-modal interactive teaching simulation training system. The system comprises the following steps: acquiring and fusing voice, action and visual data to form a unified operation behavior flow, and decomposing the unified operation behavior flow into a task sequence according to knowledge points; extracting an interactive quality feature vector from each task segment, and inputting the interactive quality feature vector into an evaluation network constructed based on a knowledge graph; and obtaining a mastery state evaluation result through layer-by-layer mapping with a standard vector. The results are link-fused to generate a skill mastery degree total score and a defect distribution diagram, and then a training intervention scheme containing a knowledge point backtracking path, an interactive complexity ladder and an auxiliary strategy is adaptively generated according to the diagram. The system realizes fine and structured evaluation of complex operation skills, and can dynamically provide accurate and personalized training guidance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent teaching simulation training, in particular to a multi-modal interactive teaching simulation training system. BACKGROUND

[0002] The existing interactive simulation training system usually relies on independent recording and analysis of a single behavior mode, such as processing voice instructions, operation actions or eye movement trajectories respectively. Its evaluation mechanism is mostly based on overall scoring or simple segmented isolated judgment of the training process, and the evaluation index and the knowledge system structure of the teaching content lack deep coupling. This analysis method results in discrete numerical values or general comments, which cannot accurately track the mapping relationship between the operation behavior sequence of the trainee and the specific knowledge mastery point, and it is difficult to locate the specific coordinates of the skill short board in the knowledge structure.

[0003] Due to the disconnection between the evaluation process and the knowledge topology, the intervention strategies generated by the existing technology are usually based on the final total score or simple rules, providing unified review suggestions or fixed mode difficulty adjustments. Such intervention lacks dynamic insight into the internal logical association of individual skill defects of the trainee, and cannot build personalized backtracking paths and sequential training steps that follow the knowledge dependency relationship, resulting in insufficient pertinence and adaptability of training adjustment, which restricts the efficiency and depth of skill cultivation.

[0004] There is a need for a technical means that can fuse and deconstruct multi-modal behavior data into task sequences aligned with knowledge points, and then perform deep state evaluation based on the knowledge network, and dynamically generate structured intervention schemes according to the defect atlas generated by the evaluation. SUMMARY

[0005] The purpose of the present application is to provide a multi-modal interactive teaching simulation training system to solve the problems raised in the background art.

[0006] To achieve the above purpose, the present application provides a multi-modal interactive teaching simulation training system, which comprises:

[0007] A behavior flow acquisition module acquires the operation behavior flow of the target object in the simulation training process, and the operation behavior flow is generated by fusing voice interaction records, body movement records and visual gaze coordinate records;

[0008] A task disassembly module disassembles the operation behavior flow in stages to generate a time sequence containing multiple independent task segments, and each task segment is associated with a preset teaching knowledge point label;

[0009] A feature extraction module separates an interaction quality feature vector from each task segment, and the interaction quality feature vector includes intention recognition accuracy, action execution consistency and visual focus shift degree.

[0010] An evaluation network module constructs a skill mastery evaluation network based on a knowledge graph, and performs layer-by-layer mapping of the interaction quality feature vector and a preset knowledge point mastery standard vector to generate a mastery state evaluation result of the current task;

[0011] A result fusion module performs link-type fusion of the mastery state evaluation results of the multiple task segments to generate a skill mastery degree total score and a skill defect distribution map for the complete training process;

[0012] An intervention generation module adaptively injects a training intervention scheme in subsequent task segments according to the skill defect distribution map, the training intervention scheme including a knowledge point backtracking path, an interaction complexity ladder, and an auxiliary information delivery strategy.

[0013] Preferably, the operation behavior flow is divided into stages to generate a time sequence including multiple independent task segments, including:

[0014] On a preset teaching task timeline, multiple task segmentation points are determined according to knowledge point conversion nodes and training target change points;

[0015] According to the task segmentation points, voice segments corresponding to each segmentation interval are intercepted from the voice interaction record to form voice task units;

[0016] According to the task segmentation points, action data in each segmentation interval is extracted from the limb action record to form action task units;

[0017] According to the task segmentation points, a gaze point sequence in each segmentation interval is extracted from the visual gaze coordinate record to form visual task units;

[0018] The voice task units, action task units, and visual task units in the same segmentation interval are aligned and packaged to generate an independent task segment, and the corresponding teaching knowledge point label is labeled for the task segment.

[0019] Preferably, the interaction quality feature vector is separated from each task segment, including:

[0020] For the voice task units in the task segment, the operation instructions contained therein are parsed through a semantic understanding model, the parsing result is matched with the standard instructions corresponding to the teaching knowledge point label of the task segment, and the intent recognition accuracy is calculated;

[0021] For the action task unit in the task segment, a core action sequence associated with the current task is extracted from the limb action record, the core action sequence is compared with a standard action template, a similarity average of key postures in the action execution process is calculated, and an action execution consistency degree is generated;

[0022] For the visual task unit in the task segment, a key visual target area associated with the current task is extracted from the visual fixation coordinate record, a proportion of a frequency that an actual fixation coordinate falls into the key visual target area in a total fixation frequency is counted, and a visual focus shift degree is generated;

[0023] The intention recognition accuracy, the action execution consistency degree, and the visual focus shift degree are combined into a multi-dimensional vector as the interaction quality feature vector.

[0024] Preferably, the skill mastery evaluation network based on the knowledge graph is constructed, the interaction quality feature vector is mapped layer by layer with a preset knowledge point mastery standard vector, and a mastery state evaluation result of the current task is generated, including:

[0025] A knowledge point network graph is pre-constructed, nodes of the knowledge point network graph represent teaching knowledge points, edges represent logical dependency relationships between knowledge points, and a knowledge point mastery standard vector is stored for each node;

[0026] A target node corresponding to a teaching knowledge point label of the task segment is located from the knowledge point network graph;

[0027] The knowledge point mastery standard vector is read from the target node;

[0028] The interaction quality feature vector corresponding to the task segment is input into the skill mastery evaluation network, and the skill mastery evaluation network includes a multi-layer mapping structure;

[0029] In the first layer mapping structure of the skill mastery evaluation network, a gap value of the interaction quality feature vector and the knowledge point mastery standard vector in each dimension is calculated, and a primary gap vector is generated;

[0030] In the subsequent mapping structure of the skill mastery evaluation network, the primary gap vector is adjusted in a cross-knowledge point weight according to a logical dependency relationship defined by the edges of the knowledge point network graph, and a weighted gap vector is generated;

[0031] The weighted gap vector is normalized and synthesized to generate a numerical form of the mastery state evaluation result of the current task.

[0032] Preferably, the mastery state evaluation results of the multiple task segments are link-fused to generate a skill mastery degree total score and a skill defect distribution map for a complete training process, including:

[0033] acquire a time sequence of all task fragments generated by a complete training process;

[0034] read the mastery state evaluation results of each task fragment in the time sequence in chronological order;

[0035] label the read multiple mastery state evaluation results in a preset knowledge mastery progress coordinate system according to their corresponding teaching knowledge point labels, to form multiple evaluation points;

[0036] connect adjacent evaluation points in the knowledge mastery progress coordinate system to form a mastery degree change curve;

[0037] calculate the ratio of the area under the mastery degree change curve to the total time and the area under the ideal mastery curve, to generate a skill mastery degree total score;

[0038] identify continuous or discrete intervals in the mastery degree change curve that are below a preset threshold, map the continuous or discrete intervals back to a knowledge point network graph, and identify the nodes of the corresponding knowledge points on the graph with highlighted markers to generate a skill defect distribution graph.

[0039] Preferably, the adaptive injection of a training intervention scheme in subsequent task fragments according to the skill defect distribution graph comprises:

[0040] analyze the skill defect distribution graph, identify the highlighted knowledge point nodes, extract their knowledge point labels and dependency relationships, and form a set of knowledge points to be intervened;

[0041] retrieve teaching resources related to the set of knowledge points to be intervened from a preset teaching resource library, the teaching resources including explanation videos, interactive illustrations, and operation examples;

[0042] determine the urgency level of the training intervention scheme based on the decline slope of the mastery degree change curve, the urgency level being used to determine the intervention timing and intensity;

[0043] determine whether the teaching knowledge point label of the current task fragment to be executed is associated with the set of knowledge points to be intervened;

[0044] if there is an association, select the corresponding teaching resources from the knowledge point backtracking path of the training intervention scheme and deliver them before the start of the task or during the execution of the task;

[0045] Meanwhile, adjust the interactive complexity ladder of the training intervention scheme according to the urgency level to dynamically reduce the difficulty and complexity of the interactive steps in the current task fragment;

[0046] According to the urgency level, the auxiliary information delivery strategy of the training intervention scheme is adjusted, and the frequency and detail level of the prompt information are increased.

[0047] Preferably, the adjustment process of the interaction complexity level ladder of the training intervention scheme comprises:

[0048] A reverse correlation model of interaction complexity and knowledge point mastery degree is established.

[0049] From the skill mastery degree total score or from the historical mastery state evaluation result of the knowledge point associated with the current to-be-executed task segment, a real-time mastery degree value is read;

[0050] The real-time mastery degree value is input into the reverse correlation model, and a recommended interaction complexity level is calculated;

[0051] According to the recommended interaction complexity level, the standard interaction process of the current task segment is simplified, and the simplification process includes reducing the number of operation steps, prolonging the single-step operation judgment time, or reducing the operation precision requirement;

[0052] A simplified interaction process conforming to the recommended interaction complexity level is generated to replace the original standard interaction process.

[0053] Preferably, for the voice task unit in the task segment, the operation instruction contained therein is parsed by a semantic understanding model, the parsing result is matched with the standard instruction corresponding to the teaching knowledge point label marked by the task segment, the intent recognition accuracy is calculated, including:

[0054] The voice signal in the voice task unit is preprocessed to remove background noise and segmented into independent voice segments, and the voice segments are converted into text sequences by a voice recognition engine;

[0055] The text sequence is parsed using a semantic understanding model to extract operation instruction keywords and semantic roles, and a semantic framework of the operation instruction is generated;

[0056] From a preset standard instruction library, according to the teaching knowledge point label marked by the task segment, the corresponding standard instruction semantic framework is searched;

[0057] The operation instruction semantic framework obtained by parsing is compared with the standard instruction semantic framework, and the keyword matching degree and the semantic role coverage degree are calculated;

[0058] Based on the weighted average of the keyword matching degree and the semantic role coverage degree, an intent recognition accuracy value is generated.

[0059] Preferably, in the subsequent mapping structure of the skill mastery evaluation network, the preliminary gap vector is adjusted in cross-knowledge point weight according to the logical dependency relationship defined by the edges of the knowledge point network graph to generate a weighted gap vector, which includes:

[0060] The dependency weight coefficient on the edge between the current target node and the adjacent node in the knowledge point network graph is extracted.

[0061] According to the dependency weight coefficient, the knowledge transfer influence factor of the current target node to each adjacent node is calculated to form an influence factor vector.

[0062] The dot product operation of the preliminary gap vector and the influence factor vector is performed to obtain the gap contribution value of each knowledge point.

[0063] Based on the gap contribution value, the dynamic weight distribution of each dimension of the preliminary gap vector is performed to generate a weighted gap vector.

[0064] The weighted gap vector is normalized to ensure that the values of each dimension of the vector are in a unified dimension.

[0065] Preferably, the reverse correlation model of interaction complexity and knowledge point mastery degree includes:

[0066] Collect the knowledge point mastery degree data and the corresponding interaction complexity level data in the historical training record to form a training data set.

[0067] The linear regression algorithm is used to fit the training data set to establish a mathematical mapping relationship between the knowledge point mastery degree and the interaction complexity level.

[0068] Verify the prediction error of the mathematical mapping relationship, adjust the mapping parameters through iterative optimization, and obtain a stable reverse correlation model.

[0069] The reverse correlation model is integrated into the system control module to query and recommend the interaction complexity level according to the real-time knowledge point mastery degree.

[0070] Compared with the prior art, the beneficial effects of the present application are:

[0071] By fusing multiple asynchronous signals such as voice, body movement and visual gaze into a unified operation behavior stream in time sequence, and automatically and structurally decomposing the behavior stream in stages according to the preset teaching knowledge system, each generated independent task fragment is bound with a specific knowledge point label. The conversion from raw, mixed multi-modal behavior data to structured task sequence with clear teaching semantics is realized, so that the subsequent analysis can be based on objective behavior fragments directly related to teaching content, laying a data foundation for accurate evaluation.

[0072] By constructing a skill mastery evaluation network based on a knowledge graph, the extracted interaction quality feature vector and the knowledge point standard are mapped and propagated layer by layer in the network, simulating the association and diffusion process of skill mastery state between knowledge nodes. The evaluation results can be presented in a link fusion manner to generate a skill defect distribution map that not only reflects the single-point mastery situation but also reveals the association between defects. Based on this graph, the system can adaptively derive an intervention scheme containing specific backtracking paths and complexity ladders, realizing a closed loop from in-depth evaluation to individualized intervention on a unified knowledge topology structure, and making the intervention instructions directly act on the weak nodes of the knowledge network. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 A working principle diagram of the multi-modal interactive teaching simulation training system described in the present application;

[0074] Figure 2 A flowchart for generating the mastery state evaluation results of the current task;

[0075] Figure 3 A graph showing the change in skill mastery degree with training time;

[0076] Figure 4 A flowchart for adjusting the interaction complexity ladder;

[0077] Figure 5 A correlation graph of knowledge point mastery degree and recommended interaction complexity in each training phase. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0079] Please refer to Figure 1The application provides a multi-modal interactive teaching simulation training system, which comprises: a behavior flow acquisition module that synchronously collects voice signals, body movement data and eye movement data of a target object in a simulation training process through a microphone array, a motion capture sensor and an eye tracker deployed in a simulation training environment. After time stamp alignment and preliminary filtering processing, these raw data are fused to generate an operation behavior flow containing a time dimension. A task decomposition module receives the operation behavior flow and cuts the continuous behavior flow into a series of independent task segments with clear teaching semantics according to a preset teaching stage division logic, and each task segment is bound with a teaching knowledge point label that identifies the training key point. A feature extraction module processes each labeled task segment, separates and quantifies multi-dimensional features reflecting interactive quality from the task segment, and organizes the features into an interactive quality feature vector containing intention recognition accuracy, action execution consistency and visual focus shift degree.

[0080] An evaluation network module internally pre-constructs a knowledge graph depicting the correlation between knowledge points, and constructs a skill mastery evaluation network based on the graph. The network compares and performs layer-by-layer mapping operation on the received interactive quality feature vector and the knowledge point mastery standard vector retrieved from the knowledge graph, and finally outputs an evaluation result representing the current task mastery level. A result fusion module collects the evaluation results of all task segments in a complete training process, comprehensively analyzes them according to their time sequence and knowledge point correlation, calculates a comprehensive skill mastery degree total score, and visualizes the skill defect distribution graph. An intervention generation module analyzes the weak links revealed by the skill defect distribution graph, dynamically plans and injects individualized training intervention schemes for the task segments to be executed subsequently in combination with the current training progress, which specifically specifies the knowledge point backtracking path, the interactive complexity ladder and the adjustment content of the auxiliary information delivery strategy.

[0081] Embodiment 1: The operation behavior flow is disassembled in stages to generate a time sequence containing multiple independent task segments. The system runs on a preset teaching task timeline that marks the complete training process. The disassembly process determines multiple task segmentation points based on knowledge point conversion nodes and training target change points. These segmentation points divide the continuous timeline into several intervals. Based on the determined task segmentation points, the system extracts the speech segments corresponding to each segmentation interval from the speech interaction record, which are organized into speech task units. At the same time, based on the same task segmentation points, the system extracts all action data occurring in each segmentation interval from the body movement record to form action task units. Based on the same task segmentation points, the system extracts the gaze point sequence in each segmentation interval from the visual gaze coordinate record to form visual task units. After completing the unit extraction of each modality data, the system time-aligns and data-packs the speech task units, action task units, and visual task units belonging to the same segmentation interval to generate an independent and complete task segment. Each generated task segment is automatically labeled with its corresponding teaching knowledge point label.

[0082] In a specific implementation, consider a laparoscopic surgery simulation training scenario, the teaching task timeline records the entire process from the beginning to the end of the surgery, and the behavior flow acquisition module synchronously collects and generates an operation behavior flow containing the fusion of speech interaction records, body movement records, and visual gaze coordinate records. In this scenario, the preset teaching task timeline is divided into multiple macro stages, such as "tissue separation", "vessel clamping", and "tissue resection", the knowledge point conversion node is defined as the time when different surgical instruments are required to be used, or the theoretical boundary point when the surgical operation target changes, and the training target change point corresponds to the preset time marker when the operation precision requirement or operation speed requirement changes. The system determines a series of accurate time stamps as task segmentation points on the continuous teaching task timeline based on these predefined knowledge point conversion nodes and training target change points.

[0083] In some embodiments, the determination of the task segmentation points not only relies on the preset nodes, but also fine-tunes through real-time analysis of key events in the behavior stream, for example, when the instruction keyword "change instrument" appears in the voice interaction record, or the limb motion record shows that the surgical instrument has been in a stationary state for more than a threshold length of time, the system will add a task segmentation point near this time point, and according to all the finally determined task segmentation points, the teaching task timeline is divided into several continuous and non-overlapping time intervals. For each time interval, the system performs the extraction and unitization operation of multi-modal data. The system extracts all audio signals in the time interval from the voice interaction record to form a voice task unit. The system extracts the spatial coordinate sequence, attitude angle sequence and operator's hand joint motion data of all surgical instrument end effectors in the same time interval from the limb motion record to form a motion task unit. The system extracts the two-dimensional coordinate sequence of the operator's line of sight focus on the display screen in the same time interval from the visual gaze coordinate record to form a visual task unit.

[0084] In a specific implementation, the three units need to be accurately aligned and packaged. Since the multi-channel data stream has been time-synchronized with high precision during acquisition, the system binds the voice task unit, motion task unit and visual task unit belonging to the same time interval through a unified timestamp. After binding, the three units are packaged into an independent task segment data structure. The process of labeling the teaching knowledge point tag for each task segment is automated. The system retrieves the standard knowledge point description mapped by the stage name from the knowledge base according to the preset teaching stage corresponding to the time interval of the task segment. For example, the teaching knowledge point tag of the task segment generated corresponding to the time interval of the "vessel clip closure" stage is labeled as "laparoscopic vessel clip closure, precise placement and excitation of the clip applier".

[0085] It can be understood that the number and position of the task segmentation points directly affect the granularity of the task segments. In an alternative specific implementation, the system allows the definition of multiple teaching task timelines with different granularities. A fine-grained timeline will set more knowledge point conversion nodes and training target change points, resulting in a larger number of shorter task segments and more refined teaching knowledge point labels. A macro-granularity timeline produces fewer task segments that cover a wider range of operations. The total number of task segmentation points It can be planned by the following formula:

[0086]

[0087] Wherein: represents the total number of finally determined task segmentation points, represents the number of preset knowledge point conversion nodes, representing the preset number of training target change points, representing the number of task segmentation points added through real-time behavior analysis, by adjusting the preset value of and the generation threshold of , the degree of task disassembly can be flexibly controlled to adapt to training courses of different complexity and different teaching purposes. Finally, the operation behavior flow of a complete simulation training process is converted into a task segment sequence arranged in chronological order, and each task segment contains time-aligned multi-modal data units and an explicit teaching knowledge point label.

[0088] In embodiment 2, the process of separating the interaction quality feature vector from each task segment is described in detail, with emphasis on the calculation method of intent recognition accuracy. For the speech task unit in the task segment, the system first preprocesses it, including removing background noise and dividing continuous speech into independent speech segments. The preprocessed speech segments are converted into text sequences by a speech recognition engine. Then, a semantic understanding model analyzes the text sequence to extract the operation instruction keywords and semantic roles contained therein, and constructs an operation instruction semantic framework. The system retrieves the corresponding standard instruction semantic framework from the preset standard instruction library according to the teaching knowledge point label of the task segment. Align and compare the operation instruction semantic framework obtained by parsing with the retrieved standard instruction semantic framework, and calculate the keyword matching degree and semantic role coverage between them. Based on the calculated keyword matching degree and semantic role coverage, the system takes their weighted average to generate the final intent recognition accuracy value. For the action task unit in the task segment, the system extracts the core action sequence associated with the current task from the limb action record, compares the core action sequence with the pre-stored standard action template frame by frame or key frame, and calculates the average similarity of key poses in the action execution process. This average is defined as the action execution consistency. For the visual task unit in the task segment, the system extracts the key visual target area associated with the current task from the visual gaze coordinate record, and calculates the proportion of the frequency of actual gaze coordinates falling into the key visual target area in the total gaze frequency. This proportion is calculated as the visual focus shift degree. Finally, the system combines the intent recognition accuracy, action execution consistency, and visual focus shift degree into a multi-dimensional vector, which is used as the interaction quality feature vector of the task segment.

[0089] In a specific implementation, a specific simulation circuit welding training scene can be further illustrated in a task fragment about "capacitor polarity identification and welding", the teaching knowledge point label is "positive and negative electrode discrimination of polar capacitors and pad alignment", the system needs to extract the intention recognition accuracy, action execution consistency and visual focus shift from the speech task unit, action task unit and visual task unit contained in the task fragment respectively and combine them into a vector. For the speech task unit in the task fragment, the system performs intention recognition accuracy calculation. The original speech signal in the speech task unit is first preprocessed by the preprocessing module, the preprocessing module uses the algorithm based on spectral subtraction to remove environmental background noise, and uses the endpoint detection technology based on short-time energy to segment continuous speech into independent speech segments containing complete instructions. The preprocessed independent speech segments are sent to the speech recognition engine, which converts the speech segments into text sequences. The semantic understanding model analyzes the text sequence converted, the semantic understanding model uses the sequence-to-sequence model based on attention mechanism to extract the operation instruction keywords and semantic roles in the text, and generates the semantic framework of the operation instruction. The system retrieves the corresponding standard instruction semantic framework from the pre-set standard instruction library according to the teaching knowledge point label "positive and negative electrode discrimination of polar capacitors and pad alignment" annotated in the task fragment. The standard instruction semantic framework includes standard keywords such as "pick up the capacitor", "observe the color band mark", "align the color band corresponding negative electrode pin to the shadow mark area on the PCB board", and standard semantic roles such as "action subject", "operation object" and "target state". The system aligns and compares the parsed operation instruction semantic framework with the retrieved standard instruction semantic framework, calculates the keyword matching degree and semantic role coverage. The keyword matching degree is defined as the proportion of the number of matching keywords in the two frameworks to the total number of keywords in the standard framework, and the semantic role coverage is defined as the proportion of the number of matching semantic roles in the two frameworks to the total number of semantic roles in the standard framework. The intention recognition accuracy value is generated by the weighted average of the keyword matching degree and the semantic role coverage, and its calculation formula is expressed as:

[0090]

[0091] wherein: represents the intention recognition accuracy value, represents the calculated keyword matching degree, represents the calculated semantic role coverage, is a preset weight coefficient between 0 and 1, used to adjust the contribution proportion of the keyword matching degree and the semantic role coverage in the final intention recognition accuracy value.

[0092] In some embodiments, the calculation process of action execution consistency is described in detail for the action task unit in the task segment. The system extracts the core action sequence associated with the knowledge point of "polarity discrimination and pad alignment of polar capacitors" from the limb action data recorded by the action task unit. The core action sequence includes "the posture of tweezers picking up capacitors", "the stability in the process of moving capacitors above the PCB board", and "the fine adjustment action when aligning the polarity". The system compares the extracted core action sequence with the standard action template of the knowledge point. The comparison process first aligns the core action sequence and the standard action template in time, and then calculates the cosine similarity of the operator's hand joint angle vector and the corresponding angle vector in the standard template at multiple preset key time points. The action execution consistency is defined as the arithmetic mean of the cosine similarity values at all key time points. In some embodiments, the system extracts the key visual target area associated with the current task from the visual gaze coordinate record recorded by the visual task unit. For the knowledge point of "polarity discrimination and pad alignment of polar capacitors", the key visual target area is preset as the color band area of the capacitor element and the polarity shadow mark area on the PCB board. The system counts the frequency of the operator's actual gaze coordinates falling into the aforementioned preset key visual target area within the duration of the task segment, and calculates the visual focus deviation as the proportion of the frequency to the total gaze frequency of the task segment.

[0093] It can be understood that the generation of the interaction quality feature vector is a combination of the calculation results of the above three dimensions. The system combines the calculated intention recognition accuracy value, action execution consistency value, and visual focus deviation value into a three-dimensional vector in a predetermined order, such as [intention recognition accuracy, action execution consistency, visual focus deviation]. This multi-dimensional vector is used as the interaction quality feature vector of the current task segment, which is used to represent the multi-modal interaction quality of the operator when performing the task corresponding to the knowledge point. Optionally, in another task segment involving "soldering of integrated circuit chip pins", the core action sequence becomes "electric solder movement trajectory" and "coordination of solder wire feeding action", and the key visual target area becomes "the contact point of the chip pin and the pad", but the overall process of separating the interaction quality feature vector remains consistent with the above description.

[0094] Embodiment 3: see Figure 2, the process of constructing a skill mastery evaluation network based on a knowledge graph, mapping the interaction quality feature vector and the preset knowledge point mastery standard vector layer by layer, and generating the mastery state evaluation result of the current task is described in detail, and the method of cross-knowledge point weight adjustment is further described. The system pre-constructs a knowledge point network graph, the nodes of the graph represent independent teaching knowledge points, the directed edges between the nodes represent the logical dependency relationship between knowledge points, and each node stores a knowledge point mastery standard vector defining the mastery degree. When a task segment needs to be evaluated, the system locates the target node corresponding to the teaching knowledge point label of the task segment from the knowledge point network graph. The system reads the knowledge point mastery standard vector stored in the target node. The interaction quality feature vector corresponding to the task segment is input into the skill mastery evaluation network, which includes multiple mapping structures. In the first mapping structure, the network calculates the gap value of the interaction quality feature vector and the knowledge point mastery standard vector in each dimension, and these gap values form a primary gap vector. In subsequent mapping structures, the primary gap vector is adjusted in cross-knowledge point weight to generate a weighted gap vector according to the logical dependency relationship defined by the edges of the knowledge point network graph. The adjustment process is as follows: analyze the directed edges between the current target node and adjacent nodes in the knowledge point network graph, and extract the dependency weight coefficients on the edges; calculate the knowledge transfer influence factor of the current target node to each adjacent node according to the dependency weight coefficients to form an influence factor vector; perform dot product operation on the primary gap vector and the influence factor vector to obtain the gap contribution value of each knowledge point; based on the gap contribution value, dynamically allocate the weight of each dimension of the primary gap vector to generate a weighted gap vector; normalize the weighted gap vector to make the values of each dimension of the vector in a unified dimension. The network performs synthesis processing on the normalized weighted gap vector, and finally outputs a numerical form of the mastery state evaluation result of the current task.

[0095] In a specific implementation, consider a task segment about "landing approach under instrument flight rules", the teaching knowledge point label of which is "ILS instrument landing system localizer and glideslope intercept and tracking", the system pre-constructed knowledge point network graph contains multiple nodes representing teaching knowledge points such as "basic attitude control", "radio navigation instrument tuning", "localizer identification", "glideslope identification", "localizer tracking", "glideslope tracking", and edges representing the logical dependency relationship between knowledge points, for example, "radio navigation instrument tuning" is a prerequisite knowledge point of "localizer identification", and each node stores a knowledge point mastery standard vector that defines the expected value of each dimension of the interaction quality feature vector in the ideal mastery state. The system locates the target node corresponding to the "ILS instrument landing system localizer and glideslope intercept and tracking" label from the knowledge point network graph, and reads the knowledge point mastery standard vector from the target node. The interaction quality feature vector corresponding to the task segment is input into the skill mastery evaluation network, which contains multiple layers of mapping structures. In the first layer of mapping structure, the network calculates the gap value of the interaction quality feature vector and the knowledge point mastery standard vector in each dimension, generating a primary gap vector.

[0096] In some embodiments, the process of adjusting the cross-knowledge point weight after the generation of the primary gap vector is specifically described. In the subsequent mapping structure of the skill mastery evaluation network, the system adjusts the primary gap vector according to the logical dependency relationship defined by the edges of the knowledge point network graph. The system analyzes the directed edges between the current target node and adjacent nodes in the knowledge point network graph, extracts the dependency weight coefficient on the edge, and the dependency weight coefficient quantifies the influence strength of the prerequisite knowledge point on the mastery degree of the current knowledge point. According to the dependency weight coefficient, the system calculates the knowledge transfer influence factor of the current target node to each adjacent node, forming an influence factor vector. The primary gap vector and the influence factor vector are point multiplied to obtain the gap contribution value of each knowledge point. Based on the gap contribution value, the system dynamically allocates weights to each dimension of the primary gap vector to generate a weighted gap vector. The weighted gap vector generation process can be represented by the following formula:

[0097]

[0098] Wherein: represents the component of the weighted gap vector in the i-th dimension, represents the component of the primary gap vector in the i-th dimension, represents the normalized value of the component of the influence factor vector in the i-th dimension, ​​​is a preset global adjustment coefficient for controlling the strength of the cross-knowledge-point impact. The weighted gap vector is normalized to ensure that the values of each dimension of the vector are in the same dimension. The system synthesizes the normalized weighted gap vector, which can be the mean or weighted sum of all dimensions of the vector, and finally outputs a numerical value between 0 and 1 as the evaluation result of the current task mastery state.

[0099] In some embodiments, the definition of the knowledge point mastery standard vector is related to the specific dimension. For the "ILS instrument landing system heading and glide path interception and tracking" knowledge point, the knowledge point mastery standard vector can be set as , which respectively correspond to the ideal values of intent recognition accuracy, action execution consistency, and visual focus shift. If the interaction quality feature vector extracted from the task segment is , the primary gap vector is calculated as . Assuming that the influence factor vector is calculated through the knowledge graph dependency relationship as , the preset adjustment coefficient is 0.5, the weighted gap vector is calculated as , that is , and the normalized synthesis obtains the mastery state evaluation result.

[0100] Optionally, the number of mapping layers of the skill mastery evaluation network can be configured according to the complexity of the knowledge graph. For a knowledge point network graph with simple dependency relationships, the network can only include two mapping layers of primary gap calculation and weighted synthesis; for a knowledge point network graph with complex dependency relationships, the network can include multiple intermediate mapping layers, each layer introducing more extensive adjacent node dependency relationships for weight adjustment. It can be understood that the cross-knowledge-point weight adjustment makes the mastery state evaluation result not only reflect the direct performance of the current knowledge point, but also consider the possible impact of the mastery of the previous or related knowledge points on the current performance, thereby making the evaluation more contextually relevant. Optionally, the calculation of the influence factor vector can be based on the direction and weight of the edges in the knowledge graph, considering not only the directly adjacent nodes but also the indirectly associated nodes, but the calculation always takes the knowledge point mastery standard vector read from the target node as the benchmark for adjusting the gap vector.

[0101] The process of evaluating the mastery state of multiple task fragments, link fusion, generating skill mastery total score and skill defect distribution map for the complete training process, and generating intervention scheme according to the map is described in detail. The system obtains all task fragments generated by a complete training process and their time sequence arranged in chronological order. According to the chronological order, the system reads the mastery state evaluation result of each task fragment from the sequence in turn. The multiple mastery state evaluation results are marked in the preset knowledge mastery progress coordinate system according to their corresponding teaching knowledge point labels, and each evaluation result forms an evaluation point. The system connects adjacent evaluation points in the knowledge mastery progress coordinate system to form a mastery degree change curve reflecting the change of mastery degree with time. The system calculates the ratio of the area under the mastery degree change curve to the total time length and the area under the ideal mastery curve, which is defined as the skill mastery total score. The system identifies the continuous or discrete intervals below the preset threshold in the mastery degree change curve, maps these intervals back to the knowledge point network graph, and identifies the nodes of the corresponding knowledge points on the graph with highlighted markers, thereby generating a skill defect distribution map. The intervention generation module analyzes the skill defect distribution map, identifies the highlighted knowledge point nodes, extracts their knowledge point labels and dependency relationships in the knowledge graph, and forms a set of knowledge points to be intervened. According to the set of knowledge points to be intervened, the system retrieves relevant teaching resources such as explanation videos, interactive illustrations and operation examples from the preset teaching resource library. The intervention generation module determines the urgency level of the training intervention scheme based on the decline slope of the mastery degree change curve, which is used to determine the intervention time and intensity. For the current task fragment to be executed, the system judges whether its teaching knowledge point label is associated with the set of knowledge points to be intervened; if there is an association, select the corresponding teaching resources from the knowledge point backtracking path of the training intervention scheme and send them before or during task execution. At the same time, adjust the interactive complexity ladder of the training intervention scheme according to the urgency level, dynamically reduce the difficulty and complexity of the interactive steps in the current task fragment. Adjust the auxiliary information delivery strategy of the training intervention scheme according to the urgency level, increase the frequency and detail of the prompt information.

[0102] In a specific implementation, a specific chemical experiment simulation training scenario can be further illustrated, considering a complete "titration analysis operation" training process, which includes four core task fragments: "analytical balance weighing", "volumetric flask constant volume", "pipette pipetting", and "burette use and end point judgment". Each task fragment has an independent mastery state evaluation result generated by the evaluation network module. The system obtains all task fragments and their time-ordered sequences generated by this complete training process. The system reads the mastery state evaluation results of each task fragment in sequence according to the time order, and marks the multiple mastery state evaluation results according to their corresponding teaching knowledge point labels in the preset knowledge mastery progress coordinate system. Each evaluation result forms an evaluation point with horizontal and vertical coordinates in the coordinate system.

[0103] In some embodiments, the connection and calculation process of the evaluation points in the coordinate system is specifically described. The system connects adjacent evaluation points in the knowledge mastery progress coordinate system to form a mastery degree change curve reflecting the change of mastery degree over time or operation progress. The system calculates the ratio of the area under the mastery degree change curve to the total time length and the area under the ideal mastery curve. This ratio is defined as the total skill mastery score. The system identifies continuous or discrete intervals in the mastery degree change curve that are below the preset threshold, maps these intervals back to the knowledge point network graph, and identifies the nodes of the corresponding knowledge points on the graph with highlighted markers, thereby generating a skill defect distribution graph. Refer to Table 1 for a simplified sequence of evaluation results.

[0104] Table 1: Evaluation Result Sequence Table

[0105]

[0106] The calculation formula of the total skill mastery score is:

[0107]

[0108] Where: represents the total skill mastery score, represents the time point the mastery state evaluation value determined by the mastery degree change curve, represents the time interval between adjacent evaluation points, represents the total number of task fragments, represents the area covered by the ideal mastery curve under the same total time length and evaluation point number. The calculation result is a scalar, which is used to quantify the overall operation level.

[0109] In a specific implementation, the process of generating a training intervention scheme according to the skill deficiency distribution map is specifically described. The intervention generation module analyzes the skill deficiency distribution map, identifies the highlighted knowledge point nodes, such as the corresponding "volumetric flask constant volume" and "burette use and end point judgment" nodes in the table, extracts their knowledge point labels and dependency relationships in the knowledge graph, and forms a set of knowledge points to be intervened. According to the set of knowledge points to be intervened, the system retrieves relevant teaching resources from the preset teaching resource library, including "volumetric flask constant volume operation essentials" explanation video, "titration end point color mutation" interactive diagram, and "half titration operation" operation example. The intervention generation module determines the urgency level of the training intervention scheme based on the decline slope of the mastery degree change curve, for example, the evaluation result of the "volumetric flask constant volume" node decreases gently, and the urgency level is set to "medium"; the evaluation result of the "burette use and end point judgment" node drops sharply, and the urgency level is set to "high".

[0110] For the current task fragment to be executed, the system determines whether the teaching knowledge point label is associated with the set of knowledge points to be intervened. If there is an association, select the corresponding teaching resources from the knowledge point backtracking path of the training intervention scheme and send them before the task starts or during the task execution. At the same time, adjust the interactive complexity ladder of the training intervention scheme according to the urgency level, dynamically reduce the difficulty and complexity of the interactive steps in the current task fragment, for example, for tasks with "high" urgency level, "continuous titration and end point judgment" is simplified to "single addition of preset volume and color observation". Adjust the auxiliary information delivery strategy of the training intervention scheme according to the urgency level, increase the frequency and detail of the prompt information, for example, in tasks with "medium" urgency level, only pop up text prompts before key steps; in tasks with "high" urgency level, visual highlight guidance and step-by-step voice guidance are enabled throughout the process.

[0111] Referring to Figure 3In the analysis of the composition of the overall skill mastery score, its visualization is based on the relationship between cumulative training time and mastery, displayed through the stacked area of ​​two contribution dimensions (actual mastery contribution and defect gap contribution). Specifically, the horizontal axis represents cumulative training time (in minutes), and the vertical axis represents mastery (range 0-1.2). The blue area corresponds to the actual mastery contribution, and the orange area corresponds to the defect gap contribution. The superposition of their areas reflects the overall composition of mastery. As shown in the graph, in the early stages of training (around 20 minutes), the actual mastery contribution is the highest (approximately 0.85), while the defect gap contribution is relatively small. As training progresses, the actual mastery contribution fluctuates slightly and then gradually decreases, while the defect gap contribution shows the opposite trend. The dynamic changes of both constitute the overall fluctuation characteristics of mastery. This visualization method clearly presents the changes in the contribution ratio of "actual mastery" and "defect gap" to the overall skill mastery in different training stages, providing an intuitive curve feature basis for determining the urgency level of subsequent intervention plans.

[0112] Example 5: See Figure 4 The adjustment process of the interaction complexity ladder in the training intervention program is described in detail, including the method for establishing the inverse correlation model on which it depends. The system establishes an inverse correlation model between interaction complexity and knowledge point mastery. The establishment process specifically includes: collecting knowledge point mastery data and corresponding interaction complexity level data from historical training records to form a training dataset; using a linear regression algorithm to fit the training dataset and establish a mathematical mapping relationship between knowledge point mastery and interaction complexity level; verifying the prediction error of this mathematical mapping relationship, and adjusting the mapping parameters through iterative optimization to obtain a stable inverse correlation model; and integrating the inverse correlation model into the system control module. When adjusting the interaction complexity ladder, the system reads the real-time mastery value from the total skill mastery score or from the historical mastery status evaluation results of the knowledge points associated with the current task segment. The real-time mastery value is input into the inverse correlation model to calculate the recommended interaction complexity level. Based on the recommended interaction complexity level, the system simplifies the standard interaction process of the current task segment. Simplification methods include reducing the number of operation steps, extending the single-step operation judgment time, or lowering the operation accuracy requirements. The system generates a simplified interaction flow that conforms to the recommended interaction complexity level, and replaces the original standard interaction flow with this simplified flow.

[0113] In practice, the process of inputting real-time mastery values ​​into the back association model and calculating the recommendation interaction complexity level is described in detail. The back association model is expressed as a linear function, and its formula is as follows:

[0114]

[0115] in: the recommended interaction complexity level, the real-time knowledge point mastery value, is the intercept parameter of the linear regression fitting, is the slope parameter of the linear regression fitting. The system substitutes the read real-time mastery value P into the formula to calculate the specific value of . According to the recommended interaction complexity level , the system simplifies the standard interaction process of the current task segment. For the "chest compression depth and frequency control" task of cardiopulmonary resuscitation, the standard interaction process may include "maintain a compression frequency of 100-120 times per minute", "ensure a compression depth of 5-6 centimeters", "ensure full chest recoil", and other precise steps that need to be met simultaneously. Simplification methods include reducing the number of operation steps, such as temporarily removing the real-time monitoring requirement of "ensure full chest recoil"; extending the single-step operation judgment time, such as extending the judgment time window of compression frequency from 15 seconds to 30 seconds; or reducing the operation precision requirement, such as widening the compression depth allowed error range from ±0.5 centimeters to ±1 centimeter. The system generates a simplified interaction process that meets the recommended interaction complexity level , and replaces the original standard interaction process with this simplified process, which is presented to the operator in the next training cycle.

[0116] In some embodiments, the establishment of the reverse correlation model relies on a high-quality historical training data set containing a large number of records of the knowledge point mastery of past students after training under different interaction complexity levels. The linear regression algorithm solves the parameters and by minimizing the sum of squared errors between predicted values and actual recorded values. The model verification stage calculates the prediction error by cross-validation method and adjusts the parameters through iterative optimization to ensure the stability of the model on unseen data. Optionally, the reverse correlation model can also be a piecewise linear function or a nonlinear function to adapt to more complex mapping relationships between knowledge point mastery and interaction complexity. It can be understood that the interaction complexity level is usually discretized into a limited number of levels, with level 1 representing the simplest interaction process and level 5 representing the complete standard interaction process. The system calculates the recommended interaction complexity level The numerical values are mapped to specific discrete levels after being rounded. In some embodiments, the adjustment of the interaction complexity is dynamically continuous, and the system evaluates the real-time changes in the historical mastery state of the knowledge points associated with the current task fragment to be executed, calls the reverse association model multiple times in a single training session, and adjusts the interaction complexity level of the subsequent steps. Optionally, the reverse association model can be independently trained and stored for different knowledge point categories to form a model library, and the specific model most relevant to the current knowledge point is called when adjusting the interaction complexity. It can be understood that the specific strategy and degree of simplification processing are defined by a preset rule library, which maps different interaction complexity levels to a series of specific and executable process modification instructions.

[0117] Referring to Figure 5 In the adjustment of the interaction complexity ladder of the multi-modal interactive teaching simulation training system, the correlation between the knowledge point mastery degree of different training stages and the recommended interaction complexity level is presented. Specifically, the figure shows the average mastery degree (in "points") of each training stage with a green bar chart, and the recommended interaction complexity level (discretized level) of the corresponding stage with an orange line and node. From the time sequence evolution of the training stage, the knowledge point mastery degree in the basic cognitive stage is about 65 points, and the recommended interaction complexity level is the highest (about 2.9 levels); as the stage progresses to the operation practice, comprehensive application, and evaluation stage, the knowledge point mastery degree gradually improves (close to 80 points, 85 points, and 90 points, respectively), and the recommended interaction complexity level shows a significant downward trend (reducing to about 2.4 levels, 2.1 levels, and 1.6 levels, respectively). This correlation feature is highly consistent with the "interaction complexity-knowledge point mastery degree reverse association model" established in the system: the improvement of the knowledge point mastery degree corresponds to the reduction of the interaction complexity level, which embodies the core logic of the system to dynamically simplify the interaction process according to the real-time mastery degree.

[0118] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0119] Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-modal interactive teaching simulation training system, characterized in that, The system comprises: a behavior flow acquisition module, which acquires an operation behavior flow of a target object in a simulation training process, the operation behavior flow being generated by fusion of voice interaction records, body movement records, and visual gaze coordinate records; a task disassembly module, which disassembles the operation behavior flow in stages to generate a time sequence comprising a plurality of independent task segments, each task segment being associated with a preset teaching knowledge point label; a feature extraction module, which separates an interaction quality feature vector from each task segment, the interaction quality feature vector comprising intention recognition accuracy, movement execution consistency, and visual focus shift degree; an evaluation network module, which constructs a skill mastery evaluation network based on a knowledge graph, maps the interaction quality feature vector and a preset knowledge point mastery standard vector layer by layer, and generates a mastery state evaluation result of a current task, comprising: a pre-constructed knowledge point network graph, a node of the knowledge point network graph representing a teaching knowledge point, an edge representing a logical dependency relationship between knowledge points, and a knowledge point mastery standard vector being stored for each node; locating a target node corresponding to a teaching knowledge point label of a task segment from the knowledge point network graph; reading the knowledge point mastery standard vector from the target node; inputting the interaction quality feature vector corresponding to the task segment into the skill mastery evaluation network, the skill mastery evaluation network comprising a plurality of mapping structures; in a first mapping structure of the skill mastery evaluation network, calculating a gap value of the interaction quality feature vector and the knowledge point mastery standard vector in each dimension to generate a primary gap vector; in a subsequent mapping structure of the skill mastery evaluation network, performing cross-knowledge point weight adjustment on the primary gap vector according to a logical dependency relationship defined by an edge of the knowledge point network graph to generate a weighted gap vector; performing normalization processing on the weighted gap vector to ensure that the values of each dimension of the vector are in a uniform dimension; performing normalization and synthesis processing on the weighted gap vector to generate a numerical form of the mastery state evaluation result of the current task; a result fusion module, which performs link-type fusion of the mastery state evaluation results of a plurality of task segments to generate a skill mastery degree total score and a skill defect distribution graph for a complete training process; an intervention generation module, which adaptively injects a training intervention scheme into a subsequent task segment according to the skill defect distribution graph, the training intervention scheme comprising a knowledge point backtracking path, an interaction complexity ladder, and an auxiliary information delivery strategy.

2. The multi-modal interactive teaching simulation training system of claim 1, wherein, The operation behavior flow is disassembled in stages to generate a time sequence comprising a plurality of independent task segments, comprising: determining a plurality of task segmentation points on a preset teaching task timeline according to knowledge point conversion nodes and training target change points; extracting a voice segment corresponding to each segmentation interval from the voice interaction records to form a voice task unit according to the task segmentation points; extracting movement data in each segmentation interval from the body movement records to form a movement task unit according to the task segmentation points; extracting a gaze point sequence in each segmentation interval from the visual gaze coordinate records to form a visual task unit according to the task segmentation points; Align and pack the speech task unit, the action task unit and the visual task unit in the same segmentation interval to generate an independent task segment, and label the corresponding teaching knowledge point label for the task segment.

3. The multi-modal interactive teaching simulation training system of claim 2, wherein, The interaction quality feature vector separated from each task segment includes: For the speech task unit in the task segment, the operation instruction contained therein is parsed through a semantic understanding model, the parsing result is matched with the standard instruction corresponding to the teaching knowledge point label labeled for the task segment, and the intention recognition accuracy is calculated; For the action task unit in the task segment, the core action sequence associated with the current task is extracted from the limb action record, the core action sequence is compared with the standard action template, the similarity average of the key posture in the action execution process is calculated, and the action execution consistency is generated; For the visual task unit in the task segment, the key visual target area associated with the current task is extracted from the visual fixation coordinate record, the proportion of the frequency of the actual fixation coordinate falling into the key visual target area in the total fixation video is calculated, and the visual focus shift degree is generated. The intention recognition accuracy, the action execution consistency and the visual focus shift degree are combined into a multi-dimensional vector as the interaction quality feature vector.

4. The multi-modal interactive teaching simulation training system of claim 3, wherein, The link type fusion of the mastery state evaluation results of the multiple task segments generates the skill mastery degree total score and the skill defect distribution map for the complete training process, including: Obtain the time sequence of all task segments generated by a complete training process; According to the time sequence, the mastery state evaluation results of each task segment are read in sequence from the time sequence; The read multiple mastery state evaluation results are marked in a preset knowledge mastery progress coordinate system according to the corresponding teaching knowledge point label, forming multiple evaluation points; Connecting adjacent evaluation points in the knowledge mastery progress coordinate system forms a mastery degree change curve; The ratio of the area under the mastery degree change curve to the total time and the area under the ideal mastery curve is calculated to generate the skill mastery degree total score; Identify the continuous or discrete intervals in the mastery degree change curve that are lower than the preset threshold, map the continuous or discrete intervals back to the knowledge point network map, and identify the nodes of the corresponding knowledge points on the map with highlighted markers to generate the skill defect distribution map.

5. The multi-modal interactive teaching simulation training system of claim 4, wherein, According to the skill defect distribution map, the training intervention scheme is adaptively injected in the subsequent task segment, including: Parse the skill defect distribution map to identify the highlighted knowledge point nodes, extract their knowledge point labels and dependency relationships, and form a set of knowledge points to be intervened; According to the set of knowledge points to be intervened, search for relevant teaching resources in the preset teaching resource library, including explanation videos, interactive illustrations and operation examples; Based on the descending slope of the mastery degree change curve, determine the urgency level of the training intervention scheme, which is used to determine the intervention time and intensity; For the task segment to be executed at present, it is judged whether its teaching knowledge point label is associated with the set of knowledge points to be intervened; If there is an association, the corresponding teaching resources are selected from the knowledge point backtracking path of the training intervention scheme and sent before the task starts or during the task execution; Meanwhile, the interactive complexity ladder of the training intervention scheme is adjusted according to the urgency level to dynamically reduce the difficulty and complexity of the interactive steps in the current task segment; The assistance information delivery strategy of the training intervention scheme is adjusted according to the urgency level to increase the frequency and detail of the prompt information.

6. The multi-modal interactive teaching simulation training system of claim 5, wherein, The adjustment process of the interactive complexity ladder of the training intervention scheme includes: Establishing an inverse association model of interactive complexity and knowledge point mastery degree; Reading the real-time mastery degree value from the total score of skill mastery degree or from the historical mastery state evaluation result of the knowledge point associated with the current to-be-executed task segment; Inputting the real-time mastery degree value into the inverse association model to calculate the recommended interactive complexity level; According to the recommended interactive complexity level, the standard interactive process of the current task segment is simplified, and the simplification process includes reducing the number of operation steps, prolonging the single-step operation judgment time, or reducing the operation precision requirement; The simplified interactive process conforming to the recommended interactive complexity level is generated to replace the original standard interactive process.

7. The multi-modal interactive teaching simulation training system of claim 3, wherein, For the voice task unit in the task segment, the operation instruction contained therein is analyzed by a semantic understanding model, the analysis result is matched with the standard instruction corresponding to the teaching knowledge point label marked in the task segment, the intention recognition accuracy is calculated, including: The voice signal in the voice task unit is preprocessed to remove background noise and segmented into independent voice segments, and the voice segments are converted into text sequences by a voice recognition engine; The semantic understanding model is used to analyze the text sequence, extract the operation instruction keywords and semantic roles in the text, and generate the semantic framework of the operation instruction; From the preset standard instruction library, the corresponding standard instruction semantic framework is retrieved according to the teaching knowledge point label marked in the task segment; The operation instruction semantic framework obtained by analysis is aligned and compared with the standard instruction semantic framework to calculate the keyword matching degree and semantic role coverage degree; Based on the weighted average of the keyword matching degree and the semantic role coverage degree, an intention recognition accuracy value is generated.

8. The multi-modal interactive teaching simulation training system of claim 1, wherein, In the subsequent mapping structure of the skill mastery evaluation network, the primary gap vector is adjusted across knowledge points based on the logical dependency relationship defined by the edges of the knowledge point network graph to generate a weighted gap vector, including: Analyzing the directed edges between the current target node and the adjacent nodes in the knowledge point network graph to extract the dependency weight coefficients on the edges; According to the dependency weight coefficients, calculate the knowledge transfer influence factor of the current target node to each adjacent node to form an influence factor vector; Dot product operation is performed on the primary gap vector and the influence factor vector to obtain the gap contribution value of each knowledge point; Based on the gap contribution value, dynamically allocate weights to each dimension of the primary gap vector to generate a weighted gap vector.

9. The multi-modal interactive teaching simulation training system of claim 6, wherein, The inverse association model of interactive complexity and knowledge point mastery degree includes: Collect the knowledge point mastery data and the corresponding interaction complexity level data in the historical training records to form a training data set; Use a linear regression algorithm to fit the training data set, and establish a mathematical mapping relationship between the knowledge point mastery and the interaction complexity level; Verify the prediction error of the mathematical mapping relationship, adjust the mapping parameters through iterative optimization, and obtain a stable reverse correlation model; Integrate the reverse correlation model into the system control module to query the recommended interaction complexity level according to the real-time knowledge point mastery.

Images