Method and system for dynamic reconstruction of practical training teaching tasks based on operation logic atlas

Abstract

The application discloses a practical training teaching task dynamic reconstruction method and system based on operation logic atlas, belongs to the field of intelligent education and digital twin technology, and comprises the following steps: constructing a space-time operation hypergraph of a practical training task, collecting hand movement trajectories of learners through motion capture gloves or a depth camera, collecting eye movement fixation points through an eye tracking instrument, collecting equipment pre-operation signals through pre-operation sensors arranged on practical training equipment, inputting the collected multi-modal data into a time sequence fusion network, mapping to an operation manifold space constructed based on the space-time operation hypergraph, and obtaining an operation intention vector of the learners. The application configures an error cost level for each operation base element in the space-time operation hypergraph, establishes a complete decision chain of "intention prediction, cognitive distance calculation, conflict determination, error cost query and hierarchical response", and realizes intelligent hierarchical processing of operation deviation of students.

Description

Technical Field

[0001] This invention relates to the fields of smart education and digital twin technology, and in particular to a method and system for dynamic reconstruction of practical training tasks based on operational logic graphs. Background Technology

[0002] Practical training is a core component of cultivating skilled talent. Traditional practical training typically relies on teachers demonstrating on-site and students operating according to fixed procedures, lacking flexibility and personalization in the teaching process.

[0003] In recent years, practical training-assisted teaching schemes based on operational logic graphs have emerged. For example, Chinese patent CN122044359A discloses a VR-assisted teaching system for the installation of steam turbines in thermal power plants based on logic constraint graphs. This system abstracts the steam turbine installation process as a directed acyclic graph (DAG) and uses a sequence programming engine to dynamically generate a task list. However, this scheme has the following limitations: Directed edges in a DAG can only connect two nodes, making it impossible to directly express multi-operation convergence constraints such as "D can only be executed if three conditions A, B, and C are met simultaneously". When encountering complex parallel or convergence logic, a large number of virtual nodes need to be introduced, leading to an expansion of the graph size and a decrease in maintenance and query efficiency. This solution only performs "precondition verification and interception" for deviations in user operations, and cannot distinguish between low-risk exploratory behavior and high-risk erroneous behavior. It uniformly reports errors or interrupts all deviations and does not support exploratory learning within the safety boundary. The solution only provides VR-based guidance and lacks deep integration between multimedia teaching and practical training, resulting in a cognitive gap for students when switching between theoretical learning and hands-on training.

[0004] Therefore, there is an urgent need for a method and system that can perceive trainees' operational intentions in real time in a virtual-real integrated environment, make intelligent hierarchical decisions based on operational safety levels, and dynamically reconstruct training task paths. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in the prior art, and to propose a method and system for dynamic reconstruction of practical training tasks based on operation logic graphs.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method and system for dynamically reconstructing practical training tasks based on operational logic graphs, comprising the following steps: S1. Construct a spatiotemporal operation hypergraph for the training task. The nodes of the spatiotemporal operation hypergraph represent operation primitives. Each operation primitive is configured with operation attributes, cognitive attributes and error cost levels. Hyperedges represent the order relationship, branch relationship and dependency relationship between operation primitives. S2. The learner's hand movement trajectory is collected by motion capture gloves or a depth camera, the eye gaze point is collected by an eye tracker, and the pre-operation signal of the equipment is collected by a pre-operation sensor deployed on the training equipment. The collected multimodal data is input into a temporal fusion network and mapped to the operation manifold space constructed based on the spatiotemporal operation hypergraph to obtain the learner's operation intention vector. S3. Calculate the cognitive distance between the intention operation sequence pointed to by the operation intention vector and the current preset guidance sequence in the operation manifold space; S4. If the cognitive distance exceeds a preset threshold, it is determined that an intention-guidance conflict has occurred, and the error cost level corresponding to each operation primitive in the intention operation sequence is queried from the spatiotemporal operation hypergraph. S5. If all the error cost levels found are lower than the preset safety threshold, abandon the current preset guidance sequence, take the intention operation sequence as the starting point, extract the shortest cognitive distance path to the original task goal from the spatiotemporal operation hypergraph, splice them to generate a reconstructed task sequence, and render the first operation guidance and the second operation guidance corresponding to the reconstructed task sequence through the multimedia display device and the augmented reality display device respectively; otherwise, control the augmented reality display device to render and overlay a visual force field cues on the real device operation component corresponding to the intention operation sequence, which changes color or increases resistance as the learner's hand approaches, as a third operation guidance; S6. Feed back the interactive response data under the action of the first operation guide, the second operation guide, or the third operation guide to the learner's cognitive model.

[0007] As a preferred embodiment, the operational manifold space is pre-constructed by embedding the node and hyperedge relationships in the spatiotemporal operational hypergraph into a low-dimensional continuous vector space through a graph neural network.

[0008] As a preferred option, the error cost level is dynamically updated during system operation using a multi-dimensional assessment model that integrates the degree of equipment damage, the degree of personal injury, and the cost of teaching recovery.

[0009] As a preferred embodiment, the cognitive distance is the distance between the current operation sequence embedding trajectory and the target task sequence embedding trajectory, calculated by a geodesic distance algorithm that incorporates cognitive load weights between operation primitives, wherein the cognitive load weights are pre-stored in the hyperedges of the spatiotemporal operation hypergraph.

[0010] As a preferred embodiment, the temporal fusion network includes an attention mechanism module, which is used to asynchronously align the hand movement trajectory, eye gaze point and device pre-operation signal in the time dimension, and dynamically weight and fuse the features of each modality to output the operation intention vector and the corresponding confidence level.

[0011] As a preferred embodiment, the visual force field cues work in conjunction with the force feedback glove, which is linked to the augmented reality display device to provide a sense of resistance that increases as the learner's hand approaches.

[0012] As a preferred embodiment, the first and second operation instructions are generated using a progressive display strategy or a key information partial hiding strategy based on the learner's current skill level, wherein the learner's current skill level is obtained from the learner's cognitive model.

[0013] As a preferred option, it also includes: The learner's operational motion data is collected through the motion capture glove or depth camera, eye movement data is collected through the eye tracker, and cognitive test data is pushed through the multimedia display device and the answer data is collected. Perform triangulation verification, which includes verification of action standardization, gaze accuracy, and cognitive accuracy. Only when the accuracy of the action, the accuracy of the gaze, and the accuracy of the cognition all meet their respective preset conditions will the corresponding operation node be marked as deep mastery in the learner's cognitive model. The updated learner cognitive model will be used as the input for the next cognitive distance calculation. The verification weights for action standardization, gaze accuracy, and cognitive accuracy are adaptively adjusted based on the type of the current operation primitive.

[0014] As a preferred embodiment, the construction of the spatiotemporal operational hypergraph for the training task includes: Define a set of general operation primitive templates that are independent of professional fields. The general operation primitive templates include at least operation semantic tags, spatiotemporal attribute slots, and cognitive attribute slots. Obtain the operating procedures for the target professional training, and decompose the operating procedures into a multi-level operating step tree; Each level of operation step is instantiated into the general operation primitive, and domain-specific operation attributes, cognitive attributes, and the error cost level are populated. Identify the sequential, branching, and dependency relationships among the various operational primitives and construct hyperedges to form the spatiotemporal operational hypergraph.

[0015] The system for dynamically reconstructing practical training tasks based on operational logic graphs includes a graph construction module, a multimodal acquisition hardware group, a state field and intent prediction module, a conflict resolution and reconstruction engine, an adaptive dual-end guidance module, and a cross-modal verification module. The graph construction module is used to construct a spatiotemporal operation hypergraph, where nodes represent operation primitives and each operation primitive is configured with an error cost level. The multimodal acquisition hardware group includes motion capture gloves or depth cameras, eye trackers, and pre-operation sensors deployed on the training equipment. The state field and intent prediction module is connected to the multimodal acquisition hardware group and is used to receive the hand movement trajectory, eye gaze point and device pre-operation signal acquired by the multimodal acquisition hardware group, map the operation manifold space through the built-in temporal fusion network and output the operation intent vector. The input terminals of the conflict resolution and reconstruction engine are respectively connected to the graph construction module and the state field and intent prediction module, and are used for: Receive the operation intention vector, and calculate the cognitive distance between the intention operation sequence pointed to by the operation intention vector and the preset guidance sequence in the operation manifold space; If the cognitive distance exceeds a preset threshold, query the graph construction module for the error cost level of each operation primitive in the intention operation sequence; If all the error cost levels found are below the preset safety threshold, a reconstruction task sequence is generated and output to the adaptive dual-end guidance module; otherwise, a flexible dissuasion instruction is output. The adaptive dual-end guidance module has its input end connected to the conflict resolution and reconstruction engine, and its output end connected to both a multimedia display device and an augmented reality display device, for the purpose of: According to the reconstruction task sequence, the multimedia display device and the augmented reality display device are controlled to render the first operation guide and the second operation guide respectively, and the first operation guide and the second operation guide are atomically consistent in terms of operation steps and spatial position. According to the gentle dissuasion instruction, the augmented reality display device is controlled to render and overlay a visual force field cues on the real device operation component corresponding to the intention operation sequence, which changes color or increases resistance as the learner's hand approaches. The state field and intention prediction module is also connected to the adaptive dual-end guidance module through a feedback link, which is used to receive the learner's interactive response data under each operation guidance and update the learner's cognitive model. The input end of the cross-modal verification module is connected to the adaptive dual-end guidance module and the multimodal acquisition hardware group, respectively, and the output end is connected to the state field and intent prediction module. The cross-modal verification module is used to collect action data, eye-tracking data, and cognitive test data, perform triangular consistency verification, and send the judgment result of deep mastery or mechanical imitation to the state field and intention prediction module to trigger the confidence update of the corresponding node in the learner's cognitive model. The updated learner cognitive model is used by the conflict resolution and reconstruction engine for the next cognitive distance calculation.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention, by configuring error cost levels for each operational primitive in the spatiotemporal operational hypergraph and establishing a complete decision-making chain of "intent prediction, cognitive distance calculation, conflict determination, error cost query, and graded response," achieves intelligent graded processing of student operational deviations: for low-cost errors, the task path is dynamically reconstructed to support inquiry-based learning; for high-cost errors, a visual force field is used for gentle dissuasion to ensure safety, thereby solving the technical problem that the existing "one-size-fits-all" interception mechanism cannot balance teaching safety and learning autonomy.

[0017] 2. This invention embeds a spatiotemporal operation hypergraph into the operation manifold space through a graph neural network, and uses geodesic distance with cognitive load weights as a measure of cognitive distance. This allows the deviation between the trainee's operational intention and the preset path to be continuously quantified, rather than the simple binary judgment in the prior art. This supports more accurate dynamic path replanning, and the reconstructed new path can guide the trainee from the current deviation point to the original task objective with minimal cognitive cost.

[0018] 3. This invention achieves deep teaching linkage in a virtual-real integrated environment by providing atomic-level consistent synchronous guidance between the digital twin interface on the multimedia terminal and the real device on the augmented reality terminal, as well as a comprehensive evaluation and closed-loop feedback of the trainee's action standardization, gaze accuracy, and cognitive comprehension through the "do-see-understand" triangular verification mechanism. It can intelligently distinguish between "deep mastery" and "mechanical imitation" and dynamically adjust the cognitive load weight of subsequent task paths according to the evaluation results, forming an adaptive teaching optimization closed loop. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the system hardware architecture of the method for dynamic reconstructing practical training tasks based on operation logic graphs proposed in this invention. Figure 2 This is a schematic diagram of the data structure of the spatiotemporal operation hypergraph in the method for dynamic reconstruction of practical teaching tasks based on operation logic graph proposed in this invention. Figure 3 This invention presents a method for dynamically reconstructing practical training tasks based on operational logic graphs, and a full-process signaling interaction timing diagram within the system. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] Example, refer to Figures 1 to 3 The core concept of the method and system for dynamic reconstruction of practical training tasks based on operational logic graphs is as follows: Constructing a spatiotemporal operational hypergraph that includes operational attributes, cognitive attributes, and error cost levels; real-time collection of trainees' operational behavior data through multimodal sensors to predict their operational intentions; calculating the cognitive distance between the intentional operation sequence and the preset guidance sequence in the operational manifold space; when an intention-guidance conflict occurs, a graded response is implemented based on the error cost level of each node in the intentional operation sequence: for low-cost conflicts, a new path to the achievable goal is dynamically replanned in accordance with the trainee's intention, generating a self-consistent two-way guidance; for high-cost conflicts, a visual force field prompt of gentle dissuasion is generated; finally, the interaction results are fed back to the trainee's cognitive model, forming a closed loop.

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the main terms used in the specification and claims are first clearly defined.

[0026] Spatiotemporal Operational Hypergraph: This refers to a data structure built based on hypergraph theory to describe the operational process of practical training. In graph theory, the set of nodes V in a hypergraph H=(V,E) represents an entity, and each hyperedge in the set of hyperedges E can connect two or more nodes. The reason why this invention uses a hypergraph instead of a regular directed graph is that: the edges of a regular graph can only connect two nodes, making it difficult to directly express complex relationships such as parallel operations and multi-condition convergence in practical training; a hyperedge can connect multiple predecessor nodes and one successor node simultaneously, naturally expressing the convergence constraint that "operations A, B, and C must be completed before operation D can be performed"; similarly, a hyperedge can also represent the situation where multiple parallel branches branch from one operation node, i.e., "after operation A is completed, B, C, and D can be executed in parallel". The directionality (directed or mixed) of the hyperedge is configured according to the specific constraint type of the practical training task. In addition, the hyperedge can carry multi-dimensional weights, including temporal constraints, cognitive load coefficients, and security correlations.

[0027] For example, in cardiopulmonary resuscitation training, the determination of "effective chest compressions" requires the simultaneous fulfillment of three conditions: "compression depth 5-6cm", "compression rate 100-120 times / min", and "compression location in the middle and lower part of the sternum". This convergence relationship is represented by a hyperedge, which connects the three condition nodes and one determination node.

[0028] Operational primitives are the smallest executable units in a training operation process, and cannot be further decomposed into meaningful lower-level operations. Operational primitives are node entities in the spatiotemporal operation hypergraph, and each operational primitive is configured with structured multidimensional attribute data. Taking the "tighten flange bolts" operation as an example, its operational primitive data structure is shown in the table below:

[0029] Examples of operational primitives for different specializations are as follows: Mechanical and Electrical Engineering Major: "Press the emergency stop button," "Rotate the inverter knob"; Medical profession: "puncture and needle insertion", "injection of drugs"; Electrical Engineering Major: "Open the disconnect switch", "Connect the grounding wire"; Programming major: "Define variables", "Write loop bodies".

[0030] Error cost level: This refers to a quantitative indicator that quantifies the severity of the consequences caused by an operational error in each operational primitive. This invention uses a multi-dimensional evaluation model for comprehensive assessment, and the calculation formula is as follows: Cost level = round(w1×D+w2×H+w3×R) Wherein, D is the equipment damage index (scale 1-5, where 1 represents no damage and 5 represents equipment scrap), H is the personal injury index (scale 1-5, where 1 represents no injury and 5 represents fatal injury), R is the teaching recovery cost index (scale 1-5, where 1 represents immediate correction and 5 represents the need to restart the entire process), and w1, w2, and w3 are weighting coefficients. Since the primary principle of practical training is to ensure personal safety, the personal injury index H has the highest weight, with default values ​​of w1=0.3, w2=0.5, and w3=0.2. The comprehensive evaluation results are rounded to the nearest integer and mapped to levels 1-5, where level 1 represents low cost (allowing for trial and error) and level 5 represents high cost (must be prevented).

[0031] The error cost level is initially assessed based on the operating procedures and safety manual during system construction. During system operation, it can be dynamically updated using the above model based on group learning data (such as the actual error occurrence rate of a certain operation node and the severity of the error consequences).

[0032] Operational manifold space: This refers to a structured data representation that encodes the topology and node features of a spatiotemporal operational hypergraph into a low-dimensional continuous vector space using graph neural networks, with geodesic distance as the metric for distance between any two points in this space. Functionally, this space is equivalent to an embedding manifold, allowing for continuous measurement of distances between different operation sequences via geodesic distance. Specifically, it is constructed using a Graph Attention Network (GAT) as the encoder, mapping each operational primitive node in the spatiotemporal operational hypergraph to a d-dimensional embedding vector (e.g., d=128). Input features include a combination of encodings of the node's operational attributes, cognitive attributes, and error cost levels. The training objective is to define adjacent nodes in an expert operation sequence as positive sample pairs and randomly sampled non-adjacent nodes as negative sample pairs. A contrastive learning loss function is used for training, ensuring that positive sample pairs are closer in the embedding space and negative sample pairs are farther apart. After training, each operational primitive corresponds to the coordinates of a point in the operational manifold space, and any operation sequence corresponds to a trajectory formed by connecting several points in the space.

[0033] It should be noted that, in this invention, the term "manifold" in "operational manifold space" refers to a space that is functionally equivalent to an embedded manifold: that is, in this space, the distance between any two points is measured using geodesic distance (rather than simple Euclidean distance), so that the distance can reflect the true path length along the inherent structure of the data distribution. This space is composed of a discrete set of points obtained by GAT embedding, and together with the weighted graph shortest path described below as an engineered approximation of geodesic distance, it constitutes the "operational manifold space" in the context of this invention.

[0034] Cognitive distance refers to the difference between two operation sequence trajectories in the operational manifold space. It is defined as follows: the transition cost between adjacent nodes i and j in the operational manifold space is set as the product of the Euclidean distance between the embedding vectors of the two nodes and the cognitive load coefficient for transitioning from operation i to operation j (this cognitive load coefficient is pre-stored in the corresponding hyperedge of the spatiotemporal operational hypergraph). Based on this, the cognitive distance between any two operation nodes is defined as the shortest path length from node i to node j on the weighted complete graph (calculated using Dijkstra's algorithm); the cognitive distance between two operation sequences is defined as the sum of the minimum weighted complete matching costs between the node sets of the two sequences (solved using the Hungarian algorithm). The larger this distance, the greater the deviation in cognitive cost and operational cost between the student's current intended path and the preset path. The cognitive distance of this invention is the shortest path in the weighted graph constructed in the embedding space, functionally equivalent to the geodesic distance on the manifold, and is an engineering approximation of this invention in a specific application scenario.

[0035] Atomic-level consistency: refers to the operation instructions presented synchronously on multimedia display devices and augmented reality display devices, maintaining consistency at the granularity of the smallest distinguishable operation steps; the smallest distinguishable operation step refers to a complete execution of a single operation primitive (e.g., "press the emergency stop button", "rotate the valve 90 degrees", "insert the positioning pin"), the boundary of which is defined by the start and end events of the operation primitive. Atomic-level consistency specifically includes the following three dimensions of synchronization: (1) Time synchronization: the multimedia end and the augmented reality end display the same operation step number at the same time; (2) Spatial synchronization: the three-dimensional spatial pose of the indicator arrows or guide lines superimposed on the real device components on the augmented reality end corresponds completely to the same component highlighted on the digital twin model on the multimedia end, and the spatial transformation relationship between the two is guaranteed by a pre-calibrated transformation matrix; (3) Semantic synchronization: the operation instruction text and parameter values ​​displayed on both ends are completely consistent. The underlying principle is to establish a spatial transformation matrix between the real training equipment and its digital twin model in advance through spatial registration or point cloud matching technology based on marker points, so that the guidance superimposed on the real equipment components by the augmented reality terminal and the guidance displayed on the digital twin interface by the multimedia terminal can be accurately mapped in spatial position.

[0036] Visualized force field: refers to a virtual effect presented in a visual manner in the augmented reality field of view, for prompting trainees to stay away from hazardous operation areas. Its rendering logic is: bind a spherical trigger on the hand model, and calculate in real time the shortest distance d between the center of the trigger and the surface of the hazardous operation component. Input d into the color mapping function: when d>30cm, a light green translucent halo is overlaid to indicate a safe area; when 10cm<d≤30cm, the color transitions linearly from yellow to orange with gradually increasing transparency, indicating a warning area; when d≤10cm, the color changes to a red warning color, and an increasingly thick resistive particle layer is overlaid, where particles diverge outward from the surface of the operation component, with a density proportional to 1 / d, forming a visual sense of resistance. When the system is connected to a force feedback glove, the resistive torque applied by the glove is proportional to 1 / d, providing a dissuasive effect in the tactile dimension.

[0037] On the basis of the above, the following embodiments are proposed: Embodiment 1: System hardware architecture; with reference to Figure 1 , the system of the present invention includes the following hardware components: a motion capture glove or depth camera 1, configured to collect the hand movement trajectory of a trainee; an eye tracker 2, configured to collect the eye gaze point of the trainee; pre-operation sensors 3, deployed on each operation component of the practical training apparatus 4, including but not limited to proximity sensors, touch sensors, and displacement sensors, configured to detect a pre-operation signal of the trainee's hand approaching or touching the operation component; a data processing host 5, connected to each of the above sensors and collection apparatuses, and running the core algorithm module of the present invention; an augmented reality display apparatus 6, such as AR glasses or an AR helmet, worn on the trainee's head, configured to overlay virtual operation guidance on the real practical training apparatus; a multimedia display apparatus 7, such as a liquid crystal display screen or a tablet computer, configured to display a digital twin model of the practical training apparatus and corresponding operation guidance; an optional force feedback glove 8, linked with the augmented reality display apparatus 6, configured to provide tactile resistance feedback.

[0038] The connection relationship of the above components is: the motion capture glove or depth camera 1, the eye tracker 2, and the pre-operation sensors 3 are all connected to the data acquisition interface of the data processing host 5 in a wired or wireless manner; the data processing host 5 is connected to the multimedia display apparatus 7 through a video output interface, and connected to the augmented reality display apparatus 6 through a wireless transmission module (such as Wi-Fi or Bluetooth); the force feedback glove 8 is wirelessly connected to the data processing host 5 and the augmented reality display apparatus 6.

[0039] The working signal flow of the system is: sensor data, multimodal fusion and intention reasoning performed by the data processing host 5, generation of guidance signals or dissuasion signals, separate transmission to the multimedia display apparatus 7 and the augmented reality display apparatus 6, the trainee observing the guidance and performing operations, and a new round of sensor data collection, forming a real-time closed loop.

[0040] Example 2: Construction of a spatiotemporal operation hypergraph (cardiopulmonary resuscitation training); This embodiment uses "cardiopulmonary resuscitation (CPR) training" as an example to explain in detail the construction process of the spatiotemporal operation hypergraph.

[0041] Step 1: Define a general operation primitive template. This template contains the following attribute slots: operation semantic tag (verb + object), operation attribute slot (standard duration, required tools, spatial coordinates, etc.), cognitive attribute slot (related knowledge points, common error patterns, cognitive load index, etc.), and error cost level slot (equipment damage index, personal injury index, teaching recovery cost index, and overall level).

[0042] Step Two: Obtain the CPR procedure and break it down into a multi-level procedure tree. The procedure includes: assessing environmental safety, assessing consciousness, calling for help, assessing breathing, chest compressions, opening the airway, artificial respiration, and AED defibrillation. Among these, "chest compressions" are further broken down into: identifying the compression position, overlapping hands, vertical compressions (5-6 cm depth), maintaining a rate of 100-120 compressions / minute, and a compression-to-relaxation ratio of 1:1.

[0043] Step 3: Instantiate the underlying operation steps as operation primitive nodes. Taking the "vertical compression" operation as an example, the attributes are filled as follows: operation name is "vertical compression", standard duration is 0.5-0.6 seconds per compression, cognitive load index is 0.8 (high), related knowledge point is "relationship between compression depth and cardiac output", common error patterns are "insufficient compression depth" or "excessive compression leading to rib fracture", equipment damage index is 1 (low risk of damage to CPR simulator), personal injury index is 4 (excessive compression can lead to rib fracture in real-world scenarios), teaching recovery cost index is 2, and the overall error cost level is 3.

[0044] Step 4: Identify the sequence, branching, and dependencies between operational primitives and construct hyperedges. For example, the "open airway" operation must be performed after the "assess breathing" operation, establishing a sequential hyperedge between them; "chest compressions" and "artificial respiration" form a 30:2 cyclic relationship, connected by a cyclic hyperedge. For convergence relationships that require multiple conditions to be met simultaneously, such as the "compression effective" decision node, which requires the simultaneous fulfillment of three conditions: "depth met," "frequency met," and "position correct," a hyperedge connects these three condition nodes and the decision node.

[0045] To further illustrate the cross-disciplinary versatility of this invention, a brief explanation of hypergraph construction is given using a motor control training exercise as an example; the training task is "star-delta start control of a three-phase asynchronous motor". The operation steps include: checking the power supply, closing the isolating switch, pressing the start button (star connection), setting the timer on the delay relay, switching to delta connection, observing the ammeter reading, and pressing the stop button.

[0046] Among the error cost levels of the "Close the disconnect switch" operation element, the personal injury index H is rated as Level 4 (high voltage operation), and the overall level is Level 4. The error cost level of "Press the start button" is Level 2 (low risk). When a trainee intends to skip "Close the disconnect switch" and directly "Press the start button," the system finds that the former's error cost level is Level 4 (reaching the safety threshold of Level 3), triggering a gentle dissuasion. When a trainee intends to switch to a delta connection prematurely when the star connection holding time is insufficient, the system finds that the error cost level of this operation node is Level 2 (below the safety threshold), allowing path reconstruction and pushing a supplementary prompt of "remaining time of the delay relay" on the multimedia terminal to support inquiry-based learning.

[0047] Through examples from two different professional fields, CPR (medical) and motor control (electromechanical), it can be demonstrated that the spatiotemporal operation hypergraph, error cost level, and hierarchical response mechanism of the present invention have cross-professional versatility.

[0048] Example 3: Construction of the manifold space; This embodiment details the process of constructing the manifold space.

[0049] A Graph Attention Network (GAT) is used as the embedding model. The input layer dimension of GAT is consistent with the initial feature vector dimension of the operational primitives. The initial feature vector is encoded by concatenating the operational attributes, cognitive attributes, and error cost levels of the operational primitives, and after normalization, it forms a 128-dimensional input vector. GAT contains two attention layers, each with 8 attention heads. The hidden layer dimension is 256 dimensions, and the output layer dimension is 128 dimensions, which is the dimension of the embedding vector.

[0050] Training data construction: Complete operation sequences (positive samples) from multiple experts were collected, while negative sample sequences were generated by random shuffling or introducing known error patterns. During training, for adjacent operation node pairs in the positive sample sequences, the contrastive learning loss function (InfoNCELoss) was used to shorten their distance in the embedding space; for negative sample pairs, their distance was increased. After training, each operation primitive corresponds to a unique embedding vector coordinate in the 128-dimensional operation manifold space.

[0051] Example 4: Temporal fusion network for intent prediction; This embodiment details the implementation method of predicting operation intent.

[0052] It should be noted that the multimodal data (hand movements, eye movements, and device pre-operation signals) in this embodiment are used for real-time intent prediction, which differs from the "do-see-understand" trimodal verification used for learning effect evaluation in Embodiment 7 below in terms of data source and purpose. The two are in different stages of the system closed loop.

[0053] The temporal fusion network employs a gated recurrent unit (GRU) network that includes an attention mechanism module. The input data includes time series data in three modalities: hand motion trajectory sequence (sampling rate 60Hz, including 3D position and 3D pose angle), eye-tracking gaze point thermal sequence (sampling rate 120Hz, encoded as a unique heat vector of the gaze region), and device pre-operation signal sequence (sampling rate 100Hz, a binary vector of the trigger states of each sensor).

[0054] In the data preprocessing stage, the data of the three modalities are timestamped and aligned. Based on the timestamp of the hand movement trajectory, the eye-tracking data and sensor data are linearly interpolated to form a unified three-channel time series with a sampling rate of 50Hz.

[0055] The attention mechanism module dynamically weights the features of each time step across the three modalities. The calculation method is as follows: for each time step, the features of the three modalities are concatenated and passed through a fully connected layer to calculate the attention score for each modality. This score is then normalized using Softmax to obtain the attention weight. When the hand movement amplitude is small, the attention weight of the eye-tracking modality automatically increases; when the pre-action sensor is triggered, the attention weight of the sensor modality increases sharply; when the hand performs a large-amplitude movement, the attention weight of the action modality dominates.

[0056] The weighted and fused feature sequence is input into a GRU layer (with a hidden layer dimension of 128). The hidden state of the last time step of the GRU is mapped to an operation intent vector through a fully connected layer, with the dimension consistent with the embedding vector dimension in the operation manifold space (128 dimensions). At the same time, the confidence score (0-1 scale) corresponding to this intent vector is output through a Softmax layer.

[0057] Example 5: Intent-Guidance Conflict Detection and Tiered Response; This embodiment details the intent-guided process for conflict detection and tiered response.

[0058] During real-time system operation, conflict detection is performed once every time window (e.g., 0.5 seconds). The currently predicted operation intent vector is searched in the operation manifold space for the k nearest neighbor operation primitive nodes (k=3) to form the intent operation sequence. The remaining m steps (m=5) in the current system preset guidance sequence that have not yet been executed are used as the preset guidance sequence.

[0059] Calculate the cognitive distance between two sequences in the operation manifold space. Specifically, construct a weighted complete bipartite graph, with the intention sequence nodes on the left and the guidance sequence nodes on the right. The edge weight is the Euclidean distance between the embedding vectors of the two nodes multiplied by the cognitive load coefficient of the next operation transition from that node. Solve for the minimum weighted matching using the Hungarian algorithm, and use the sum of the matching costs as the cognitive distance between the two sequences.

[0060] The cognitive distance threshold is dynamically set based on the overall difficulty of the training task and the teaching stage. In the initial stage of the task (when students are beginners), the threshold is set higher (e.g., 2.5 times the average cognitive distance between adjacent nodes in the current preset guidance sequence) to encourage exploration; in the advanced stage of the task (when students are proficient), the threshold is set lower (e.g., 1.5 times the average cognitive distance) to require more precise operation. Teachers or system administrators can manually fine-tune this threshold in the teaching configuration interface.

[0061] When the cognitive distance exceeds a preset threshold, an intent-guidance conflict is determined. Subsequently, the error cost level of each operation primitive in the intent operation sequence is queried from the spatiotemporal operation hypergraph. If the error cost level of all nodes is lower than the preset safety threshold (default setting is level 3), path reconstruction is initiated: the current preset guidance sequence is abandoned, and the end node of the intent operation sequence is used as the source node, and the original task target node is used as the sink node. An A* algorithm incorporating cognitive load weights is run on the spatiotemporal operation hypergraph to search for the shortest cognitive distance path. This path is then concatenated with the intent operation sequence to generate a new task sequence. Finally, the operation guidance corresponding to the new task sequence is rendered synchronously through multimedia display devices and augmented reality display devices.

[0062] If the error cost level of any node in the intended operation sequence reaches or exceeds a preset safety threshold, a soft dissuasion is initiated: instead of generating a new task path, the augmented reality display device renders a visual force field cues on the dangerous operation parts, and optionally applies increasing resistance through a force feedback glove, and plays a simulation animation of the possible consequences of the erroneous operation on a multimedia display device.

[0063] Example 6: Generation of a visualized force field; This embodiment details the rendering and generation process of the visualized force field.

[0064] A spherical trigger with a radius of 3cm is attached to the tip of the index finger of the trainee's hand model. Every frame (60fps), the shortest Euclidean distance *d* between the center of the trigger sphere and the surfaces of each operating component of the training equipment is calculated. The minimum value among all distances is taken as the current hazard approach distance.

[0065] The color mapping function is defined as follows: When d>30cm: do not render the force field, or render a light green translucent halo (RGBA: 0,255,0,30) to indicate a safe area; When 20cm<d≤30cm: the color transitions linearly from light green to yellow (RGBA interpolation), and the opacity increases linearly from 30 to 80; When 10cm<d≤20cm: the color transitions linearly from yellow to orange (RGBA interpolation), and the opacity increases from 80 to 150; When d≤10cm: the color abruptly changes from orange to red (RGBA: 255,0,0,200), and the rendering of the resistance particle layer is started.

[0066] The resistance particle layer is composed of a number of tiny particles emitted outward from the surface of the hazardous operating component. The particles are red, and the particle density N is proportional to 1 / d: N=floor(500 / d), where the unit of d is centimeter. The particles are emitted from the component surface in random directions, the initial velocity is proportional to (1 / d), and the life cycle is 0.5 seconds, forming a dynamic "resistance field" visual effect. When d approaches 0, the particle density tends to saturate, forming an opaque red barrier.

[0067] Meanwhile, if the system is equipped with a force feedback glove, the damping torque T of each finger joint of the glove is proportional to 1 / d: T=clamp(k / d, 0, Tmax), where k is the force feedback coefficient, and Tmax is the safe upper limit torque value.

[0068] Embodiment 7: Closed-loop logic of "Do-See-Understand" triangular verification; This embodiment takes the operation primitive of "tightening flange bolts" as an example to describe the closed-loop process of triangular verification in detail.

[0069] After the trainee completes the operation primitive, the system collects three types of data synchronously: "Do" — motion data: the wrist rotation angle sequence and applied torque sequence are collected through the motion capture glove, and compared with the standard motion template via Dynamic Time Warping (DTW) to calculate the motion standardization score (0-100 points), with the threshold set to 80 points; "See" — eye movement data: the fixation point distribution during the operation process is collected through an eye tracker, to determine whether the fixation points cover key visual areas such as the pressure gauge reading area and the flange alignment mark area, and calculate the fixation accuracy score (0-100 points), with the threshold set to 75 points; "Understand" — cognitive data: through a multimedia display device, immediately after the operation is completed, a multiple-choice question related to the principle of the current operation (for example, "Why do flange bolts need to be tightened diagonally?") is pushed, the trainee's answer result is collected, 100 points are awarded for a correct answer, and 0 points for a wrong answer.

[0070] The verification weights for the three-dimensional scoring are adaptively adjusted based on the type of the current operational primitive. The operational primitive type is automatically classified by the system based on its operational and cognitive attributes, according to the following rules: if the action complexity score is >0.6 and the cognitive load index is <0.4, it is classified as "fine-grained action type"; if the safety correlation score is >0.7, it is classified as "safety monitoring type"; if the number of associated knowledge points is ≥3 and the cognitive load index is ≥0.6, it is classified as "cognitive decision type". After classification, the weight configuration parameters pre-stored in the spatiotemporal operation hypergraph nodes (these parameters are configured by the teaching expert during hypergraph construction) are read, and corresponding weight values ​​are assigned to action standardization, gaze accuracy, and cognitive accuracy, respectively.

[0071] For example, "tightening flange bolts" belongs to the fine motor skill category, with a weight of 0.5 for action standardization, 0.3 for gaze accuracy, and 0.2 for cognitive accuracy. "Observing pressure gauge readings" belongs to the safety monitoring category, with a weight of 0.2 for action standardization, 0.6 for gaze accuracy, and 0.2 for cognitive accuracy. "Fault diagnosis decision-making" belongs to the cognitive decision-making category, with a weight of 0.1 for action standardization, 0.3 for gaze accuracy, and 0.6 for cognitive accuracy.

[0072] The weighted composite score is calculated as follows: w1 × action standardization + w2 × gaze accuracy + w3 × cognitive accuracy. When the weighted composite score is ≥80 points and each individual score meets its preset threshold, the operation node is marked as "deep mastery" in the learner's cognitive model, and the confidence level of the node is updated to high confidence (e.g., 0.95). Otherwise, it is marked as "mechanical imitation" or "not mastered," and the corresponding confidence level is reduced (e.g., 0.3 or 0.5).

[0073] The confidence level of each operational node in the learner's cognitive model directly affects the cognitive load coefficient of that node during the next task reconstruction. Specifically, in the next round of path planning, the cognitive load coefficient of that node = original cognitive load coefficient × (1 - confidence level). The higher the confidence level, the lower the cognitive load coefficient, and the more likely the reconstructed path is to quickly pass through that node; the lower the confidence level, the higher the cognitive load coefficient remains, and the system tends to arrange more reinforcement practice nodes near that node; thus forming a teaching closed loop of "evaluation - update - influence subsequent reconstruction - re-evaluation".

[0074] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for dynamically reconstructing practical training tasks based on operational logic graphs, characterized in that, Includes the following steps: S1. Construct a spatiotemporal operation hypergraph for the training task. The nodes of the spatiotemporal operation hypergraph represent operation primitives. Each operation primitive is configured with operation attributes, cognitive attributes and error cost levels. Hyperedges represent the order relationship, branch relationship and dependency relationship between operation primitives. S2. The learner's hand movement trajectory is collected by motion capture gloves or a depth camera, the eye gaze point is collected by an eye tracker, and the pre-operation signal of the equipment is collected by a pre-operation sensor deployed on the training equipment. The collected multimodal data is input into a temporal fusion network and mapped to the operation manifold space constructed based on the spatiotemporal operation hypergraph to obtain the learner's operation intention vector. S3. Calculate the cognitive distance between the intention operation sequence pointed to by the operation intention vector and the current preset guidance sequence in the operation manifold space; S4. If the cognitive distance exceeds a preset threshold, it is determined that an intention-guidance conflict has occurred, and the error cost level corresponding to each operation primitive in the intention operation sequence is queried from the spatiotemporal operation hypergraph. S5. If all the error cost levels found are lower than the preset safety threshold, abandon the current preset guidance sequence, take the intention operation sequence as the starting point, extract the shortest cognitive distance path that can reach the original task goal from the spatiotemporal operation hypergraph, splice them to generate a reconstructed task sequence, and render the first operation guidance and the second operation guidance corresponding to the reconstructed task sequence through a multimedia display device and an augmented reality display device respectively. Otherwise, the augmented reality display device is controlled to render and overlay a visual force field cues on the real device operation component corresponding to the intended operation sequence, which changes color or increases resistance as the learner's hand approaches, as a third operation guide. S6. Feed back the interactive response data under the action of the first operation guide, the second operation guide, or the third operation guide to the learner's cognitive model.

2. The method for dynamic reconstruction of practical training tasks based on operation logic graphs according to claim 1, characterized in that, The operational manifold space is pre-constructed by embedding the node and hyperedge relationships in the spatiotemporal operational hypergraph into a low-dimensional continuous vector space through a graph neural network.

3. The method for dynamic reconstruction of practical training tasks based on operational logic graphs according to claim 1, characterized in that, The error cost level is dynamically updated during system operation using a multi-dimensional assessment model that integrates the degree of equipment damage, the degree of personal injury, and the cost of teaching recovery.

4. The method for dynamic reconstruction of practical training tasks based on operation logic graphs according to claim 1, characterized in that, The cognitive distance is the distance between the current operation sequence embedding trajectory and the target task sequence embedding trajectory, calculated by a geodesic distance algorithm that incorporates cognitive load weights between operation primitives, wherein the cognitive load weights are pre-stored in the hyperedges of the spatiotemporal operation hypergraph.

5. The method for dynamic reconstruction of practical training tasks based on operation logic graphs according to claim 1, characterized in that, The temporal fusion network includes an attention mechanism module, which is used to asynchronously align the hand movement trajectory, eye gaze point and device pre-operation signal in the time dimension, and dynamically weight and fuse the features of each modality to output the operation intention vector and the corresponding confidence level.

6. The method for dynamic reconstruction of practical training tasks based on operation logic graphs according to claim 1, characterized in that, The visual force field cues work in conjunction with the force feedback gloves, which are linked to the augmented reality display device, to provide a sense of resistance that increases as the learner's hand approaches.

7. The method for dynamic reconstruction of practical training tasks based on operational logic graphs according to claim 1, characterized in that, The first and second operation instructions are generated based on the learner's current skill level using a progressive display strategy or a strategy of partially hiding key information, wherein the learner's current skill level is obtained from the learner's cognitive model.

8. The method and system for dynamic reconstruction of practical training tasks based on operation logic graphs according to claim 1, characterized in that, Also includes: The learner's operational motion data is collected through the motion capture glove or depth camera, eye movement data is collected through the eye tracker, and cognitive test data is pushed through the multimedia display device and the answer data is collected. Perform triangulation verification, which includes verification of action standardization, gaze accuracy, and cognitive accuracy. Only when the accuracy of the action, the accuracy of the gaze, and the accuracy of the cognition all meet their respective preset conditions will the corresponding operation node be marked as deep mastery in the learner's cognitive model. The updated learner cognitive model will be used as the input for the next cognitive distance calculation. The verification weights for action standardization, gaze accuracy, and cognitive accuracy are adaptively adjusted based on the type of the current operation primitive.

9. The method for dynamic reconstruction of practical training tasks based on operation logic graphs according to claim 1, characterized in that, The construction of the spatiotemporal operation hypergraph for the training task includes: Define a set of general operation primitive templates that are independent of professional fields. The general operation primitive templates include at least operation semantic tags, spatiotemporal attribute slots, and cognitive attribute slots. Obtain the operating procedures for the target professional training, and decompose the operating procedures into a multi-level operating step tree; Each level of operation step is instantiated into the general operation primitive, and the domain-specific operation attributes, cognitive attributes, and error cost level are populated. Identify the sequential, branching, and dependency relationships among the various operational primitives and construct hyperedges to form the spatiotemporal operational hypergraph.

10. A system for implementing the method for dynamic reconstruction of practical training tasks based on operational logic graphs as described in any one of claims 1-9, characterized in that, It includes a graph construction module, a multimodal acquisition hardware group, a state field and intent prediction module, a conflict resolution and reconstruction engine, an adaptive two-end guidance module, and a cross-modal verification module: The graph construction module is used to construct a spatiotemporal operation hypergraph, where nodes represent operation primitives and each operation primitive is configured with an error cost level. The multimodal acquisition hardware group includes motion capture gloves or depth cameras, eye trackers, and pre-operation sensors deployed on the training equipment. The state field and intent prediction module is connected to the multimodal acquisition hardware group and is used to receive the hand movement trajectory, eye gaze point and device pre-operation signal acquired by the multimodal acquisition hardware group, map the operation manifold space through the built-in temporal fusion network and output the operation intent vector. The input terminals of the conflict resolution and reconstruction engine are respectively connected to the graph construction module and the state field and intent prediction module, and are used for: Receive the operation intention vector, and calculate the cognitive distance between the intention operation sequence pointed to by the operation intention vector and the preset guidance sequence in the operation manifold space; If the cognitive distance exceeds a preset threshold, query the graph construction module for the error cost level of each operation primitive in the intention operation sequence; If all the error cost levels found are below the preset safety threshold, a reconstruction task sequence is generated and output to the adaptive dual-end guidance module. Otherwise, issue a gentle dissuasion command; The adaptive dual-end guidance module has its input end connected to the conflict resolution and reconstruction engine, and its output end connected to both a multimedia display device and an augmented reality display device, for the purpose of: According to the reconstruction task sequence, the multimedia display device and the augmented reality display device are controlled to render the first operation guide and the second operation guide respectively, and the first operation guide and the second operation guide are atomically consistent in terms of operation steps and spatial position. According to the gentle dissuasion instruction, the augmented reality display device is controlled to render and overlay a visual force field cues on the real device operation component corresponding to the intention operation sequence, which changes color or increases resistance as the learner's hand approaches. The state field and intention prediction module is also connected to the adaptive dual-end guidance module through a feedback link, which is used to receive the learner's interactive response data under each operation guidance and update the learner's cognitive model. The input end of the cross-modal verification module is connected to the adaptive dual-end guidance module and the multimodal acquisition hardware group, respectively, and the output end is connected to the state field and intent prediction module. The cross-modal verification module is used to collect action data, eye-tracking data, and cognitive test data, perform triangular consistency verification, and send the judgment result of deep mastery or mechanical imitation to the state field and intention prediction module to trigger the confidence update of the corresponding node in the learner's cognitive model. The updated learner cognitive model is used by the conflict resolution and reconstruction engine for the next cognitive distance calculation.

Images