A visual-motor delay-based operation proficiency recognition method in virtual practical training

Abstract

The application relates to the technical field of virtual simulation education, and particularly discloses a visual-motor delay-based operation proficiency recognition method in virtual practical training, which comprises the following steps: simulating a visual line landing point through ray detection and penetration depth checking, obtaining a corrected visual-motor delay through interframe interpolation and system error compensation, decoupling a mouse trajectory curvature into cognitive decision delay and motion execution delay, monitoring the number of pre-gaze scans and the total length of visual line confirmation to calculate a decision confidence index, combining a gaze mode entropy value to construct a multi-dimensional proficiency fingerprint; after self-recurrence individual baseline compensation and adaptive threshold calibration, identifying root cause bottlenecks and derived weak steps through a step-by-step directed dependence graph, calculating visual line forward offset to evaluate process coherence, outputting real-time visual prompts, fatigue early warning and group clustering analysis results, and dynamically calibrating thresholds according to cumulative data. The application realizes fine and adaptive quantitative evaluation of cognitive processing efficiency of students in a pure software mode.

Description

Technical Field

[0001] This invention relates to the field of virtual simulation and intelligent education evaluation technology, and in particular to a method for recognizing operational proficiency based on visual-motor delay in virtual training. Background Technology

[0002] With the widespread application of virtual reality and 3D simulation technologies in vocational education, virtual training systems developed based on 3D engines have become the core carrier for training manufacturing skills such as CNC machining and industrial robot assembly and disassembly. In training scenarios that heavily rely on hand-eye coordination, operational proficiency directly determines the quality and efficiency of operations. Existing virtual training systems mainly rely on three paths to evaluate trainees' proficiency: evaluation based on the time taken to complete the operation, but the total time is easily affected by factors such as pauses and equipment malfunctions, making it difficult to reflect the true level of proficiency; evaluation based on the sequence matching of operation steps, which uses algorithms such as dynamic time warping to determine process compliance, but cannot quantify the cognitive processing efficiency of each step; and post-operation evaluation based on the operation results, which can only provide general feedback and misses the opportunity for process intervention.

[0003] In recent years, some studies have introduced eye-tracking technology to analyze learners' visual attention, but these studies have significant limitations: they rely on dedicated head-mounted eye-tracking devices, which are costly and complex to deploy, making them difficult to popularize in large-scale teaching environments. The analytical dimensions are limited to the spatial distribution of gaze, such as fixation heatmaps and the frequency of visits to areas of interest, failing to address the crucial indicator of how long it takes for the eye to move after it lands on the target—a key indicator reflecting cognitive processing time. Current technologies fail to recognize the multiple systematic biases in measuring visual-motor delay in virtual training environments, including timestamp jitter caused by discrete sampling of rendering frames, pseudo-eye contact caused by bounding box inflation, coupling contamination of cognitive delay and mouse movement delay, dynamic drift of individual response baselines, and the cross-step transmission effect of cognitive bottlenecks. If these biases are not addressed, they will severely distort the results of proficiency evaluation.

[0004] Therefore, there is an urgent need for an operational proficiency identification method that does not rely on dedicated hardware, can accurately measure and analyze visual-motion delay, and effectively avoids systematic measurement biases. Summary of the Invention

[0005] The purpose of this invention is to address the technical problems of existing virtual training systems that rely on dedicated eye-tracking hardware, focus only on the spatial distribution of gaze while ignoring cognitive processing time, and fail to eliminate systematic measurement biases such as rendering frame sampling deviation, bounding box pseudo-contact, cognitive-motor delay coupling, and individual baseline drift, which make it impossible to conduct zero-hardware cost, refined, and process-oriented quantitative evaluation of trainees' operational proficiency. The invention proposes a method for identifying operational proficiency in virtual training based on vision-motor delay.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for recognizing operational proficiency based on view-motion delay in virtual training includes the following specific steps:

[0008] S1. During the operation of the virtual training scene, the pose data of the main virtual camera is continuously acquired at a preset sampling frequency. A ray is emitted from the virtual camera position along the direction of orientation. The collision point between the ray and the interactive object in the scene is obtained through ray detection as the simulated line of sight landing point, forming a line of sight landing point time sequence.

[0009] The gaze determination threshold for each target component is adaptively set based on the visual saliency characteristics of each target component. The visual saliency characteristics include the component's size proportion in the screen space, color contrast, and semantic relevance to the current operation step. For each target component, the penetration depth of the ray after entering the component's bounding box and reaching the component's actual geometric surface is calculated. Only when the penetration depth is within a preset effective range and the duration of the gaze point's continuous stay within the bounding box exceeds the adaptive gaze determination threshold, the collision point is determined as a valid gaze point. Based on the gaze point coordinates of adjacent frames in the gaze point time sequence, inter-frame interpolation is performed to estimate the continuous time when the gaze first enters the boundary of the target component's bounding box as the stable gaze start time.

[0010] Monitor and record the number of times the trainee's gaze scans before first entering the enclosure of the target component, as well as the total duration of gaze confirmation from first entering the enclosure to first performing the operation;

[0011] The system uses a global event listener to capture student operation events on each target component in real time, records the first operation time of each component, synchronously records the input response delay at the time of operation event, and deduplicates multiple operations on the same component.

[0012] The first operation time of each component is time-stamped and corrected with the corresponding stable gaze start time, and then paired. The first operation time is verified to be later than the stable gaze start time to filter out causal inverted data pairs. The initial visual-motion delay is calculated, and the initial visual-motion delay is compensated for system error according to the system frame rate and input response delay to obtain the corrected visual-motion delay.

[0013] Between the start of stable gaze and the first operation, the movement trajectory of the mouse cursor in the screen space is tracked. Based on the curvature characteristics of the mouse movement trajectory, the corrected visual-motor delay is decomposed into cognitive decision delay from the start of stable gaze to the start of mouse movement toward the target, and motor execution delay from the start of mouse movement to the execution of the operation.

[0014] The decision confidence index for this step is calculated based on the number of pre-fixation saccades and the total time for gaze confirmation for each component, and abnormal cognitive decision delay values ​​that exceed the reasonable cognitive processing range are filtered out.

[0015] S2. During an effective and stable gaze event, the frequency of gaze point transfer between different sub-regions within the bounding box of the target component is statistically analyzed, and the dispersion of the gaze spatial distribution is calculated as the gaze pattern entropy value. Low entropy value represents the anticipatory gaze pattern, and high entropy value represents the confirmatory gaze pattern.

[0016] A multidimensional proficiency fingerprint vector of trainees is constructed using the corrected visual-motor delay, decision confidence index, and gaze pattern entropy value for each operational step.

[0017] The cognitive decision delay sequence of each step in the trainee's historical training is called up, and the individual response baseline level of the current trainee in each step is estimated by an autoregressive prediction model. The cognitive decision delay value is converted into a standardized delay offset relative to the individual response baseline, and a standardized multidimensional proficiency fingerprint vector is constructed.

[0018] Collect corrected visual-motor delay data, decision confidence index, and gaze pattern entropy data of expert operators on the same training task to construct an expert reference fingerprint;

[0019] Based on the standardized delay offset statistical distribution of expert groups, the standardized delay offset statistical distribution of class students, or a combination of both, the delay threshold for distinguishing between proficient and unproficient students is adaptively calibrated.

[0020] The standardized delay offset of each step is compared with the delay threshold to generate a proficiency label for each step. The similarity distance between the trainee's standardized multidimensional proficiency fingerprint vector and the expert reference fingerprint is calculated, and the standardized multidimensional proficiency fingerprint vector with proficiency label, similarity distance and delay threshold are output.

[0021] S3. Construct a directed dependency graph between steps based on the logical sequence of each operation step in the training process. Map the standardized delay offset of each step to the corresponding node of the directed dependency graph. Analyze the propagation amplification coefficient of the standardized delay offset between adjacent steps. Identify the root cause bottleneck steps with abnormal delay offsets that propagate and amplify to subsequent steps, as well as the derived weak steps caused by the propagation of the root cause bottleneck from the preceding steps.

[0022] Based on the time interval between the stable gaze initiation time between adjacent operation steps and the time interval between the corresponding operation time, the trainee's visual gaze forward offset is calculated, and the trainee's operation process coherence evaluation result is generated based on the visual gaze forward offset.

[0023] Based on the standardized multidimensional proficiency fingerprint vector with proficiency tags, a personalized list of weaknesses is generated in the order of root bottleneck steps taking precedence over derived weak steps, and targeted training content for each weak step is matched from a preset training suggestion library.

[0024] Based on the changing trend of motion execution delay during the training process, when the motion execution delay value increases monotonically and the linearity index of the mouse movement trajectory decreases monotonically, a non-blocking fatigue warning prompt is output.

[0025] Within a preset time window after the trainee completes each operation step, real-time visual prompts are output in the scene user interface in a non-blocking manner based on the proficiency level of that step.

[0026] Cluster analysis was performed on the standardized multidimensional proficiency fingerprint vectors of all students in the class to divide them into different proficiency groups and automatically identify the common weak steps of each group.

[0027] The standardized multidimensional proficiency fingerprint vectors, comprehensive scores, weak point lists, gaze forward offsets, operation process coherence evaluation results, and similarity distances from each training session are stored in the database. This supports the visualization of historical proficiency change trends and group distribution characteristics. Furthermore, the delay threshold in S2 is dynamically updated based on accumulated training data to achieve adaptive calibration of the evaluation criteria.

[0028] The beneficial effects of this invention are as follows:

[0029] 1. By eliminating false line-of-sight contact of bounding boxes through penetration depth verification, stripping rendering sampling deviation through inter-frame interpolation, compensating for system frame rate and input response latency, and decoupling view-motion latency into cognitive decision latency and motion execution latency, eliminating interference from physical factors such as mouse movement distance, we can accurately extract pure indicators that reflect cognitive processing efficiency with zero hardware cost.

[0030] 2. A multidimensional proficiency fingerprint is constructed based on cognitive decision delay, decision confidence index, and gaze pattern entropy. Standardized offset is obtained through autoregressive baseline compensation. Root cause bottlenecks and derived weak steps are identified through inter-step directed dependency graphs. The process coherence is quantified by combining gaze look-ahead offset, breaking through the limitations of fixed thresholds and realizing process-level cognitive diagnosis.

[0031] 3. After the operation is completed, output non-blocking real-time visual prompts, output fatigue warnings based on the increase of motion execution delay and the decrease of mouse trajectory straightness; cluster analysis of all students in the class identifies common weak steps, and dynamically updates the delay threshold based on accumulated data, so that the evaluation criteria adapt to changes in the group level, providing data support for precision teaching. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method proposed in Embodiment 1 of the present invention;

[0033] Figure 2 This is a diagram of the architecture of the baseline prediction model in Embodiment 1 of the present invention;

[0034] Figure 3 This is a framework diagram of the system proposed in Embodiment 2 of the present invention. Detailed Implementation

[0035] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0036] Example 1

[0037] Please see the appendix Figure 1-2 A method for recognizing operational proficiency based on view-motion delay in virtual training includes the following specific steps:

[0038] S1. The gaze point is detected by ray detection, and the stable fixation start time is determined by penetration depth verification and inter-frame interpolation; the initial visual-motor delay is calculated and error compensation is performed to obtain the corrected visual-motor delay, which is decomposed into cognitive decision delay and motor execution delay; the number of pre-fixation saccades and the total duration of gaze confirmation are monitored, and the decision confidence index is calculated; specifically including:

[0039] S11. Simulation of gaze point and determination of the start time of stable fixation, specifically including:

[0040] S111. The system continuously acquires the pose data of the main virtual camera at a sampling frequency of not less than 30Hz, preferably 60Hz. At each sampling moment, the position vector and orientation vector of the virtual camera are recorded, where the position vector is represented in three-dimensional Cartesian coordinates and the orientation vector is represented in normalized three-dimensional orientation vectors.

[0041] The system emits a ray from the virtual camera's position along the directional direction. The maximum ray detection distance is set to 100m, which is sufficient to cover the spatial range of a typical virtual training scenario. The ray detection identifies the first collision point between the ray and an interactive object in the scene, and the 3D spatial coordinates of this collision point are used as the simulated gaze point at the current sampling moment. The system performs ray detection once per sampling frame, arranging the gaze point coordinates obtained from multiple consecutive sampling frames in chronological order to form a gaze point temporal sequence. Each element in the sequence corresponds to the gaze point coordinates of the corresponding sampling frame, i.e.: In the formula: This is the time sequence of the line of sight landing. For the first The coordinates of the gaze point in each sampled frame.

[0042] S112. For each operational target component in the scene, the system calculates its projected area ratio in the screen space, i.e.: , In the formula: This represents the percentage of the component's projected area in the screen space, with a value ranging from 0.01 to 0.5. This refers to the number of pixels covered in screen space by the component's bounding box after the model-view projection transformation. This represents the total number of pixels in the screen display area. For screen horizontal resolution, This refers to the screen's vertical resolution.

[0043] Simultaneously, the color difference between the main color of the component surface and the background color of the scene is calculated and quantified using the CIEDE2000 color difference formula, namely:

[0044] In the formula: This is the CIEDE2000 color difference value between the main color of the component surface and the background color, ranging from 0 to 100. This represents the lightness coordinates of the primary color of the component's surface in the CIELAB color space. Let a be the a-axis chromaticity coordinate of the primary color of the component surface in the CIELAB color space. Here are the b-axis chromaticity coordinates of the primary color of the component surface in the CIELAB color space. The value represents the lightness coordinates of the scene's background color in the CIELAB color space. Here are the chromaticity coordinates of the scene's background color along the a-axis in the CIELAB color space. Here are the b-axis chromaticity coordinates of the scene's background color in the CIELAB color space. The difference in lightness is calculated according to the CIEDE2000 standard algorithm. The chroma difference is calculated according to the CIEDE2000 standard algorithm. The hue difference is calculated according to the CIEDE2000 standard algorithm. The brightness weight adjustment factor is calculated according to the CIEDE2000 standard algorithm. This is the chroma weighting adjustment factor calculated according to the CIEDE2000 standard algorithm. The hue weighting adjustment factor is calculated according to the CIEDE2000 standard algorithm. The rotation terms are calculated according to the CIEDE2000 standard algorithm. This is the brightness parameter factor, which defaults to 1. This is the chroma parameter factor, which defaults to 1. This is the hue parameter factor, which defaults to 1.

[0045] Further calculations are made to determine the semantic similarity between the component name text and the current operation step description text, using a cosine similarity metric based on word vectors:

[0046] In the formula: The semantic similarity between the component name and the description text of the current operation step, with a value ranging from 0 to 1. The text vector obtained after word segmentation and average pooling of the component name text is in the th order. The components of the dimension, The text vector obtained after word segmentation and average pooling of the current operation step is at the [number]th position. The components of the dimension, The dimension of the pre-trained word vector model is 100 to 768, with 300 being the preferred value.

[0047] The three indicators mentioned above are weighted and fused to obtain a comprehensive visual saliency score, namely:

[0048] In the formula: The comprehensive visual saliency score ranges from 0 to 1. An adaptive mapping is used to determine the gaze determination threshold for each component based on the comprehensive visual saliency score. The mapping relationship is that the higher the score, the lower the threshold. The gaze determination threshold ranges from 0.1s to 1.0s, with a preferred value of 0.3s.

[0049] S113. The system calculates the penetration depth of the ray after it enters the bounding box of the target component and reaches the actual geometric surface of the component. This penetration depth is the distance from the intersection of the ray and the bounding box to the nearest intersection of the ray and the component mesh surface, i.e.: In the formula: The penetration depth is expressed in meters (m). The orientation vector of the main virtual camera, in a unitized form; The parameter value is the intersection point of the ray and the near surface of the component enclosure, which is the distance of the ray from the origin along the direction to the entry point of the enclosure, in meters; This is the parameter value for the intersection point of the ray and the actual mesh geometry of the component, i.e., the distance from the origin along the direction to the intersection point of the actual geometry, in meters. The lower limit of the preset effective range is 0m, the upper limit is 0.5m, and the preferred range is 0m to 0.1m. Only when the penetration depth is within the preset effective range does it indicate that the line of sight actually falls on the solid surface of the component and is not a pseudo-contact area caused by the expansion of the bounding box.

[0050] Simultaneously, the duration of the gaze's focus within the bounding box is monitored. Only when this duration exceeds the gaze determination threshold for that component is the collision point considered a valid gaze point. Through this dual filtering mechanism of penetration depth verification and gaze duration determination, false gaze contact events caused by the student's gaze accidentally sweeping over a component or the bounding box being larger than its actual geometry are effectively eliminated.

[0051] S114. Based on the established gaze-ground timing sequence, extract the gaze-ground coordinates between two adjacent sampling frames, and perform inter-frame interpolation between adjacent frames. The interpolation step size is set to 1ms to 10ms, with a preferred step size of 1ms. In the formula: For a moment The coordinates of the line of sight after interpolation. For the first The coordinates of the gaze point of the sampled frame. For the first The sampling time of the sampling frame.

[0052] By reconstructing the motion trajectory of the gaze point in the continuous time domain through inter-frame interpolation, the continuous moments when the gaze first enters the bounding box boundary of the target component are estimated. The estimation accuracy of these continuous moments reaches the millisievert level, effectively eliminating the systematic time bias caused by discrete sampling of rendering frames, which is typically between 16.7 ms and 33.3 ms. The estimated continuous moments when the gaze first enters the bounding box boundary are recorded as the stable gaze start time of the target component. This moment serves as the reference time point for subsequent calculations of the gaze-motion delay, ensuring the continuity and accuracy of the recording of the first gaze point time.

[0053] S12. Correction of visual-motor delay calculation and cognitive-motor delay decoupling, specifically including:

[0054] S121. The system registers a global operation event listener in the event bus of the virtual training scenario. This listener is attached to the input event system of the 3D engine in the form of a callback function, capturing in real time the click or drag events performed by the trainee on the target component using the mouse or gamepad. When an event is triggered, the unique identifier of the operated component is extracted from the event callback parameters, and the system's high-precision performance counter is called to obtain the current absolute timestamp, which is recorded as the first operation time of the component, with a timestamp precision of 0.001s.

[0055] The system synchronously records the system frame rate of the current rendering loop, which ranges from 1fps to 120fps, with a normal operating value of 60fps. It also records the input response latency, the time interval from the generation of a signal by the physical input device to the generation of a corresponding operation event by the system, ranging from 0ms to 50ms, with a preferred value of 8ms to 16ms. The system maintains a set of identifiers for operated components. When the same component is operated on multiple times, only the timestamp and event data corresponding to the first operation are retained. Subsequent duplicate operations are identified as duplicate data and discarded, ensuring that each target component corresponds to only one unique first operation moment.

[0056] S122. The system establishes a time mapping function between the gaze data pipeline and the operation event pipeline. This mapping function uses frame sequence numbers as a synchronization bridge and maintains a lookup table between gaze sampling frame sequence numbers and system absolute time. The stable gaze start time obtained in S114 is converted from the gaze sampling clock domain to the unified time reference of the operation event pipeline through a lookup table using frame sequence numbers, with a conversion accuracy of 0.001s. Based on the unique identifier of the target component, an association matching is performed, and the converted stable gaze start time is paired with the corresponding component's first operation time recorded in S121 to form a gaze operation data pair.

[0057] The data pair is subjected to temporal causality verification to determine whether the initial operation time is strictly later than the stable fixation start time. If the initial operation time is earlier than or equal to the stable fixation start time, it indicates that the operation occurred before the stable fixation, which does not conform to normal opto-motor causal logic, and the data pair is judged as invalid and filtered out. If the verification passes, the difference between the two times is calculated as the initial opto-motor delay, i.e.: In the formula: The initial visual-motion delay, in seconds, is the raw time difference without system error compensation. This is the moment of first operation; To stabilize the starting point of fixation.

[0058] S123. The system establishes a unified time reference mapping between the gaze sampling clock and the operation event clock, using the system monotonic clock as the root clock source, and normalizes the timing values ​​of the two heterogeneous clock domains to the same absolute time axis. A sampling clock deviation correction is performed at the stable gaze initiation moment. This correction addresses the periodic jitter of the gaze sampling frequency and frame scheduling deviation, namely: In the formula: This is the stable gaze start time after correction for sampling clock deviation. The sampling clock offset correction amount is calculated from the system frame rate, i.e.: In the formula: The system frame rate recorded in S121, in Hz; The value ranges from 0ms to 17ms at the normal operating frame rate (not less than 30fps).

[0059] Input response delay compensation is applied to the initial operation moment. This compensation amount is based on the input response delay value recorded in S121, and is subtracted from the initial operation moment to restore the physical moment when the student's actual operation intention was generated. In the formula: This is the first operation time after input response delay compensation. This is the input response delay compensation amount, with a value ranging from 8ms to 16ms.

[0060] The difference between the stable gaze start time after sampling clock deviation correction and the first operation time after response delay compensation is the corrected gaze-motion delay, i.e.: In the formula: To correct for visual-motor delay, the unit is seconds, with an accuracy of 0.001 seconds. This corrected visual-motor delay eliminates measurement errors introduced by clock domain differences and system response delays, and more realistically reflects the cognitive processing time of trainees from visual recognition to action execution.

[0061] S124. The system tracks the mouse cursor's movement trajectory in screen space at a frequency of no less than 60Hz, recording the cursor's position point sequence in a two-dimensional screen coordinate system, with the coordinate unit being pixels. After smoothing the position point sequence, the trajectory curvature of each point is calculated. The discrete curvature is calculated using the three-point approximation method, i.e.:

[0062] ;

[0063] In the formula: For the first The curvature of the trajectory at each location point, with the curvature value in units of one pixel and ranging from 0.01 to 0.1; , , The screen coordinates of the mouse cursor are the coordinates of three adjacent sampling times. The turning point of the mouse movement state is identified based on the curvature characteristics. This turning point is the critical point at which the mouse changes from a stationary or random wandering state to directional movement toward the target part. The curvature change threshold is set to 0.05 to 0.2, with a preferred value of 0.1.

[0064] Starting from the moment of stable fixation initiation and dividing by the curvature abrupt change point, the corrected visual-motor delay obtained in S123 is decomposed into two sub-intervals: the time difference between the moment of stable fixation initiation and the curvature abrupt change point is the cognitive decision delay, and the time difference between the curvature abrupt change point and the moment of the first operation is the motor execution delay, i.e.: , , In the formula: For cognitive decision delay, For motion execution delay, This represents the moment corresponding to the point where the curvature of the mouse cursor's movement trajectory abruptly changes.

[0065] The reasonable range for cognitive decision delay is set to 0.05s to 10s, with an optimal range of 0.1s to 5s. When the cognitive decision delay for a certain step is lower than the lower limit or higher than the upper limit, the value is marked as an outlier and removed from subsequent analysis. The proportion of outliers is controlled to be within 5% of the total sample size. When this proportion is exceeded, the system issues a data quality alarm and suspends the proficiency assessment for that step.

[0066] S13. Prefixation saccade monitoring and decision confidence index calculation, specifically including:

[0067] S131. Retrieve and extract the sampling moment when the line of sight first enters the bounding box of the target component of the current operation from the line of sight landing time sequence. This moment is defined as the first entry moment. Using this first entry moment as the time origin, backtrack to extract a time segment of a preset length. The preset de-jitter time window ranges from 0.05s to 0.5s, with a preferred value of 0.1s.

[0068] Within this retrospective time segment, the spatial containment relationship between the gaze point and the bounding box is detected frame by frame, identifying events where the gaze point leaves the bounding box and then re-enters it. To prevent false saccade counts caused by gaze jitter near the bounding box boundary, a preset departure threshold is set. This threshold represents the shortest duration for which the gaze point is continuously outside the bounding box, ranging from 0.02s to 0.2s, with a preferred value of 0.05s. Only when the gaze point leaves the bounding box for a continuous duration exceeding the preset departure threshold and then re-enters it is the event counted as a valid saccade. The entire retrospective time segment is traversed, and the number of valid saccades is accumulated to obtain the pre-fixation saccade count, which is an integer greater than or equal to zero.

[0069] S132. Record the time span from the first entry moment to the first operation moment of the target component. This time span is defined as the total duration of gaze confirmation, in seconds, with a value range of 0 to 30 seconds, and an optimal value range of 0.5 to 5 seconds. Normalize the number of pre-gaze saccades using a preset historical maximum number of saccades or a fixed upper limit as the normalization benchmark. Map the actual number of saccades to the range of 0 to 1. The fewer the number of saccades, the closer the normalized value is to 1, i.e.: In the formula: This is a normalized index of the number of prefixation saccades. The number of prefixation saccades. The baseline value for normalizing the number of saccades is set to 10; when hour, Cut off and take 0.

[0070] The total time for line-of-sight confirmation is normalized using a preset maximum confirmation time as the normalization benchmark. The actual time is mapped to the range of 0 to 1; the shorter the time, the closer the normalized value is to 1.

[0071] In the formula: To confirm the total duration of the gaze after normalization, To confirm the total duration of the visual confirmation, To confirm the normalization baseline value for duration, 10 seconds is used;

[0072] when hour, Cut off and take 0.

[0073] The normalized number of prefixation saccades and the total duration of gaze confirmation are weighted and combined, and the combined result is the decision confidence index. In the formula: The decision confidence index ranges from 0 to 1. The fewer the number of pre-fixation glances and the shorter the total time for eye contact confirmation, the closer the decision confidence index is to 1, indicating that the trainee is more decisive in identifying the target for the operation step and has more confidence in the operation decision.

[0074] S2. Calculate the gaze pattern entropy value, and construct an initial multidimensional proficiency fingerprint vector by combining cognitive decision delay, motor execution delay, and decision confidence index. After baseline compensation, obtain a standardized multidimensional proficiency fingerprint vector. Construct an expert reference fingerprint, adaptively calibrate the delay threshold, compare it with the standardized multidimensional proficiency fingerprint vector, generate proficiency labels, and calculate the similarity distance. Specifically, this includes:

[0075] S21. Calculation of gaze pattern entropy and construction of initial multidimensional proficiency fingerprint, specifically including:

[0076] S211. During a valid and stable gaze event, the system performs spatial discretization on the bounding box of each operational target component. Based on the component's geometric characteristics, the internal space of the bounding box is divided into several rectangular or regular geometric sub-regions, with the number of sub-regions ranging from 4 to 64, and a preferred value of 16. The division method uses a uniform grid based on the length, width, and height of the bounding box, and the size of each sub-region is not less than 0.005m.

[0077] The system tracks all coordinates falling within the bounding box of a component in the time sequence of gaze points, determining the sub-region number of each gaze point frame by frame to form a sub-region access sequence. When the gaze points of two adjacent sampling frames belong to different sub-regions, a cross-sub-region transfer event is recorded. The entire effective stable gaze event cycle is traversed, accumulating the number of transfers between each pair of sub-regions to obtain a gaze point transfer frequency matrix between different sub-regions. Each element of this matrix represents the frequency of gaze transfers between the corresponding two sub-regions, used for subsequent quantification of the discrete characteristics of the gaze spatial distribution.

[0078] S212. Based on the transfer frequency matrix obtained in S211, the system calculates the probability distribution of the gaze point in each sub-region. Specifically, this involves calculating the total frequency of visits to each sub-region, including the initial frequency of entering that sub-region and the sum of transfer frequencies from other sub-regions. The probability distribution of the gaze point in each sub-region is obtained by dividing the total frequency of each sub-region by the sum of the total frequencies of all sub-regions. In the formula: For the first Total frequency of visits to the viewpoints of each sub-region The total number of sub-regions. For the first The probability distribution of the point of view in each sub-region.

[0079] The gaze pattern entropy value is calculated based on this probability distribution. Using the definition of information entropy, the distribution probability of each sub-region is multiplied by the logarithm of that probability, and the sum is taken as the negative value. The result is the gaze pattern entropy value. In the formula: The entropy value represents the gaze pattern and ranges from 0.5 to 2.0. The closer the entropy value is to 0, the more concentrated the gaze is on a specific sub-region, which is an anticipatory gaze pattern; the higher the entropy value, the more dispersed the gaze distribution, which is a confirmatory gaze pattern.

[0080] The system extracts the cognitive decision-making delay, motor execution delay, decision confidence index, and gaze pattern entropy value of the current operation step, arranges and combines them in a fixed order to construct an initial multidimensional proficiency fingerprint vector containing four dimensions, namely:

[0081] In the formula: This is the initial multidimensional proficiency fingerprint vector. For cognitive decision delay, For motion execution delay, As a confidence index for decision-making.

[0082] S22. Individual response baseline prediction and standardized multidimensional proficiency fingerprint construction, specifically including:

[0083] S221. The system establishes a relational database to store evaluation data from each training session. The database uses either MySQL or SQLite. Before the first training session, a data table is initialized. Table fields include: unique student identifier, training project identifier, training timestamp, cognitive decision delay value for each operation step, motor execution delay value, decision confidence index, gaze pattern entropy value, standardized multidimensional proficiency fingerprint vector, comprehensive score, list of weaknesses, gaze forward offset, operation process coherence evaluation result, similarity distance, and calibration delay threshold. The unique student identifier and training project identifier are stored as strings, the training timestamp is stored as a date and time type with a precision of seconds, and all delay values ​​are stored as floating-point types with a precision of 0.001 seconds. After each training session, data is automatically written, appending all evaluation data generated from S1 to S3 as a complete record to the database.

[0084] The system uses the current student's unique identifier and the current training project identifier as joint query conditions to retrieve the student's past training operation records for the same training project from the database. It extracts the cognitive decision-making delay value and the motor execution delay value for each step, constructing a cognitive decision-making delay time series and a motor execution delay time series. Missing values ​​in the time series are imputed using linear interpolation of adjacent valid values, with an upper limit of three consecutive missing points in the interpolation window. Outliers are removed using a three-standard-deviation criterion; values ​​deviating more than three standard deviations from the series mean are removed, and gaps are filled using the same interpolation method. The preprocessed time series has a length of at least five data points.

[0085] S222. Establish baseline prediction models for the preprocessed time series, using an autoregressive structure. First, determine the model order based on the information criterion. ,Right now: In the formula: order The corresponding Akaike information criterion value, The maximum likelihood estimate of the model is selected; smallest order The preferred order for the model is between 1 and 3. After determining the order, the model is then compared to the model before the current time step. The model output is equal to the sum of the historical observations multiplied by their corresponding autoregressive coefficients plus a constant bias term, and the model parameters are estimated using the least squares method.

[0086] During training, the first 80% of the preprocessed time series data is used as the training set, and the last 20% as the validation set. After parameter estimation using the training set, the historical delay values ​​of the current step are input into the trained model, which outputs the baseline prediction values ​​of cognitive decision delay and motor execution delay for that step. This baseline represents the learner's inherent response level without additional cognitive load. Specifically:

[0087] ,

[0088] ,

[0089] In the formula: This represents the baseline predicted value for cognitive decision-making delay. For the first Cognitive decision-making delay observations at a historical moment; Delaying cognitive decision-making Autoregressive coefficient of order; This is the bias term for the cognitive decision delay constant; Baseline prediction values ​​for motion execution delay; For the first The motion execution delay observation value at each historical moment; For the motion execution delay Autoregressive coefficients; The motion is subjected to a delay constant bias term; This is the optimal model order determined by AIC in S222.

[0090] S223. Subtract the corresponding cognitive decision delay baseline from the cognitive decision delay value of each step in the current training, and then divide by the historical standard deviation of the cognitive decision delay for that step to obtain the standardized offset of cognitive decision delay, i.e.: In the formula: Standardized offset for cognitive decision delay This represents the cognitive decision delay value for the current practical training. This represents the baseline predicted value for cognitive decision delay. The historical standard deviation of cognitive decision-making delay.

[0091] Subtract the baseline motion execution delay for each step in the current training from the baseline motion execution delay for that step, and then divide by the historical standard deviation of the motion execution delay for that step to obtain the standardized offset of the motion execution delay, i.e.: In the formula: The offset is standardized to the motion execution delay. This represents the motion execution delay value during the current training exercise. The baseline prediction value for motion execution delay, The historical standard deviation of motion execution delay.

[0092] The two standardized offsets, along with the decision confidence index (normalized to the 0-1 range) and the gaze pattern entropy value, are arranged in a fixed-dimensional order to form a standardized multidimensional proficiency fingerprint vector, namely:

[0093] In the formula: To standardize the multidimensional proficiency fingerprint vector, This is a normalized decision confidence index. This represents the normalized gaze mode entropy value. The normalization of the gaze mode entropy value uses a Min-Max mapping, i.e.: In the formula: This represents the original gaze pattern entropy value. The preset minimum entropy value is set to 0.5; The preset maximum entropy value is set to 2.0.

[0094] Meanwhile, the cognitive decision delay value, motor execution delay value, decision confidence index, and gaze pattern entropy value of each step are arranged in a fixed-dimensional order to form the original multidimensional proficiency fingerprint vector, which is used to compare similarity with the expert reference fingerprint vector.

[0095] S23. Expert-referenced fingerprint construction and adaptive latency threshold calibration, specifically including:

[0096] S231. Organize no fewer than three expert operators to perform the same practical training task. Experts must have more than 5 years of relevant work experience and their operational assessment score must be no less than 95 points. The system records the experts' cognitive decision-making delay, motor execution delay, decision confidence index, and gaze pattern entropy value at each operational step in a manner completely consistent with that of the trainees. Calculate the arithmetic mean of the four indicators for the data from multiple experts at the same step to obtain the expert group mean for that step.

[0097] The mean cognitive decision delay, mean motor execution delay, mean decision confidence index, and mean gaze pattern entropy value for each step are arranged in a fixed-dimensional order to construct an expert reference fingerprint vector, namely:

[0098] In the formula: For experts to refer to fingerprint vectors, The average cognitive decision delay of the expert group. The mean execution delay of the expert group's movement. The average decision-making confidence index of the expert group. The mean entropy value of the gaze pattern of the expert group is given. The optimal number of experts is 5 to 10 to ensure statistical stability. For the cognitive decision delay value and motor execution delay value of the expert group at each step, the mean value of the expert group at that step is subtracted and divided by the standard deviation of the expert group at that step to obtain the standardized offset distribution of the cognitive decision delay and the standardized offset distribution of the motor execution delay of the expert group.

[0099] S232. The system provides three methods for calibrating the delay threshold: Method 1 calibrates the first delay threshold based on the statistical distribution of the standardized delay offset of the expert group. The calculation formula is the mean of the standardized delay offset of the expert group plus a coefficient multiplied by the standard deviation, i.e.: In the formula: The first delay threshold, Standardize the mean delay offset for the expert group. Standardize the standard deviation of the delay offset for the expert group. This is a preset coefficient, with a value range of 1.5 to 3.0, and a preferred value of 2.0.

[0100] Method 2 calibrates the second delay threshold based on the statistical distribution of standardized delay offsets among students in the class. The calculation formula is the class mean plus the class standard deviation, i.e.: In the formula: The second delay threshold, Standardize the mean delay offset for each student in the class. Standardize the standard deviation of the delay offset for each student in the class.

[0101] Method three is a hybrid calibration, where the smaller of the first delay threshold and the second delay threshold is taken as the final hybrid delay threshold, i.e.: In the formula: This is the mixed latency threshold. The system defaults to using the mixed calibration method, and automatically switches to the corresponding single method when only one type of data is available.

[0102] S233. Extract the standardized offset of cognitive decision delay for each step in the standardized multidimensional proficiency fingerprint vector, and compare it step by step with the calibrated delay threshold. If the standardized offset of cognitive decision delay for a step is less than or equal to the delay threshold, the step is marked as proficient. If the standardized offset of cognitive decision delay for a step is greater than the delay threshold, the step is marked as unproficient. This determination process traverses all operation steps in the training process, generates a complete set of step-level proficiency tags, and embeds them into the corresponding metadata fields of the standardized multidimensional proficiency fingerprint vector to form a standardized multidimensional proficiency fingerprint vector with proficiency tags.

[0103] S234. Calculate the similarity distance between the trainee's original multidimensional proficiency fingerprint vector and the expert reference fingerprint vector. The Euclidean distance is used, calculated by squaring the differences in corresponding dimensions of the two vectors, summing them, and then taking the square root. In the formula: The similarity distance ranges from 0 to positive infinity. The closer it is to 0, the closer the student's operation mode is to the expert level. The original multidimensional proficiency fingerprint vector is the first... The dimensions of the component are consistent with those of the expert reference fingerprint vector; For experts to refer to the fingerprint vector, the first Multidimensional components. The system outputs a standardized multidimensional proficiency fingerprint vector with proficiency tags, similarity distance values, and the currently used calibration delay threshold.

[0104] S3. Identify root bottlenecks and derived weak steps based on step dependencies, calculate gaze-forward offset, and generate operational process coherence evaluation results; generate a personalized list of weak points, output real-time visual cues and fatigue warnings, perform cluster analysis on standardized multidimensional proficiency fingerprint vectors, and adaptively calibrate the delay threshold; specifically including:

[0105] S31. Construction of directed dependency graph between steps and identification of root cause bottlenecks, specifically including:

[0106] S311. The system extracts the logical dependencies between operation steps from the standard operation process document or pre-configured logical relationship table of the training task. This dependency represents the sequential constraints of step execution; for example, if a preceding step is not completed, a subsequent step cannot or should not be executed. Using each operation step as a node in a directed dependency graph, and the logical dependencies between steps as directed edges, a directed dependency graph is constructed between steps. The direction of the directed edges strictly points from the preceding step to the following step, forming a topological sequence from the start step to the end step of the training. The total number of nodes equals the total number of operation steps in the training process, and the number of directed edges depends on the degree of coupling between steps, typically 1.0 to 2.0 times the total number of steps. This directed dependency graph is stored using an adjacency matrix or adjacency list, supporting fast lookup of the preceding and following node sets for any node.

[0107] S312. The system obtains the normalized delay offset of each step from S223 and maps this value to the attribute value field of the corresponding node in the directed dependency graph. It iterates through all directed edges in the directed dependency graph, and for each pair of adjacent steps connected by a directed edge, extracts the normalized delay offset of the preceding step node and the normalized delay offset of the succeeding step node. The ratio of the normalized delay offset of the succeeding step to the normalized delay offset of the preceding step is calculated as the propagation amplification factor for that pair of adjacent steps, i.e.: In the formula: Preliminary steps The standardized delay offset, For subsequent steps The standardized delay offset, For the preceding steps Pointing to the successor step The propagation amplification factor, ranging from 1.2 to 2.0, characterizes the degree to which delay anomalies in preceding steps are propagated and amplified in subsequent steps. When the propagation amplification factor is greater than 1, it indicates that the delay anomaly in the subsequent step exceeds that of the preceding step, indicating an amplification effect. The system calculates and stores the propagation amplification factor for all adjacent step pairs edge by edge.

[0108] S313. The system traverses each node in the directed dependency graph in topological order, performing root cause bottleneck determination for each node. The determination criteria are: the normalized delay offset of the node exceeds a preset offset threshold, and the normalized delay offset of at least one subsequent step it points to also exceeds the preset offset threshold; the propagation amplification factor between the node and its pointed-to subsequent step is greater than a preset amplification factor threshold; and there is no step marked as a root cause bottleneck in the node's preceding step set. The preset offset threshold ranges from 1.0 to 3.0, with a preferred value of 2.5; this threshold is independent of the delay threshold in S232 and is used for bottleneck propagation determination between steps. The preset amplification factor threshold ranges from 1.0 to 2.0, with a preferred value of 1.2. When all four conditions are met simultaneously, the system marks the node as a root cause bottleneck step, indicating that this step is the original source of the abnormal amplification of delays in subsequent steps.

[0109] S314. The system continues to traverse the unmarked nodes in the directed dependency graph. When the normalized delay offset of a node exceeds a preset offset threshold, and the normalized delay offsets of all its preceding steps do not exceed the preset offset threshold, the node is directly marked as the root cause bottleneck step, indicating that the step is an independently occurring abnormal node.

[0110] When the normalized delay offset of a node exceeds a preset offset threshold, and its preceding steps contain at least one step that has been marked as a root cause bottleneck, and the node does not meet the root cause bottleneck determination condition in S313, the system marks the node as a derived weak step. The preset offset threshold ranges from 1.0 to 3.0, with a preferred value of 2.0. This threshold is set independently to adapt to independent bottleneck determination scenarios without preceding anomalies. Through the above dual determination mechanism, it is ensured that each abnormal node in the directed dependency graph is assigned only a unique bottleneck type label.

[0111] S32. Evaluation of the consistency of the line-of-sight offset calculation and operation process, specifically including:

[0112] S321. The system extracts data from adjacent pairs of operation steps sequentially according to the standard execution order of each operation step in the training process. For each pair of adjacent steps, the system reads the stable gaze start time and the first operation time from the records of the preceding and subsequent steps, respectively. The difference between the stable gaze start times of two adjacent steps is calculated, and this difference is defined as the gaze transfer time interval, i.e.: In the formula: For the first The time interval between gaze shifts between adjacent steps, expressed in seconds, represents the time required for the trainee to shift their gaze from the target of the preceding step to the target of the following step. For the first The stable gaze initiation time of each step; For the first The starting point of stable gaze in each step.

[0113] The difference between the first operation times of two adjacent steps is calculated synchronously. This difference is defined as the operation transition time interval, i.e.: In the formula: For the first The time interval for operation transition between adjacent steps, in seconds, represents the time interval between when a student completes the previous step and starts the subsequent step. For the first The first operation moment of each step, For the first The first operation time of each step. Traverse all adjacent step pairs in the training process, calculate and store the above two time intervals for each pair, forming the line-of-sight transfer time interval sequence and the operation transfer time interval sequence.

[0114] S322. The system calculates the difference between the operation transfer time interval and the line-of-sight transfer time interval for each pair of adjacent steps. This difference is defined as the line-of-sight look-ahead offset, i.e.: In the formula: The gaze-forward offset ranges from -1s to +3s. A positive gaze-forward offset indicates that the learner's gaze has already begun to focus on the target of the subsequent step before completing the first operation of the preceding step. A larger positive value indicates a higher level of foresight and stronger operational planning. A negative gaze-forward offset indicates that the learner only begins searching for the target of the subsequent step after completing the preceding step. A smaller negative value indicates a higher degree of lag and poorer continuity of the operational process. The system calculates and stores the gaze-forward offset for each pair of adjacent steps.

[0115] S323. The system iterates through the set of look-ahead offsets for all adjacent step pairs in the training process, counts the number of positive offsets, and calculates their proportion of the total number of offsets. This proportion is defined as the percentage of positive offsets, i.e.: In the formula: This represents the percentage of positive offset values, ranging from 0 to 1. This represents the number of adjacent step pairs where the look-ahead offset is positive. This represents the total number of adjacent step pairs in the training process.

[0116] Simultaneously, the arithmetic mean of all positive offsets is calculated, and this mean is defined as the average of positive offsets, i.e.: In the formula: This represents the average positive offset, expressed in seconds (s), and ranges from 0 to 5 seconds. For the first A positive forward-looking offset.

[0117] The preset proportion threshold ranges from 0.5 to 0.8, with a preferred value of 0.6. The preset look-ahead threshold ranges from 0.1s to 1.0s, with a preferred value of 0.3s. If the proportion of positive offsets exceeds the preset proportion threshold and the average value of positive offsets exceeds the preset look-ahead threshold, the system generates a positive operational process coherence evaluation result, indicating that the trainee possesses good operational foresight and process coherence. Conversely, the system generates a negative operational process coherence evaluation result, indicating that the trainee's operational process coherence is insufficient and requires enhanced training in step connection.

[0118] S33. Personalized feedback generation and group cognitive pattern mining, specifically including:

[0119] S331. The system filters all operation steps marked as unfamiliar based on standardized multidimensional proficiency fingerprint vectors with proficiency labels. A personalized list of weaknesses is generated according to a sorting rule prioritizing root bottleneck steps over derived weak steps, with root bottleneck steps at the front of the list and derived weak steps at the back, sorted in descending order of standardized delay offset within the same priority level. Targeted training content matching each type of weak step is retrieved from a preset training suggestion library, which contains a mapping table between step types and training content, with no fewer than 50 mapping entries. Targeted training content covers categories such as component position memory enhancement, operation sequence rehabilitation, and hand-eye coordination. The personalized list of weaknesses includes the step name, bottleneck type label, standardized delay offset value, and corresponding training content, with a maximum of 80% of the total number of training steps in the list.

[0120] S332. Within a preset time window after the trainee completes each operation step, the system outputs real-time visual prompts on the scene user interface based on the trainee's proficiency level for that step. The preset time window ranges from 0.3s to 2.0s, with a preferred value of 0.5s. The real-time visual prompts are presented in a non-interrupting manner, including color changes of the step list items or text label overlays. The prompt duration ranges from 1.0s to 3.0s, with a preferred value of 2.0s.

[0121] The system continuously monitors the trend of motion execution latency during training. When the motion execution latency shows a monotonically increasing trend for at least three consecutive steps, and the mouse movement trajectory straightness index shows a monotonically decreasing trend with its latest value below 0.3, a non-blocking fatigue warning is triggered. The mouse movement trajectory straightness index is defined as the ratio of the Euclidean distance between the starting and ending points of the trajectory to the actual path length, i.e.:

[0122] In the formula: This is the straightness index of the mouse movement trajectory, with a value ranging from 0 to 1. The closer it is to 1, the closer the trajectory is to a straight line. The starting screen coordinates of the mouse cursor movement path; The final screen coordinates are used; the denominator is the actual path length of the trajectory. The fatigue threshold is set to 0.3. Fatigue warnings are displayed as flashing icons in the corner of the interface or as text banners, without interrupting the student's current operation.

[0123] S333. The system extracts the standardized multidimensional proficiency fingerprint vectors of all students in the class and uses the K-means clustering algorithm to divide the group. The number of clusters K is set to 3, corresponding to the high proficiency group, the medium proficiency group, and the group that needs improvement, respectively. During the clustering process, the Euclidean distance from each student's standardized multidimensional proficiency fingerprint vector to each centroid vector is calculated:

[0124] In the formula: For the first The students reached the first The distance between the centroids, For the first The standardized multidimensional proficiency fingerprint vector of each student dimensional components, For the first The centroid vector of the nth centroid vector Dimensional components. Each student is assigned to the category of the nearest centroid, and then the centroid of each category is updated to the arithmetic mean of all sample vectors within that category: In the formula: For the first A collection of student indexes in each category; For the first The number of students in each category. Iteratively execute the above assignment and update steps until the change in centroid is less than the convergence threshold. Cluster initialization uses the K-means++ strategy, with a maximum number of iterations ranging from 100 to 500, preferably 300. The convergence threshold ranges from 0.0001 to 0.001, preferably 0.0001.

[0125] After clustering, the percentage of trainees marked as unfamiliar with each operation step is calculated for each group. When the percentage of trainees unfamiliar with a certain step exceeds a preset threshold, that step is identified as a common weak step for that group; where: In the formula: This represents the percentage of trainees in the group who are marked as unfamiliar with a particular step. This represents the number of trainees in the group who were marked as unfamiliar with this step. This represents the total number of trainees in the group. The preset ratio threshold ranges from 0.5 to 0.8, with an optimal value of 0.6. The results of identifying common weak points are output in groups for teachers to provide targeted group instruction.

[0126] S334. The system stores standardized multidimensional proficiency fingerprint vectors, comprehensive scores, weakness lists, gaze-forward offsets, operational process coherence evaluation results, and similarity distances from each training session in a database. The database adopts a relational architecture or a lightweight embedded architecture, with records written in batches of one record after each training session. The system supports visualization of historical proficiency trends, displaying the evolution of the comprehensive score with the number of training sessions using a line graph, and showing the gradual improvement of the delay offset for each weak step using a bar chart.

[0127] The system dynamically updates the delay threshold based on the accumulated training data, with an update cycle of 1 to 3 months, the preferred value being 1 month. When the accumulated number of trainees in a single cycle is no less than 50, the system triggers a recalculation of the class mean and standard deviation and updates the threshold, or an update is forcibly triggered once every quarter.

[0128] Example 2

[0129] Please see the appendix Figure 3 A vision-motion delay-based operation proficiency recognition system for virtual training, used to implement a vision-motion delay-based operation proficiency recognition method in virtual training, comprising:

[0130] The gaze acquisition module is used to obtain the simulated gaze point through ray detection, and determine the stable gaze start time through penetration depth verification and inter-frame interpolation.

[0131] The delay calculation module is used to calculate the initial visual-motor delay and obtain the corrected visual-motor delay after error compensation. The corrected visual-motor delay is decomposed into cognitive decision delay and motor execution delay. It monitors the number of pre-fixation saccades and the total duration of gaze confirmation, and calculates the decision confidence index.

[0132] The fingerprint construction and calibration module is used to calculate the gaze pattern entropy value, combine cognitive decision delay, motor execution delay and decision confidence index to construct an initial multidimensional proficiency fingerprint vector, obtain a standardized multidimensional proficiency fingerprint vector through baseline compensation, construct an expert reference fingerprint, adaptively calibrate the delay threshold, compare it with the standardized multidimensional proficiency fingerprint vector, generate proficiency labels and calculate similarity distance.

[0133] The process evaluation and feedback module is used to identify root bottlenecks and derived weak steps based on step dependencies, calculate the look-ahead offset, generate operation process coherence evaluation results, generate a personalized list of weak points, output real-time visual cues and fatigue warning prompts, perform cluster analysis on standardized multidimensional proficiency fingerprint vectors, and adaptively calibrate the delay threshold.

[0134] Example 3

[0135] To verify the effectiveness of the operation proficiency recognition method and system based on vision-motion delay in virtual training described in this invention, a comparative experiment was conducted with 40 students majoring in manufacturing.

[0136] I. Experimental Environment and System Deployment

[0137] The experiment was conducted on a standard computer equipped with an NVIDIA GTX 1660 graphics card and a 144Hz monitor, running a virtual training system for disassembling the J1-axis motor of an industrial robot developed using the Unity 3D engine. The system deployment was as described in Example 2: an operation proficiency recognition system, including a gaze acquisition module, a delay calculation module, a fingerprint construction and calibration module, and a process evaluation and feedback module.

[0138] The gaze acquisition module acquires the pose data of the main virtual camera at a sampling frequency of 60 frames per second and emits rays to detect gaze points. For each target component, the visual saliency submodule calculates its projected area ratio in screen space, the difference between the surface primary color and the background color, and the semantic similarity between the component name and the current operation step description text. These are weighted and fused to obtain a comprehensive visual saliency score, which is then mapped to determine the gaze determination threshold. The effective gaze determination submodule calculates the penetration depth of the ray after it enters the component's bounding box and reaches the actual geometric surface. A gaze point is determined to be effective only if the penetration depth is within a preset effective range and the duration of its stay exceeds the adaptive gaze determination threshold. The stable gaze determination submodule performs linear interpolation based on the gaze point coordinates of adjacent frames to estimate the consecutive moments when the gaze first enters the bounding box boundary as the stable gaze start time.

[0139] The latency calculation module captures operation events through a global event listener, records the initial operation time, and simultaneously records the system frame rate and input response latency. The time synchronization pairing submodule uses frame numbers as a bridge to convert the stable gaze start time to a unified time base, pairs it with the initial operation time, and filters out causal inversion data pairs. The error compensation submodule corrects the sampling clock deviation for the stable gaze start time and compensates for the response latency of the initial operation time, obtaining the corrected visual-motion latency. The latency decoupling submodule tracks the mouse cursor's movement trajectory in screen space, decomposes the corrected visual-motion latency into cognitive decision latency and motion execution latency based on its curvature characteristics, and marks cognitive decision latency exceeding a preset reasonable range as outliers for removal.

[0140] The decision confidence analysis submodule extracts the first entry time, retrospectively counts the number of pre-gaze scans, records the total duration of gaze confirmation from the first entry time to the first operation time, and normalizes and weights the two indicators to obtain the decision confidence index.

[0141] The gaze pattern analysis submodule of the fingerprint construction and calibration module divides the bounding box region according to the geometric shape of the component during effective and stable gaze events, counts the frequency of gaze point shifts, and calculates the gaze pattern entropy value. The initial fingerprint construction submodule constructs an initial multidimensional proficiency fingerprint vector using cognitive decision delay, motor execution delay, decision confidence index, and gaze pattern entropy value. The baseline compensation submodule calls up the student's past operation records, extracts cognitive decision delay value and motor execution delay value according to the step dimension to form a time series, predicts the individual response baseline using an autoregressive model, subtracts the baseline from the current delay value and divides by the historical standard deviation to obtain the standardized offset, and combines them to form a standardized multidimensional proficiency fingerprint vector.

[0142] The expert reference construction submodule collects data from five expert operators with over five years of work experience and an assessment score of at least 95 points to construct expert reference fingerprint vectors. The threshold calibration submodule calibrates delay thresholds based on the statistical distribution of standardized delay offsets between the expert group and the class of students. The proficiency labeling submodule compares the standardized delay offsets of each step with the delay thresholds to generate proficiency labels. The similarity calculation submodule calculates the Euclidean distance between the student and the expert reference fingerprint vectors.

[0143] The dependency graph construction submodule of the process evaluation and feedback module constructs a directed dependency graph between steps based on the logical sequence dependencies of the 13 operation steps. The propagation amplification calculation submodule maps the standardized delay offset of each step to the corresponding node and calculates the propagation amplification coefficient of adjacent step pairs. The root cause identification submodule identifies root cause bottleneck steps and derived weak steps based on preset offset thresholds and amplification coefficient thresholds. The look-ahead offset calculation submodule obtains the stable gaze start time and the first operation time of adjacent step pairs, and calculates the difference between the gaze transfer time interval and the operation transfer time interval as the gaze look-ahead offset. The coherence evaluation submodule statistically analyzes the proportion and average of positive offsets to generate the operation process coherence evaluation results.

[0144] II. Experimental Grouping and Tasks

[0145] The participants were divided into two groups: an experimental group of 20 people, who received real-time evaluation and feedback using the method of this invention; and a control group of 20 people, who received a traditional method (scoring based solely on completion time and final result). There were no significant differences between the two groups in terms of age, gender ratio, or prior course performance (p>0.05). The experimental task was a virtual training exercise on disassembling the J1 axis motor of an industrial robot, consisting of 13 operational steps, with each training session lasting no more than 20 minutes. Participants in the experimental group received real-time proficiency markers and weaknesses indicators for each step after each operation; participants in the control group only received a final score at the end of the training session.

[0146] III. Experimental Results

[0147] After 5 training sessions, the changes in the overall proficiency scores (out of 100) of the two groups of trainees are shown in Table 1 below:

[0148] Table 1: Comparison of Comprehensive Scores of Proficiency in 5 Practical Training Sessions Between the Two Groups of Trainees

[0149]

[0150] The experimental group's average proficiency score after the 5th training session (84.3 points) was significantly higher than that of the control group (66.9 points), with an improvement of 17.4 points. The experimental group improved by 32.0 points from the 1st to the 5th training session, while the control group only improved by 13.8 points, meaning the experimental group's improvement was 2.3 times that of the control group.

[0151] The average visual-motion delay variation (in seconds) for each step in the 5 training sessions for the 20 trainees in the experimental group is shown in Table 2 below:

[0152] Table 2: Average visual-motion delay variation in 5 training sessions for each operation step in the experimental group

[0153]

[0154] The visual-motor delay of the experimental group students showed a continuous downward trend in all steps, and the improvement rate of each step in 5 training sessions ranged from 61.6% to 68.9%, with an average improvement rate of 66.4%.

[0155] The weakness identification results of this invention are compared and verified with the results of teacher manual evaluation, as shown in Table 3 below:

[0156] Table 3: Comparison and Verification Table of Weakness Identification Results and Teacher Manual Assessment

[0157]

[0158] The accuracy rate of this invention in identifying trainees' weak steps was 89.6% in the first training session and increased to 94.4% in the fifth training session, with an average accuracy rate of 91.5%.

[0159] A correlation analysis was performed on all data from 5 training sessions for 40 trainees (a total of 200 data samples). Pearson correlation coefficients were calculated with visual-motor delay on the horizontal axis and the proficiency level (1-5) assessed by the instructors on the vertical axis. The results are shown in Table 4 below:

[0160] Table 4: Statistical Analysis of the Correlation between Visual-Motion Delay and Proficiency Level

[0161]

[0162] There was a highly significant negative correlation between visual-motor delay and proficiency level (r=-0.82, p<0.001).

[0163] IV. Results Analysis

[0164] Experimental data show that the operation proficiency recognition method based on visual-motor delay proposed in this invention can effectively distinguish the proficiency level of trainees in different operation steps. By decoupling the corrected visual-motor delay into cognitive decision-making delay and motor execution delay, the system can accurately extract pure indicators reflecting cognitive processing efficiency and eliminate interference from physical factors such as mouse movement distance. The introduction of gaze pattern entropy and decision confidence index enables the system to distinguish between anticipatory gaze and confirmatory gaze, further enhancing the evaluation dimensions.

[0165] At the step-level diagnostic level, the dependency graph construction submodule and the root cause identification submodule successfully identified root cause bottleneck steps such as encoder connector removal, as well as the derived weak steps caused by these steps, enabling teachers to focus on the real teaching intervention points. The calculation of gaze deviation showed that, with the increase in the number of training sessions, the proportion of positive gaze deviation in the experimental group increased from 32% in the first session to 71% in the fifth session. The evaluation result of operational process coherence changed from negative to positive, indicating that the trainees gradually formed an expert cognitive mode of simultaneous operation and gaze.

[0166] Real-time, non-interrupted visual cues and fatigue warning mechanisms more than doubled the learning speed of trainees, with a weak step identification accuracy exceeding 90%. The high correlation between visual-motion delay and teacher subjective evaluation (r = -0.82) further validates the effectiveness of this indicator as a quantitative evaluation standard for proficiency. This invention provides a large-scale, automated proficiency evaluation solution for virtual training teaching that requires no dedicated hardware.

[0167] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity.

[0168] This invention is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this specification. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for recognizing operational proficiency based on view-motion delay in virtual training, characterized in that, include: The starting time of stable fixation is determined by detecting the point of gaze using ray detection, and then verifying the penetration depth and inter-frame interpolation. After calculating the initial visual-motor delay and performing error compensation, the corrected visual-motor delay is obtained. The corrected visual-motor delay is then decomposed into cognitive decision-making delay and motor execution delay. Monitor the number of pre-fixation saccades and the total duration of gaze confirmation, and calculate the decision confidence index; Calculate the gaze pattern entropy value, and construct an initial multidimensional proficiency fingerprint vector by combining cognitive decision delay, motor execution delay and decision confidence index. After baseline compensation, a standardized multidimensional proficiency fingerprint vector is obtained. Construct an expert reference fingerprint, adaptively calibrate the delay threshold, compare it with the standardized multidimensional proficiency fingerprint vector, generate proficiency labels and calculate similarity distance; Identify root bottlenecks and derived weak steps based on step dependencies, calculate the look-ahead offset, and generate an evaluation result of the operational process coherence. Generate a personalized list of weaknesses, output real-time visual cues and fatigue warnings, perform cluster analysis on standardized multidimensional proficiency fingerprint vectors, and adaptively calibrate the delay threshold.

2. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 1, characterized in that, The process of detecting the gaze point using ray detection, and determining the stable gaze initiation time through penetration depth verification and inter-frame interpolation, specifically includes: The pose data of the main virtual camera is continuously acquired at a preset sampling frequency. A ray is emitted from the camera position along the direction of orientation. The collision point between the ray and the interactive object in the scene is obtained through ray detection as the simulated gaze point, forming a gaze point timing sequence. Calculate the percentage of the projected area of ​​each target component. The difference between the main color of the surface and the background color Semantic similarity between component names and the text describing the current operation steps The weighted fusion yields the comprehensive visual saliency score. ,Right now: Based on this mapping, the gaze determination threshold for each component is determined. Calculate the penetration depth of the ray after it enters the component bounding box and reaches the actual geometric surface of the component. ,Right now: In the formula: The orientation direction vector of the normalized main virtual camera. The parameter value is the intersection point of the ray and the near-surface of the component enclosure. The parameter value is the intersection point of the ray and the actual mesh geometry surface of the component; the collision point is determined as a valid line of sight only when the penetration depth is within the preset effective range and the duration of the line of sight stay within the bounding box exceeds the gaze determination threshold. Inter-frame interpolation is performed based on the coordinates of the gaze points in adjacent frames in the gaze point time sequence to estimate the consecutive moments when the gaze first enters the bounding box boundary of the target component as the stable gaze start moment.

3. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 2, characterized in that, The initial visual-motor delay is calculated and error compensation is performed to obtain the corrected visual-motor delay. This corrected visual-motor delay is then decomposed into cognitive decision-making delay and motor execution delay, specifically including: The system captures student operation events on the target component through a global event listener, extracts the component identifier and records the current timestamp as the first operation time, synchronously records the system frame rate and input response latency, and deduplicates multiple operations on the same component. Establish a time mapping function for the gaze-to-action event pipeline, using frame numbers as a bridge to transform the stable gaze initiation time to a unified time base, pair it with the corresponding first action time, filter out data pairs where the first action time is earlier than the stable gaze initiation time, and calculate the initial gaze-motion delay. ,Right now: In the formula: This is the moment of first operation. To stabilize the starting point of fixation; Establish a unified time reference mapping between the gaze sampling clock and the operational event clock, and correct for sampling clock deviation at the start time of stable gaze, i.e.: In the formula: This is the stable gaze start time after correction for sampling clock deviation. This is the sampling clock offset correction amount; response delay compensation is performed at the first operation time based on the input response delay, i.e.: In the formula: This is the first operation time after input response delay compensation. The input response delay compensation amount is used; the difference between the two compensated moments is used as the corrected visual-motion delay. ,Right now: ; By tracking the movement of the mouse cursor in screen space and decomposing the corrected visual-motor delay into cognitive decision-making delay and motor execution delay based on its curvature characteristics, that is: ， ， ， In the formula: For cognitive decision delay, For motion execution delay, The moment corresponding to the abrupt change in the curvature of the mouse cursor movement trajectory; cognitive decision delays exceeding the preset reasonable range are marked as outliers and removed.

4. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 3, characterized in that, The monitoring of the number of pre-fixation saccades and the total duration of gaze confirmation is used to calculate the decision confidence index, specifically including: The moment when the gaze first enters the bounding box of the target component is taken as the first entry moment. The time segment before this moment is traced back. Within the anti-shake time window, the continuous duration of the gaze leaving the bounding box exceeds the preset departure threshold and then falls back in is counted as a valid saccade. The number of saccades before fixation is counted. The total duration of gaze confirmation is recorded as the time span from the first entry moment to the first operation moment. The number of pre-fixation saccades and the total gaze confirmation duration are normalized and then weighted and fused to obtain the decision confidence index. ,Right now: In the formula: This is a normalized index of the number of prefixation saccades. To confirm the total duration of the line of sight after normalization.

5. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 1, characterized in that, The calculation of gaze pattern entropy, combined with cognitive decision delay, motor execution delay, and decision confidence index, constructs an initial multidimensional proficiency fingerprint vector, specifically including: Within the bounding box of each target component, several sub-regions are divided according to the component's geometry. During an effective and stable gaze event, the trajectory of the gaze point shifting between the sub-regions is tracked, and the frequency of gaze point shifting between different sub-regions is statistically analyzed. Calculate the gaze pattern entropy value based on the probability distribution of gaze points in each sub-region. ,Right now: In the formula: For the first The probability distribution of gaze points in each sub-region; an initial multidimensional proficiency fingerprint vector is constructed using the cognitive decision delay, motor execution delay, decision confidence index, and gaze pattern entropy value of each operation step.

6. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 5, characterized in that, The standardized multidimensional proficiency fingerprint vector obtained after baseline compensation specifically includes: The system retrieves the current trainee's operation records for the same training project from the database, extracts the cognitive decision delay value and motor execution delay value for each step according to the step dimension, constructs a time series, and performs preprocessing. Baseline prediction models are trained on the preprocessed time series data. Using the trained models and the corresponding delay values ​​from each step as input, the cognitive decision-making delay baseline and motor execution delay baseline for the current learner are predicted at each step. ； ； In the formula: This represents the baseline predicted value for cognitive decision delay. For the first Cognitive decision-making delay observations at a historical moment Delaying cognitive decision-making Autoregressive coefficient of order, For the cognitive decision delay constant bias term, The baseline prediction value for motion execution delay, For the first The motion execution delay observation value at a historical moment For the motion execution delay Autoregressive coefficients of order 1 For the motion, perform a delay constant bias term. The optimal model order; The standardized offset is obtained by subtracting the corresponding individual response baseline from the cognitive decision delay value and motor execution delay value of each step, and then dividing by the historical standard deviation of the corresponding dimension. ； ； In the formula: Standardized offset for cognitive decision delay This represents the cognitive decision delay value for the current practical training. For the historical standard deviation of cognitive decision delay, The offset is standardized to the motion execution delay. This represents the motion execution delay value during the current training session. The historical standard deviation of motor execution delay is used; the two standardized offsets of each step are combined with the normalized decision confidence index and gaze pattern entropy value to form a standardized multidimensional proficiency fingerprint vector; at the same time, the cognitive decision delay value, motor execution delay value, decision confidence index and gaze pattern entropy value of each step are arranged in a fixed dimension order to form the original multidimensional proficiency fingerprint vector.

7. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 6, characterized in that, The process of constructing an expert reference fingerprint, adaptively calibrating a delay threshold, comparing it with a standardized multidimensional proficiency fingerprint vector, generating proficiency tags, and calculating similarity distances specifically includes: Data on cognitive decision-making delay, motor execution delay, decision confidence index, and gaze pattern entropy of expert operators on the same training task were collected. The mean value of the expert group at each step was calculated as the expert standard value, and an expert reference fingerprint vector was constructed. The first delay threshold is determined based on the statistical distribution of delay offset standardized by an expert group. ,Right now: In the formula: Standardize the mean delay offset for the expert group. Standardize the standard deviation of the delay offset for the expert group. The second delay threshold is determined by either a preset coefficient or a statistical distribution of standardized delay offsets among students in the class. ,Right now: In the formula: Standardize the mean delay offset for each student in the class. Standardize the standard deviation of the delay offset for each student in the class; or take the smaller of the two as the mixed delay threshold. The standardized delay offset of each step is compared with the delay threshold. Steps that are less than or equal to the delay threshold are marked as proficient, and steps that are greater than the delay threshold are marked as unproficient. Calculate the similarity distance between the trainee's original multidimensional proficiency fingerprint vector and the expert reference fingerprint vector. ,Right now: In the formula: The original multidimensional proficiency fingerprint vector is the first... dimensional components, For experts to refer to the fingerprint vector, the first Multidimensional components; outputs a standardized multidimensional proficiency fingerprint vector with proficiency labels, similarity distance, and calibrated delay threshold.

8. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 1, characterized in that, The step-by-step identification of root bottlenecks and derived weak steps based on step dependencies specifically includes: Obtain the logical sequential dependencies between each operation step in the training process, construct a directed dependency graph between steps with each operation step as a node and the dependencies between steps as directed edges, and the direction of the directed edges is from the previous step to the subsequent step. The normalized delay offsets of each step are mapped to the corresponding nodes in the directed dependency graph. The ratio of the normalized delay offset of the successor step to the preceding step in adjacent step pairs is calculated and used as the propagation amplification factor. ,Right now: In the formula: Preliminary steps The standardized delay offset, For subsequent steps The standardized delay offset; When the standardized delay offset of a node and at least one of its successor steps exceeds a preset offset threshold, and the corresponding propagation amplification factor is greater than a preset amplification factor threshold, and there is no root cause bottleneck step in the preceding steps of the node, the node is marked as a root cause bottleneck step. When the standardized delay offset of a node exceeds a preset offset threshold and none of its preceding steps exceed the threshold, it is directly marked as a root cause bottleneck step; when the standardized delay offset of a node exceeds the preset offset threshold and there is a preceding root cause bottleneck step, but the root cause bottleneck determination condition is not met, it is marked as a derived weak step.

9. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 8, characterized in that, The calculation of the forward gaze offset, generating an evaluation result for the coherence of the operational process, specifically includes: Following the execution order of the operation steps, obtain the stable gaze start time and the first operation time for each pair of adjacent operation steps, and calculate the difference in stable gaze start times between adjacent step pairs as the gaze transfer time interval. The difference between the times of the first operation is calculated as the operation transition time interval. ; The difference between the operation transfer time interval and the gaze transfer time interval is calculated as the gaze forward offset. ,Right now: A positive value indicates that the student has begun to focus on the target of the next step before completing the previous step, while a negative value indicates that the search for the target of the next step only begins after the previous step is completed. Calculate the look-ahead offset for all adjacent step pairs and determine the percentage of positive offsets. and average ,Right now: , In the formula: This represents the number of adjacent step pairs where the look-ahead offset is positive. This represents the total number of adjacent step pairs in the training process. For the first A positive line-of-sight look-ahead offset is generated; if the proportion exceeds the preset proportion threshold and the average value exceeds the preset look-ahead threshold, a positive operation process coherence evaluation result is generated, otherwise a negative evaluation result is generated.

10. The method for recognizing operational proficiency based on view-motion delay in virtual training according to claim 9, characterized in that, The process of generating a personalized list of weaknesses, outputting real-time visual cues and fatigue warnings, performing cluster analysis on standardized multidimensional proficiency fingerprint vectors, and adaptively calibrating the delay threshold specifically includes: Based on the standardized multidimensional proficiency fingerprint vector with proficiency tags, each unproficient step is arranged in the order of root bottleneck steps taking precedence over derived weak steps, generating a personalized list of weaknesses, and matching targeted training content for each weak step from a preset training suggestion library. Within a preset time window after the trainee completes each operation step, real-time visual cues are output in a non-blocking manner based on the proficiency level of that step; when the motion execution latency value monotonically increases and the linearity index of the mouse movement trajectory... When the output is monotonically decaying, a non-blocking fatigue warning is provided, wherein: In the formula: These are the starting screen coordinates of the mouse cursor movement path. Terminating screen coordinates; Cluster analysis was performed on the standardized multidimensional proficiency fingerprint vectors of all students in the class, dividing them into high proficiency group, medium proficiency group and need-to-improve group, and identifying common weak steps in each group that exceeded the preset proportion threshold. The standardized multidimensional proficiency fingerprint vectors, comprehensive scores, weak point lists, line-of-sight offsets, operation process coherence evaluation results, and similarity distances from each training session are stored in the database. This supports the visualization of historical trends and group distribution, and the delay threshold is dynamically updated based on the accumulated training data.

Images