Enhanced Instructional Interaction Methods and Systems Based on Eye-Hand Coordination Prediction

By constructing a unified benchmark for multimodal data acquisition and processing, and combining eye-hand collaborative prediction and safety early warning mechanisms, the problem of lack of benchmarks for multi-sensor data in XR teaching scenarios has been solved, improving interaction accuracy and security, achieving efficient data management, and adapting to virtual experiment scenarios in primary and secondary schools and universities.

CN123114249APending Publication Date: 2026-11-17HUAZHONG NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611159592.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-02
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing XR teaching scenarios suffer from problems such as a lack of unified benchmarks for multi-sensor data, insufficient forward-looking prediction for eye-hand coordination, lack of rendering adaptation and security mechanisms, and incomplete management of teaching data, resulting in low interaction accuracy, numerous security risks, and ineffective data management.

Method used

By initializing and subdividing the data, a unified benchmark for multimodal data acquisition and processing is constructed. Combined with an eye-hand coordination prediction mechanism, spatiotemporal alignment and feature extraction of multi-sensor data are achieved. A gaze-driven rendering algorithm and a safety warning mechanism are adopted, and a teaching data management system is constructed.

Benefits of technology

It improves the accuracy and security of XR teaching interaction, achieves efficient data management and security protection, is compatible with virtual experiment scenarios in primary and secondary schools and universities, and enhances the intelligence and practicality of teaching.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114249A_ABST
    Figure CN123114249A_ABST
Patent Text Reader

Abstract

This invention belongs to the cross-disciplinary field of XR virtual-real fusion teaching and multimodal human-computer collaborative interaction, and provides an enhanced teaching interaction method and system based on eye-hand coordination prediction. The method of this invention includes: (1) initialization calibration subdivision; (2) multimodal data acquisition; (3) feature extraction; (4) motion trend prediction; (5) interaction control; (6) visual output of the screen; and (7) teaching data management. This invention integrates multi-source data of eye movement, hand movement, and head movement, and combines motion temporal models and filtering algorithms to realize short-term prediction of gaze direction and hand movements. It relies on collision detection, layered rendering, and visual enhancement to optimize the interactive experience and scene security; and completes the functions of learning focus assessment, encrypted storage of interaction logs, teaching retrospection, and breakpoint continuation of learning. This invention can effectively improve the accuracy of XR teaching interaction, screen presentation effect, and data security capabilities, and enhance the intelligence and practicality of virtual teaching scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of XR virtual-real fusion teaching and multimodal human-computer collaborative interaction technology, and more specifically, relates to an enhanced teaching interaction method and system based on eye-hand coordination prediction. Background Technology

[0002] With the continuous improvement of hardware computing power in Extended Reality (XR), which is composed of Virtual Reality (VR) and Augmented Reality (AR), virtual experiments and immersive online and offline classrooms that blend the virtual and real worlds are becoming increasingly common. Compared to the limitations of traditional physical teaching, which is restricted by venues, experimental equipment, and high-risk operating conditions, XR teaching scenarios can reproduce a full range of simulation instruments, significantly improving students' sense of immersion in hands-on practice and their efficiency in understanding abstract knowledge. At the same time, multimodal perception technologies such as infrared eye tracking, 3D hand vision capture, and visual-inertial odometry pose calculation continue to mature. Proactive and predictive interaction based on human motion time sequence has become the mainstream research and development direction in the XR field. Multi-sensor fusion acquisition solutions can simultaneously acquire multi-dimensional behavioral data of user gaze, hand operation, and head posture, breaking away from the passive interaction mode of traditional fixed buttons and preset static gestures. This provides underlying technical support for quantifying student learning status, low-latency human-computer interaction, and data retention throughout the teaching process. The industry urgently needs a dedicated augmented teaching interaction solution that balances interaction accuracy, training safety, and teaching data management.

[0003] There are still many technical shortcomings in the existing XR eye-hand interaction system for teaching scenarios. Moreover, the defects are coupled with each other and are difficult to solve by optimizing a single module: (1) The lack of multi-sensor spatiotemporal reference. The devices on the market only independently collect eye-tracking, RGB and inertial raw data. There is no integrated joint calibration process. There is no unified head center coordinate system and global temporal reference. Multi-source images and inertial data have spatial offset and temporal misalignment, which directly reduces the basic accuracy of subsequent feature extraction and motion prediction; (2) There is no eye-hand collaborative look-ahead prediction mechanism. The existing solutions only rely on the real-time sensor data of the current frame to trigger the interaction. (2) The system does not extrapolate the user's future gaze and hand movements based on the historical movement sequence. Due to the inherent delay of the XR imaging and rendering link, the virtual knob and precision experimental control operation has high latency, floating accidental touch and missed touch problems frequently occur; (3) The rendering adaptation and training safety mechanism is blank. The general XR uses global unified resolution rendering and does not allocate rendering computing power according to the student's gaze area. The recognition of small scales and instrument readings is poor; and there is no hand trajectory prediction logic. The prompt is only triggered after touching the danger zone. There is no advance warning and interaction restriction. There are operational safety hazards in high-risk experiments such as virtual chemistry and electricity; (4) The teaching full-link data management system is missing. The existing system only records the interaction actions simply and lacks a multi-dimensional attention measurement model based on the gaze behavior of the interest area; the interaction log has no anti-tampering verification and end-to-end encryption mechanism. It does not support the three-dimensional trajectory playback of the teaching process, the course breakpoint continuation, and restricted encrypted sharing. It cannot meet the rigid needs of campus teaching data privacy protection and teacher classroom review. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of existing technologies, this invention provides an enhanced teaching interaction method and system based on eye-hand coordination prediction, which can effectively improve the accuracy of XR teaching interaction, the effect of image presentation and data security capabilities, enhance the intelligence and practicality of virtual teaching scenarios, and provide a new, intelligent and systematic implementation solution for XR enhanced teaching interaction.

[0005] The objective of this invention is achieved through the following technical measures.

[0006] This invention provides an enhanced teaching interaction method based on eye-hand coordination prediction, comprising the following steps:

[0007] (1) Initialize calibration subdivision, start XR enhanced teaching application, configure eye movement detection parameters, calibrate user's eye movement parameters, and fit and generate user-specific eye model, and encapsulate and save the model parameters; construct the topology of five fingers bones based on human hand features, calculate and generate hand kinematic constraint matrix and initial posture data of all joints; complete the calibration of multiple cameras and inertial measurement units, coordinate system one, and time synchronization, generate external parameter transformation matrix and timestamp configuration file, and construct a unified spatiotemporal reference for multimodal data acquisition and pose calculation;

[0008] (2) Multimodal data acquisition: long and short exposure infrared images are acquired in time-division using an infrared eye-tracking unit and time-series alignment is achieved to generate long and short exposure image sequences with unified timestamps; images are acquired using an RGB camera and semantic segmentation is performed, and time-series aligned hand region image sequences and environmental background image sequences are generated after synchronizing frame time; head inertial data is acquired through an IMU, and cross-modal time-series matching of inertial data with eye-tracking and hand image data is completed to generate a time-series unified head inertial data sequence and corresponding timestamps;

[0009] (3) Feature extraction: Weighted least squares ellipse and subpixel fitting algorithms are used to process long and short exposure images respectively to extract the coordinates of pupil and corneal reflection points; the target detection model and geometric discrimination method are used in combination to extract the three-dimensional coordinates of hand joints and complete the chirality determination; the visual inertial odometry processing pipeline is used to fuse images and inertial data to calculate the 6DoF pose sequence of the XR head-mounted display.

[0010] (4) Motion trend prediction: Combine the corneal spherical reflection optical model and head position and angle compensation algorithm to calculate the user's real-time gaze direction vector; Based on joint constraints, use forward and reverse kinematics algorithm and time series model to predict the short-time three-dimensional coordinates of each joint of the hand and fingertips; Use neural extended Kalman filter and linear extrapolation method to obtain the predicted gaze direction.

[0011] (5) Interactive control: Construct a spatiotemporal reference sequence by combining short-term gaze direction and hand prediction coordinates, use a directed OBB intersection detection algorithm to determine the target interactive object, obtain the ID and interaction priority of the selected object; adaptively adjust the activation threshold of the selected object according to the interaction priority to avoid false triggering; verify the distance between the fingertip and the target object and the consistency of the ID, trigger double-click to confirm the interaction and generate interactive event data;

[0012] (6) Visual output: Based on the predicted gaze direction and XR head-mounted display parameters, the rendering center point is determined. High-definition images are generated by using gaze-driven region of interest multi-level rendering technology and dynamic field radius adjustment algorithm. Super-resolution, contrast enhancement and background blurring are performed on low-resolution, low-contrast, and occluded areas. Dangerous actions are predicted through collision detection and safety protection is achieved by combining icon warning and interactive response freezing algorithm.

[0013] (7) Teaching data management: divide ROI and configure weighting parameters, accumulate user gaze duration, and obtain focus score and level after pupil standard deviation correction; collect interactive event data and complete serialization storage, and use incremental hash algorithm to complete data verification and deduplication; the teacher retrieves log files, reconstructs three-dimensional scene, uses encryption technology to ensure data security, and supports trajectory playback, breakpoint continuation learning and encrypted sharing links and restricted log distribution.

[0014] Furthermore, the initial calibration subdivision described in step (1) specifically includes:

[0015] (1-1) Initialize the eye tracking channel, start the XR-enhanced teaching application, and configure the eye recognition threshold, effective eye detection area, and eye pose calculation benchmark parameters in sequence; calibrate the user's corneal curvature, pupil size, anterior chamber depth, and Kappa angle eye parameters; refer to the publicly available standard eye dataset, fit and generate a user-specific eye mathematical model, and encapsulate the model parameters into a local configuration file;

[0016] (1-2) Initialization of the hand tracking channel: Based on the anatomical parameters of the human hand, determine the rotation axis direction, motion freedom, and physiological motion limits of the left and right palmar interphalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints of the user in front of the camera; with the wrist as the topological root node, construct the topological structure of the five fingers bones using hierarchical chain kinematic rules, and calculate and generate the hand kinematic constraint matrix and the initial posture data of all joints;

[0017] (1-3) Eye-hand coordination reference calibration: Zhang Zhengyou calibration method is used to uniformly solve the lens distortion parameters of the devices including infrared eye-tracking camera, visible light RGB camera, depth sensor and IMU; cross-device coordinate system transformation is completed through hand-eye joint calibration algorithm, and all sensing devices are uniformly mapped to the XR head center reference coordinate system; the sampling time sequence of multiple devices is aligned with the hardware synchronization unit, and external parameter transformation matrix and timestamp configuration file are generated to construct a unified spatiotemporal reference for multimodal data acquisition and pose calculation.

[0018] Furthermore, the multimodal data acquisition in step (2) specifically includes:

[0019] (2-1) Eye-tracking data acquisition: Based on the configuration parameters, the time-slice timing scheduling algorithm is used to drive the infrared eye-tracking unit to alternately acquire long and short exposure infrared images. Long exposure images are used to capture the iris and pupil boundary, and short exposure images are used to locate the infrared reflection spot on the corneal surface. The global clock is used to mark the timestamp frame by frame, and the time interpolation frame registration algorithm is used to complete the temporal alignment of images with different frame rates, generating a sequence of long and short exposure images with a unified timestamp.

[0020] (2-2) Hand data acquisition: A visible light RGB camera is used to continuously acquire images of the user's hand, experimental table and experimental equipment in the virtual scene; U-Net3+ full-scale semantic segmentation network is used to distinguish the hand foreground from the table and experimental equipment background; a global clock is used to complete frame timing synchronization and generate timing-aligned hand region image sequence and environmental background image sequence.

[0021] (2-3) Head motion data acquisition: IMU is used to continuously acquire raw inertial data of the user's head in three axes of acceleration and angular velocity; hardware edge-triggered synchronization algorithm is applied to bind sampling timestamps to each set of inertial data; according to the IMU-visual timestamp mapping protocol, cross-modal temporal matching of inertial data with eye movement and hand image data is completed to generate a time-unified head inertial data sequence and corresponding timestamps.

[0022] Furthermore, the feature extraction in step (3) specifically includes:

[0023] (3-1) Eye feature extraction: The weighted least squares ellipse fitting algorithm is used to process long exposure images, segment the pupil and iris boundaries, and calculate the pixel coordinates of the pupil center; the four-neighbor Gaussian subpixel fitting algorithm is used to process short exposure images, and the peak value of the surface is taken as the coordinates of the corneal reflection point; after the feature calculation is completed, the image cache is uniformly recycled and the running memory is released to generate the inter-frame temporally aligned sequence of pupil center and spot center coordinates.

[0024] (3-2) Hand feature extraction and chirality determination: The YOLOv7-m lightweight target detection model was used to detect the hand region image and calculate the two-dimensional coordinates of the palm and the bounding box parameters of the hand orientation. After image cropping and pose correction based on OBB, the FastMETRO hand pose estimation model was used to extract the 3D coordinates of 21 hand joints and the pose legality was verified by combining the hand kinematic constraint matrix. Spatial reference vectors were constructed based on the metacarpophalangeal joints of the index and middle fingers. The left and right hand nature was determined by the phalangeal angle sign statistical method, and the three-dimensional coordinate sequence of hand joints and the chirality label sequence were generated.

[0025] (3-3) Head pose calculation: The SuperVINS visual inertial odometry processing pipeline is used to process RGB images and IMU inertial data. The SuperPoint feature extraction network is called to obtain sparse corner points and edge features of the image. The LightGlue feature matching network is used to complete the pairing, and the LK optical flow algorithm is used to track feature motion. Standard closed pre-integration is performed on the IMU data. The nonlinear equation is constructed by combining visual reprojection error and IMU pre-integration residual to solve iteratively and obtain the 6DoF pose sequence in the XR head-mounted display coordinate system.

[0026] Furthermore, the motion trend prediction in step (4) specifically includes:

[0027] (4-1) Fixation of gaze direction: Based on the spatial position of the infrared light source, the coordinates of the pupil and light spot and the 6DoF pose sequence of the head, the three-dimensional coordinates of the corneal curvature center are solved using the corneal spherical reflection optical model; the optical axis of the eyeball is obtained by connecting the corneal center and the pupil center, and the optical axis deviation is compensated by using the Rodriguez rotation formula in combination with the Kappa angle parameter, so as to obtain the user's real-time gaze direction vector.

[0028] (4-2) Future gaze prediction: NEKF is used to process the user's continuous multi-frame real-time gaze direction sequence, adaptively correct process noise, optimize eye movement parameters, and obtain the current gaze state and movement rate; the sliding window linear extrapolation method is used to extrapolate the gaze angle at future moments and obtain the predicted gaze direction.

[0029] (4-3) Hand motion prediction: Based on the three-dimensional coordinates of the hand joints and the limit constraints of each joint's movement, the inverse kinematics algorithm is used to solve the joint rotation angle; the joint angular velocity and angular acceleration are calculated by the inter-frame difference method, and the joint angle in future frames is inferred by the motion time-series prediction model; the three-dimensional predicted coordinates of the fingertip and each phalanx are generated by mapping along the skeletal chain step by step using the forward kinematics algorithm.

[0030] Furthermore, the interactive control described in step (5) specifically includes:

[0031] (5-1) Intent object determination: Based on the predicted gaze direction and combined with the fingertip three-dimensional predicted coordinates obtained in step (4), a spatiotemporal reference sequence is constructed; the directed OBB intersection detection algorithm is used to determine the intersection state of the gaze, fingertip and the object in the enhanced teaching process; if the intersection point points stably to the same object for multiple consecutive frames and the number of dwell frames reaches the preset threshold, the object is determined to be the user's target interaction object, and the interaction object ID, spatial coordinates and interaction priority are determined;

[0032] (5-2) Dynamic threshold adjustment: Based on the selected object ID and interaction priority parameters, an adaptive dynamic distance threshold algorithm is used to reduce the activation threshold of the selected object to adapt to the fine-tuning, rotation, knob and button click operations of the virtual scene; the trigger threshold of non-selected objects is simultaneously amplified to avoid false triggers caused by the user's gaze scanning or slight hand tremors, and the list of effective activation thresholds of interactive objects is updated.

[0033] (5-3) Double confirmation trigger: Based on the three-dimensional predicted coordinates of the fingertip, the dynamic activation threshold of the object, and the ID of the selected object, calculate the Euclidean distance between the predicted fingertip and the center of the bounding box of the selected object frame by frame; when the distance is less than the dynamic threshold of the object, and the ID of the object is consistent with the ID of the interactive object jointly determined by the gaze and the hand, the interaction condition is determined to be met, and the preset action instructions of clicking, sliding, and rotating are executed to complete the double confirmation interaction of eye and hand, and generate interaction event data.

[0034] Furthermore, the visual output of the image in step (6) specifically includes:

[0035] (6-1) Gaze-based rendering: Combining the predicted gaze direction with XR head-mounted display parameters, the rendering center point of the entire image is determined; gaze-driven ROI multi-level resolution rendering technology is adopted to divide the image into a gaze core area, a teaching content area, and an outer background area, and match high, medium, and basic resolutions respectively; a dynamic field of view radius adjustment algorithm is used to adaptively adjust the high-definition area range according to the user's gaze speed, render the virtual objects that the user is focused on in high definition, and generate a layered composite image;

[0036] (6-2) Visual enhancement: Based on the predicted user gaze direction, locate the target area of ​​the screen. If the pixel size of the area is lower than the preset recognizable threshold, the screen contrast is insufficient, or there is foreground object occlusion, the enhancement logic is automatically triggered. Pixel-level local super-resolution, adaptive contrast enhancement, and foreground and background blurring are executed in sequence to highlight key teaching information and generate enhanced screen frames.

[0037] (6-3) Safety warning: Based on the predicted hand coordinates and predicted gaze direction, a sweeping collision detection algorithm is used to predict the collision with the pre-marked danger zone surrounding the virtual teaching scene. If it is determined that the fingertip is about to enter the danger zone and the gaze direction is not aligned with the area, a high-frequency flashing warning icon is added to the center of the gaze point through real-time layer overlay rendering technology. The interactive response freeze algorithm is enabled to restrict hand operation and realize real-time safety protection of the virtual experimental scene.

[0038] Furthermore, the teaching data management described in step (7) specifically includes:

[0039] (7-1) Attention assessment: In the virtual experimental scene, ROIs are pre-divided and differentiated weight parameters are configured. Based on the real-time gaze direction frame by frame, the three-dimensional coordinates of the gaze point are obtained through scene intersection operation. The gaze point is matched to the corresponding ROI, and the effective gaze duration of each region is accumulated. The weighted correction is completed by combining the standard deviation of pupil diameter. The normalized attention score and attention level label are obtained to analyze the student's learning status.

[0040] (7-2) Interaction log storage: When an eye-hand coordination interaction event is triggered, the event timestamp, target object ID, predicted gaze direction, hand joint angle and chirality label, and current teaching segment code are collected; the data is structured and serialized in Protocol Buffers format, written to a local flash log file and asynchronously uploaded to a cloud storage server; each record uses an incremental hash algorithm to generate a check code to achieve data integrity verification and duplicate data filtering;

[0041] (7-3) Teaching Replay and Resume Learning from Breakpoints: Teachers can pull log files from the cloud server to reconstruct a 3D virtual scene locally, restore the user's gaze trajectory and hand movements along the original timeline, and support playback operations such as pause, speed adjustment, and keyframe jump; local logs are stored using the AES-256 advanced encryption standard, and the cloud transmission link uses the TLS protocol to ensure the security of teaching data; when students exit the course, a breakpoint snapshot is automatically generated, and users can resume their learning status by logging in again; teachers can create encrypted sharing links with passwords and expiration dates to achieve restricted distribution of logs.

[0042] This invention also provides an enhanced teaching interaction system based on eye-hand coordination prediction, including modules for initialization calibration subdivision, multimodal data acquisition, feature extraction, motion trend prediction, interactive control, visual output, and teaching data management.

[0043] The initialization calibration subdivision module completes the calibration of eye-tracking, hand, and head sensor parameters, constructs a dedicated model and skeletal topology, and unifies the coordinate system and sampling timing of multiple devices.

[0044] The multimodal data acquisition module acquires eye-tracking images, hand images, and head inertial data, completes the temporal alignment of multi-source data, and generates a standard data sequence with timestamps.

[0045] The feature extraction module extracts the coordinate information of the pupil and reflective point, calculates the three-dimensional coordinates of the hand joints and determines the hand nature, and fuses the data to solve the 6DoF pose of the XR head-mounted display.

[0046] The motion trend prediction module combines an optical model to calculate the real-time gaze direction, relies on a time-series model to predict hand movement posture, and uses a filtering algorithm to deduce the subsequent gaze angle.

[0047] The interaction control module determines the interaction object and classifies the interaction priority, adaptively adjusts the interaction trigger threshold, verifies information, and completes the eye-hand coordination interaction trigger.

[0048] The image visual output module performs multi-level rendering based on the gaze area, performs super-resolution and contrast enhancement on the image, detects dangerous areas, and achieves experimental safety protection.

[0049] The teaching data management module counts gaze duration to assess learning focus, verifies, encrypts, and stores interaction logs, and enables trajectory playback, breakpoint continuation of learning, and log sharing.

[0050] The beneficial effects of this invention are as follows: By unifying the multi-sensor coordinate system and global temporal reference, it eliminates the spatial and temporal misalignment defects of multi-source data; relying on the prior constraints of hand skeletal anatomy to optimize image segmentation, it improves the accuracy of eye, hand, and head data acquisition and feature extraction from the source; it innovatively introduces an eye-hand collaborative look-ahead prediction mechanism, which is not limited to real-time frame data for interactive judgment and image rendering, significantly reducing the inherent transmission and rendering latency of XR devices, and effectively reducing the phenomenon of floating accidental touches and missed touches in virtual precision knobs and instrument operation scenarios; it pioneers a gaze-driven three-level resolution adaptive rendering and teaching target visual enhancement algorithm to improve the visibility of small experimental scales and instrument readings; and it adds a feature based on... The system features a pre-collision warning and interaction freeze mechanism based on hand trajectory prediction, filling the gap in active safety protection for existing XR virtual experiments. It also constructs a quantitative attention assessment model based on gaze behavior, complete with log hash verification, end-to-end encrypted storage, 3D playback, breakpoint continuation learning, and encrypted sharing, balancing classroom teaching review needs with campus teaching data privacy and security. The technical process fully covers the entire process of hardware calibration, multimodal perception, motion prediction, intelligent interaction, adaptive rendering, and teaching data, distinguishing it from general consumer-grade AR gesture interaction solutions. It is highly adaptable to teaching scenarios such as virtual experiments and immersive classrooms in primary and secondary schools and universities, significantly improving system intelligence and scenario specificity. Attached Figure Description

[0051] Figure 1 This is a flowchart of the enhanced teaching interaction system framework based on eye-hand coordination prediction in an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram of the topological structure of the five fingers in an embodiment of the present invention. 201 - Wrist topological root node, 202 - Thumb carpal joint, 213 / 223 / 233 / 243 / 253 - Thumb / index / middle / ring / little metacarpophalangeal joints, 224 / 234 / 244 / 254 - Proximal interphalangeal joints of index / middle / ring / little fingers, 225 / 235 / 245 / 255 - Distal interphalangeal joints of index / middle / ring / little fingers, 215 - Thumb interphalangeal joint, 216 / 226 / 236 / 246 / 256 - Thumb / index / middle / ring / little fingertip.

[0053] Figure 3 These are schematic diagrams of long-exposure and short-exposure infrared imaging of the human eye in embodiments of the present invention, 301 - long-exposure infrared image of the human eye, 302 - short-exposure infrared image of the human eye.

[0054] Figure 4 This is a schematic diagram of the hand foreground segmentation process of the U-Net3+ network in an embodiment of the present invention, 401-input layer, 402-downsampling encoding layer, 403-fusion decoding layer, 404-convolutional feature mapping layer, 405-output layer.

[0055] Figure 5 This is a schematic diagram of the XR head-mounted display reference coordinate system in an embodiment of the present invention, with 501-X-axis, 502-Y-axis, and 503-Z-axis.

[0056] Figure 6 This is a schematic diagram of the layered ROI region division of the virtual teaching scene in this embodiment of the invention, where 601 is the gaze core rendering area, 602 is the teaching content rendering area, and 603 is the peripheral background rendering area.

[0057] Figure 7 This is a schematic diagram comparing the images before and after visual enhancement of the gaze area in an embodiment of the present invention. 701 - Original image before visual enhancement, 702 - Image after visual enhancement.

[0058] Figure 8 This is a schematic diagram of a virtual experiment danger zone warning in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0060] like Figure 1 As shown, this embodiment provides an enhanced teaching interaction system based on eye-hand coordination prediction, including an initialization calibration subdivision module, a multimodal data acquisition module, a feature extraction module, a motion trend prediction module, an interaction control module, a visual output module, and a teaching data management module.

[0061] This embodiment also provides an enhanced teaching interaction system based on eye-hand coordination prediction, including the following steps:

[0062] (1) Initialize calibration subdivision. Start the XR-enhanced teaching application, configure eye-tracking detection parameters, calibrate the user's eye-tracking parameters, and fit and generate a user-specific eye model. Encapsulate and save the model parameters. Construct the topology of the five-finger skeleton based on human hand features, calculate and generate the hand kinematic constraint matrix and the initial posture data of all joints. Complete the calibration, coordinate system unification, and time synchronization of the multi-camera and inertial measurement unit (IMU), generate the extrinsic parameter transformation matrix and timestamp configuration file, and construct a unified spatiotemporal reference for multimodal data acquisition and pose calculation.

[0063] (1-1) Initialization of the eye tracking channel. Start the XR-enhanced teaching application, and configure the pupil grayscale segmentation threshold to 45, the pupil contour area determination threshold to 120 pixels, and the eyelid occlusion determination threshold to 0.32 to form the eye movement recognition threshold. Configure a rectangular area with the center of the screen as the origin, ±160 pixels horizontally and ±120 pixels vertically as the effective detection area of ​​the eye. Configure the pitch reference angle 0°, yaw reference angle 0°, and roll reference angle 0° as the reference parameters for eye posture calculation. Calibrate the user's corneal curvature, pupil size, anterior chamber depth, and Kappa angle eye parameters. Referring to the publicly available standard eye dataset, fit and generate a user-specific eye mathematical model, and encapsulate the model parameters into a local configuration file.

[0064] (1-2) Initialization of the hand tracking channel. Based on the anatomical parameters of the human hand, the rotational axes, degrees of freedom, and physiological limits of flexion and abduction of the following joints are determined for the user's left and right hands directly in front of the camera: the carpometacarpal joint (CMC) of the thumb, the metacarpophalangeal joints (MCP) of the five fingers, the proximal interphalangeal joints (PIP) of the four fingers, the distal interphalangeal joints (DIP) of the four fingers, and the interphalangeal joint (IP) of the thumb. Using the wrist as the topological root node, a hierarchical chain kinematics rule is used to construct the topological structure of the five-finger skeleton, and the hand kinematic constraint matrix and initial posture data of all joints are calculated and generated. The calculation steps for the initial posture data of all hand joints are as follows:

[0065] Ⅰ: with Figure 2 As shown, 201 serves as the root node, with each of the five fingers forming an independent kinematic chain. The nodes of each chain are arranged sequentially from the wrist to the fingertips, from near to far:

[0066] {Thumb chain: 201→202→213→215→216;}

[0067] Index finger chain: 201→223→224→225→226;

[0068] Middle finger chain: 201→233→234→235→236;

[0069] Ring finger chain: 201→243→244→245→246;

[0070] Little finger chain: 201→253→254→255→256}.

[0071] A total of 21 skeletal nodes are involved in the recursive calculation of spatial coordinates.

[0072] II: Let the three-dimensional spatial coordinates of 201 be P0=[x0,y0,z0] T The bone segment between two adjacent nodes is defined as a link. The links from 201 to the fingertip in a single finger are sequentially assigned the index i. The thumb link contains 4 nodes and 3 links, and the value of i is in the range [1,3]. The links of the other four fingers contain 5 nodes and 4 links, and the value of i is in the range [1,4].

[0073] III: Calculate the three-dimensional coordinates P of the node on the fingertip side of the i-th link using the following formula. i :

[0074]

[0075] Among them, l i Let α be the length of the link. i Let θi be the rotation angle of the joint at the end of the i-th segment of the link closest to 201, and d be the fixed torsion angle. i R is the axial offset of the connecting rod, and R is the value derived from θ. i and α i The constructed third-order rotation matrix, e i-1 Let e0 be the unit vector of the i-th link segment. The thumb with i=1 represents link 201-202, and the vector e0 indicates the direction from P0 to P1.

[0076] IV: When 201 rotates or translates, its pose change is transmitted level by level along each finger chain, updating the spatial coordinates of CMC, MCP, PIP, DIP, IP and the fingertip point level by level, and finally obtaining the coordinates of all nodes of the complete five-finger skeleton topology.

[0077] (1-3) Eye-hand coordination reference calibration. The Zhang Zhengyou calibration method is used to uniformly solve the lens distortion parameters of the devices including the infrared eye-tracking camera, visible light RGB camera, depth sensor, and IMU; the cross-device coordinate system transformation is completed through the hand-eye joint calibration algorithm, and all sensing devices are uniformly mapped to the XR head center reference coordinate system; the sampling time sequence of multiple devices is aligned with the help of the hardware synchronization unit, and the external parameter transformation matrix and timestamp configuration file are generated to construct a unified spatiotemporal reference for multimodal data acquisition and pose calculation.

[0078] (2) Multimodal data acquisition. Long and short exposure infrared images were acquired in a time-division manner using an infrared eye-tracking unit and time-series alignment was achieved to generate long and short exposure image sequences with unified timestamps; images were acquired using an RGB camera and semantically segmented, and time-series aligned hand region image sequences and environmental background image sequences were generated after synchronizing frame time; head inertial data was acquired through an IMU, and cross-modal time-series matching of inertial data with eye-tracking and hand image data was completed to generate a time-series unified head inertial data sequence and corresponding timestamps.

[0079] (2-1) Eye-tracking data acquisition. Based on the configuration parameters, a time-sharing scheduling algorithm is used to drive the infrared eye-tracking unit to alternately acquire long and short exposure infrared images. Long exposure images are used to capture the iris and pupil boundaries, while short exposure images are used to locate the infrared reflection spot on the corneal surface. The human eye's long and short exposure infrared imaging is as follows: Figure 3 As shown, a global clock is used to mark timestamps frame by frame, and a time interpolation frame registration algorithm is used to complete the temporal alignment of images at different frame rates, generating long and short exposure image sequences with a unified timestamp.

[0080] (2-2) Hand Data Acquisition. Using a visible light RGB camera, 30 frames were continuously acquired in a single batch at a fixed frame rate, containing original images of the user's hands, the experimental table, and the virtual experimental equipment within the teaching scene; Figure 4 The U-Net3+ full-scale semantic segmentation network shown performs pixel-level foreground segmentation, reuses a global clock to unify the time stamps of all image frames, and finally outputs a time-aligned sequence of hand region images and a sequence of environmental background images. The specific steps of the hand skeleton prior constraint-assisted accurate segmentation are as follows:

[0081] Ⅰ: Input 30 frames of original RGB images into the U-Net3+ network for inference, and output a pixel-level hand confidence probability map P frame by frame. k (x,y), k is the frame number, (x,y) are the two-dimensional coordinates of the image pixels, P k (x,y)∈[0,1] represents the prediction confidence that the pixel belongs to the hand region.

[0082] II: Load the 21-node 3D skeleton point set obtained during initialization and the camera intrinsic and extrinsic parameter matrix, project all 3D skeleton points onto the 2D image plane, and obtain a 2D standard skeleton point set. , where i is the index of the bone node, and a binary bone constraint mask M(x,y) is generated based on the range of link distribution.

[0083] III: The following formula is used to quantify the degree of overlap between the predicted high-confidence region and the standard skeletal points in the current frame, to determine whether a real human hand exists in the image:

[0084]

[0085] Where Sim represents the similarity of skeletal point matching. The value of the mask corresponding to the position of the i-th bone node is set. If Sim < 0.6, it is determined that there is no real hand area in the current frame; when Sim ≥ 0.6, it is confirmed that there is a human hand in the picture, and subsequent fine segmentation processing is carried out.

[0086] IV: Using M(x,y) as prior weights for P kThe (x,y) region is corrected to enhance the confidence of skeletal region prediction and filter background interference. The corrected probability map P′(x,y) is calculated using the following formula:

[0087]

[0088] Among them, the area covered by the skeleton links and joint points satisfies M(x,y)=1, and the background area satisfies M(x,y)=0.

[0089] V: Set the discrimination threshold to 0.5 for P k Binarization of (x,y) is performed to generate a binary mask B for the hand foreground in the current frame. k (x, y), the pixel category is determined using the following formula:

[0090]

[0091] B k (x,y)=1 represents the pixel region that has been verified as a hand after skeletal point matching. B k (x,y)=0 corresponds to the desktop and equipment background. Finally, the mask B can be smoothly segmented using the hand kinematics constraint matrix. k (x,y) further optimizes the accuracy of the hand-cut boundary.

[0092] (2-3) Head motion data acquisition. The IMU is used to continuously acquire raw inertial data of the user's head in three axes of acceleration and angular velocity; a hardware edge-triggered synchronization algorithm is applied to bind sampling timestamps to each set of inertial data; according to the IMU-visual timestamp mapping protocol, cross-modal temporal matching of inertial data with eye movement and hand image data is completed to generate a time-unified head inertial data sequence and corresponding timestamps.

[0093] (3) Feature extraction. Weighted least squares elliptic and subpixel fitting algorithms were used to process long and short exposure images respectively to extract the coordinates of pupil and corneal reflection points; the target detection model and geometric discrimination method were used to extract the three-dimensional coordinates of hand joints and complete the chirality determination; the visual inertial odometry processing pipeline was used to fuse the image and inertial data to calculate the 6DoF pose sequence of the XR head-mounted display.

[0094] (3-1) Eye feature extraction. The weighted least squares elliptic fitting algorithm is used to process long exposure images, segment the pupil and iris boundaries, and calculate the pixel coordinates of the pupil center; the four-neighbor Gaussian subpixel fitting algorithm is used to process short exposure images, and the peak value of the surface is taken as the coordinates of the corneal reflection point; after the feature calculation is completed, the image cache is uniformly reclaimed and the running memory is released to generate the inter-frame temporally aligned sequence of pupil center and spot center coordinates.

[0095] (3-2) Hand Feature Extraction and Chirality Determination. The YOLOv7-m lightweight object detection model was used to detect hand region images, and the two-dimensional coordinates of the palm and the oriented bounding box (OBB) parameters of the hand were calculated. After image cropping and pose correction based on the OBB, the FastMETRO hand pose estimation model was used to extract the 3D coordinates of 21 hand joints, and the pose validity was verified by combining the hand kinematic constraint matrix. Spatial reference vectors were constructed based on the index and middle finger MCPs, and the left and right hand nature was determined using the phalangeal angle sign statistical method, generating a three-dimensional coordinate sequence of hand joints and a chirality label sequence. The specific steps for generating hand features and chirality labels are as follows:

[0096] Ⅰ: Mask the hand foreground in frame k. k Inputting (x,y) into the YOLOv7-m detection model, the model detects hand targets based on effective hand pixels within the mask and outputs hand orientation OBB parameters, including center coordinates (o,y). x,k ,o y,k Width w k Height h k Deflection angle φ k The pixel boundary range of the cropped image is calculated using the following formula:

[0097]

[0098] Based on OBB boundary [x min,k , x max,k ]、[y min,k , y max,k Rotation correction and partial cropping of the original image yield a local image patch I containing only the hand. k .

[0099] II: The FastMETRO hand pose estimation model was used to extract the hand contour edges, finger bone orientation, and joint spatial positions. (Based on I...) k Using this as input, we obtain the real-time 3D coordinates of 21 hand skeleton nodes in the current frame. , i∈{201,202,213,215,…,256}, ultimately forming the set of hand spatial features of the current gesture posture. .

[0100] III: Select F k Inner wrist root node index finger MCP joint The middle finger's MCP joint Construct two sets of spatial reference vectors - , - This yields the cross product of the two vectors. .

[0101] IV: with c z As a basis for determining the chiral spatial orientation relationship, the chiral label for this frame is obtained using the following formula. k :

[0102] .

[0103] (3-3) Head pose calculation. The SuperVINS visual inertial odometry processing pipeline is used to process RGB images and IMU inertial data. The SuperPoint feature extraction network is called to obtain sparse corner points and edge features of the image. The LightGlue feature matching network is used to complete the pairing, and the LK (Lucas-Kanade) optical flow algorithm is used to track feature motion. Standard closed pre-integration is performed on the IMU data, and nonlinear equations are constructed and iteratively solved by combining visual reprojection error and IMU pre-integration residuals to obtain the 6DoF pose sequence in the XR head-mounted display coordinate system.

[0104] (4) Motion trend prediction. Combining the corneal spherical reflection optical model and head pose and angle compensation algorithm, the real-time gaze direction vector of the user is calculated; relying on joint constraints, forward and reverse kinematics algorithm and time series model are used to predict the short-term three-dimensional coordinates of each joint of the hand and fingertips; Neural Extended Kalman Filter (NEKF) and linear extrapolation method are used to obtain the predicted gaze direction.

[0105] (4-1) Fixation of gaze direction. Combining the spatial position of the infrared light source, and based on the pupil and light spot coordinates with the 6DoF head pose sequence, the three-dimensional coordinates of the corneal curvature center are solved using a corneal spherical reflection optical model. Connecting the corneal center and the pupil center yields the optical axis of the eyeball. Combining the Kappa angle parameters, the Rodrigues rotation formula is used to compensate for optical axis deviation, obtaining the result based on... Figure 5 The real-time gaze direction vector of the XR headset's reference coordinate system.

[0106] (4-2) Future gaze prediction. NEKF is used to process the user's continuous multi-frame real-time gaze coordinate sequence, adaptively correct process noise, optimize eye movement parameters, and obtain the current gaze point and eye movement rate; the sliding window linear extrapolation method is used to extrapolate the three-dimensional spatial coordinates of the gaze point at future moments to obtain the predicted gaze trajectory.

[0107] (4-3) Hand motion prediction. Based on the three-dimensional coordinates of the hand joints and the limit constraints of each joint's movement, the inverse kinematics algorithm is used to solve for the joint rotation angles; the joint angular velocity and angular acceleration are calculated using the inter-frame difference method, and the joint angles in future frames are inferred using a motion timing prediction model; the forward kinematics algorithm is used to map along the skeletal chain level by level to generate the predicted three-dimensional coordinates of all 21 skeletal nodes in the next frame, predicting the hand movements that the user is about to perform. The specific steps for predicting the three-dimensional coordinates of future hand skeletal nodes (including fingertips and phalanges) are as follows:

[0108] I: Taking 201 as the starting point of the five-finger kinematic chain, and combining the calibrated link lengths, fixed torsion angles, and joint physiological rotation constraints, inverse kinematic geometry is used to perform segmented execution of each link of the five fingers, solving for the bending angle θ of each joint in the current t-th frame. i,t For i∈{201,202,213,215,…,256}, the formula is as follows:

[0109]

[0110] Where p is the node number on the wrist side of the same finger chain, and e p,t Let e ​​be the link unit vector pointing from node p to node i. i,t Let i be the link unit vector pointing from node i to the adjacent node on the side closest to the fingertip.

[0111] II: Retrieve the rotation angle of the same joint in two consecutive frames t-1 and t, and calculate the angular velocity and angular acceleration using the following formulas to quantify the speed of hand movements and the trend of finger flexion and extension, thereby capturing the dynamic characteristics of the movement changes:

[0112]

[0113] Where Δt is a fixed time interval between frames. The historical joint angles, angular velocities, and angular accelerations from multiple consecutive frames are concatenated into a temporal feature sequence, which is then input into a specific motion temporal prediction model. The output is the predicted joint angle for the (t+1)th frame, satisfying the range of motion of human joints. It can predict the user's hand gestures in the next frame.

[0114] III: Will Substituting the forward kinematics recursive formula, the spatial position of each link is recursively calculated layer by layer from 201 towards the fingertip, solving for the predicted 3D coordinates of all 21 skeletal nodes in frame t+1. Restore the hand after prediction

[0115] Partial spatial form:

[0116]

[0117] Where l is the link length, α is the link twist angle, and R is the third-order joint rotation matrix.

[0118] (5) Interaction control. A spatiotemporal reference sequence is constructed by combining short-term gaze coordinates and hand prediction coordinates. The directed OBB intersection detection algorithm is used to determine the target interaction object and obtain the ID and interaction priority of the selected object. The activation threshold of the selected object is adaptively adjusted according to the interaction priority to avoid false triggering. The distance between the fingertip and the target object and the ID consistency are verified, and double-click confirmation interaction is triggered to generate interaction event data.

[0119] (5-1) Intent Object Determination. Based on the predicted gaze point coordinates and the predicted 3D coordinates of the fingertips, a spatiotemporal reference sequence is constructed. A directed OBB intersection detection algorithm is used to determine the intersection status of the gaze, fingertips, and objects in the enhanced learning process. If the intersection point stably points to the same object for multiple consecutive frames, and the number of dwell frames reaches a preset threshold, the object is determined to be the user's target interaction object, and the interaction object ID, spatial coordinates, and interaction priority are determined. The specific steps for determining an object as the user's target interaction object are as follows:

[0120] Ⅰ: Construct the gaze ray r based on the predicted gaze point and gaze direction in the (t+1)th frame. eye Using the coordinates of the five fingertips , i∈{216,226,236,246,256}, finger bone orientation constructs fingertip detection line segment r hand The expressions for the two detection vectors are shown in the following formulas:

[0121]

[0122] Where k and m are the extension length coefficients, To predict the gaze coordinates, u eye The unit vector of the gaze direction. Predicting 3D coordinates for the five fingertips, e tip Let be the unit vector of the finger bone link.

[0123] II: Calculate vector r using the following formula eye and r hand The included angle Determine whether the direction of the person's gaze and the direction of the hand reaching towards the target are consistent:

[0124]

[0125] If φ is less than 30°, it means that both are pointing to the same area, and the intersection detection process begins.

[0126] III: Extract the OBB parameters of all virtual interactive objects in the scene, including the three axes of length, width, and height a, b, and c, and the center coordinate o(o). x ,oy ,o z Using the ray OBB intersection determination function, r is obtained. eye The intersection interval t with the interaction object OBB min t max As shown in the formula below:

[0127]

[0128] Among them, (G) x G y G z (u) represents the predicted gaze point coordinates on the three axes. x ,u y ,u z ) is the unit vector of the gaze direction. When t min <t max When, it represents r eye Intersect with the interactive object OBB, and then use the fingertip detection line segment to perform a second verification of the intersecting object. The object that intersects with both the line of sight and the fingertip is the candidate interactive object.

[0129] IV: Count the number of consecutive frames in which the gaze ray and the fingertip line segment intersect among the candidate interactive objects. When it exceeds 12 frames, it is determined to be the target interactive object, and its ID, spatial coordinates and interaction priority are output.

[0130] (5-2) Dynamic threshold adjustment. Based on the selected object ID and interaction priority parameters, an adaptive dynamic distance threshold algorithm is used to lower the activation threshold of the selected object to adapt to the fine-tuning, rotation, knob, and button click operations of the virtual scene; at the same time, the trigger threshold of non-selected objects is increased to avoid false triggers caused by the user's gaze scanning or slight hand tremors, and the list of effective activation thresholds for interactive objects is updated.

[0131] (5-3) Double confirmation trigger. Based on the predicted three-dimensional coordinates of the fingertip, the dynamic activation threshold of the object, and the ID of the selected object, the Euclidean distance between the predicted fingertip and the center of the bounding box of the selected object is calculated frame by frame. When the distance is less than the dynamic threshold of the object, and the ID of the object is consistent with the interactive object ID jointly determined by the gaze and hand, the interaction condition is determined to be met. Then, the preset action instructions of clicking, sliding, and rotating are executed to complete the double confirmation interaction of eye and hand and generate interaction event data.

[0132] (6) Visual output. Based on the predicted gaze point coordinates and XR head-mounted display parameters, the rendering center point is determined, and a high-definition image is generated using gaze-driven Region of Interest (ROI) multi-level rendering technology and dynamic field of view radius adjustment algorithm; super-resolution, contrast enhancement and background blurring are performed on low-resolution, low-contrast and occluded areas; dangerous actions are predicted through collision detection, and safety protection is achieved by combining icon warning and interactive response freeze algorithm.

[0133] (6-1) Foveatory Rendering. Combining the predicted foveatory coordinates with XR headset parameters, the rendering center point of the entire image is determined; foveatory-driven ROI multi-level resolution rendering technology is used to divide the virtual scene into three types of ROI rendering areas: such as... Figure 6 As shown, the core area of ​​focus is the blackboard, the podium, and all the experimental equipment on it, rendered at high resolution; the teaching content area is the blackboard and all the students' desks and chairs, rendered at medium resolution; the outer background area is the door, walls, and ceiling, rendered at basic resolution; a dynamic view radius adjustment algorithm is used to adaptively adjust the high-definition area range according to the user's viewing speed, rendering the virtual objects that the user is focusing on in high definition, and generating a layered composite image.

[0134] (6-2) Visual Enhancement. Based on the predicted user gaze point coordinates, the target area of ​​the image is located. If the pixel size of the area is lower than the preset recognizable threshold, the image contrast is insufficient, or there is foreground object occlusion, the enhancement logic is automatically triggered; pixel-level local super-resolution, adaptive contrast enhancement, and foreground / background blurring are executed sequentially to highlight key teaching information and generate enhanced image frames, such as... Figure 7 As shown, 701 is the original image, and 702 is the enhanced image. The specific steps for visual enhancement are as follows:

[0135] Ⅰ: Project the interactive object onto a two-dimensional imaging plane, crop to obtain a local image, where I(x,y) is the original pixel grayscale value at (x,y). Use bicubic interpolation with the following formula to magnify the details of the interactive object and restore its operable surface structure:

[0136]

[0137] Where u and v are the horizontal and vertical pixel offsets of I(x,y), respectively. When u takes -1, 0 and 1 in sequence, v completely traverses -1, 0 and 1, and the combination results in 9 sets of neighborhood coordinates, forming a 3×3 pixel neighborhood.

[0138] II: The pixel brightness and darkness values ​​are redistributed using the following weighted calculation formula:

[0139]

[0140] in, isI new (x,y) represents the average gray level of all pixels, indicating the average brightness of the interactive object area.

[0141] III: Use the following formula for 3×3 mean filtering to smooth and blur non-interactive background pixels in the image:

[0142]

[0143] Among them, I bg (x,y) represents the grayscale values ​​of the blurred background pixels, I all To determine the pixel grayscale values ​​of the complete scene image, I out (x,y) and I bg (x,y) is used to separate and stitch the images, and the final enhanced image frames are output.

[0144] (6-3) Safety Warning. Based on the predicted hand coordinates and the predicted gaze point coordinates, a sweeping collision detection algorithm is used to predict collisions with the pre-marked danger zone enclosing body within the virtual teaching scene. If it is determined that the fingertip is about to enter the danger zone and the gaze direction is not aligned with that zone, a high-frequency triangular warning icon is added to the center of the gaze point using real-time layer overlay rendering technology, such as... Figure 8 As shown, the interactive response freeze algorithm is enabled to restrict hand operations, thereby achieving real-time security protection for virtual experimental scenarios.

[0145] (7) Teaching data management. ROI is divided and weighted parameters are configured. User gaze duration is accumulated and corrected for pupil standard deviation to obtain focus score and level. Interaction event data is collected and serialized for storage. Incremental hashing algorithm is used to complete data verification and deduplication. Log files are retrieved on the teacher's end to reconstruct the 3D scene. Encryption technology is used to ensure data security. Track playback, breakpoint continuation of learning, encrypted sharing links, and restricted distribution of logs are supported.

[0146] (7-1) Attention Assessment. Differentiated weight parameters are configured for the ROIs in the virtual classroom. Based on the real-time 3D coordinates of the fixation points frame by frame, the fixation points are matched to the corresponding ROIs, and the effective fixation time of each region is accumulated. Weighted correction is then performed using the standard deviation of pupil diameter. Normalized attention scores and attention level labels are obtained to analyze students' learning status. The specific steps for dividing attention level labels are as follows:

[0147] I: Three types of ROI regions are defined: the fixation core region, the teaching content region, and the peripheral background region, with pre-set teaching association weights ω1, ω2, and ω3 respectively. Based on the type of teaching segment, region-independent adaptation coefficients are configured: λ1 = 1.5 for the fixation core region during the explanation segment, λ2 = 0.9 for the teaching content region during the practice segment, and λ3 = 0.5 for the peripheral background region. The dynamic weights W1, W2, and W3 for the three regions are calculated using the following formula:

[0148]

[0149] II: A valid fixation is defined as when a user continuously fixates on the same ROI for 12 frames. The valid fixation durations T1, T2, and T3 for the three ROI types are calculated, where T is the total duration of the entire evaluation. Simultaneously, the standard deviation of pupil diameter σ and the fixation point switching frequency r are collected during the current evaluation period. The attention score S is calculated using the following formula:

[0150]

[0151] Where σ0 and r0 are the reference values ​​for pupil diameter fluctuation and fixation point switching frequency, respectively, α is the pupil fluctuation attenuation coefficient, and β is the gaze jump attenuation coefficient.

[0152] III: Based on the obtained focus score S, the focus level label L is constructed using the following formula.

[0153]

[0154] (7-2) Interaction log storage. When an eye-hand coordination interaction event is triggered, collect the event timestamp, target object ID, predicted gaze direction, hand joint angle, and chirality label. k The current teaching segment is coded; the data is structured and serialized according to Protocol Buffers format, written to a local flash log file and asynchronously uploaded to a cloud storage server; each record uses an incremental hash algorithm to generate a checksum, realizing data integrity verification and duplicate data filtering.

[0155] (7-3) Teaching Replay and Resume Learning from Breakpoints. Teachers can pull log files from the cloud server to reconstruct a 3D virtual scene locally, restore the user's gaze trajectory and hand movements along the original timeline, and support playback operations such as pause, speed adjustment, and keyframe jump. Local logs are stored using the AES-256 advanced encryption standard, and the cloud transmission link uses the TLS protocol to ensure the security of teaching data. When students exit the course, a breakpoint snapshot is automatically generated, and users can resume their learning status upon logging in again. Teachers can create encrypted sharing links with passwords and expiration dates to achieve restricted distribution of logs.

[0156] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0157] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An enhanced teaching interaction method based on eye-hand coordination prediction, characterized in that... The method includes the following steps: (1) Initialize calibration subdivision, start XR enhanced teaching application, configure eye movement detection parameters, calibrate user's eye movement parameters, and fit and generate user-specific eye model, and encapsulate and save the model parameters; construct the topology of five fingers bones based on human hand features, calculate and generate hand kinematic constraint matrix and initial posture data of all joints; complete the calibration of multiple cameras and inertial measurement units, coordinate system one, and time synchronization, generate external parameter transformation matrix and timestamp configuration file, and construct a unified spatiotemporal reference for multimodal data acquisition and pose calculation; (2) Multimodal data acquisition: long and short exposure infrared images are acquired in time-division using an infrared eye-tracking unit and time-series alignment is achieved to generate long and short exposure image sequences with unified timestamps; images are acquired using an RGB camera and semantic segmentation is performed, and time-series aligned hand region image sequences and environmental background image sequences are generated after synchronizing frame time; head inertial data is acquired through an IMU, and cross-modal time-series matching of inertial data with eye-tracking and hand image data is completed to generate a time-series unified head inertial data sequence and corresponding timestamps; (3) Feature extraction: Weighted least squares ellipse and subpixel fitting algorithms are used to process long and short exposure images respectively to extract the coordinates of pupil and corneal reflection points; the target detection model and geometric discrimination method are used in combination to extract the three-dimensional coordinates of hand joints and complete the chirality determination; the visual inertial odometry processing pipeline is used to fuse images and inertial data to calculate the 6DoF pose sequence of the XR head-mounted display. (4) Motion trend prediction: Combine the corneal spherical reflection optical model and head position and angle compensation algorithm to calculate the user's real-time gaze direction vector; Based on joint constraints, use forward and reverse kinematics algorithm and time series model to predict the short-time three-dimensional coordinates of each joint of the hand and fingertips; Use neural extended Kalman filter and linear extrapolation method to obtain the predicted gaze direction. (5) Interactive control: Construct a spatiotemporal reference sequence by combining short-term gaze direction and hand prediction coordinates, use a directed OBB intersection detection algorithm to determine the target interactive object, obtain the ID and interaction priority of the selected object; adaptively adjust the activation threshold of the selected object according to the interaction priority to avoid false triggering; verify the distance between the fingertip and the target object and the consistency of the ID, trigger double-click to confirm the interaction and generate interactive event data; (6) Visual output: Based on the predicted gaze direction and XR head-mounted display parameters, the rendering center point is determined. High-definition images are generated by using gaze-driven region of interest multi-level rendering technology and dynamic field radius adjustment algorithm. Super-resolution, contrast enhancement and background blurring are performed on low-resolution, low-contrast, and occluded areas. Dangerous actions are predicted through collision detection and safety protection is achieved by combining icon warning and interactive response freezing algorithm. (7) Teaching data management: divide ROI and configure weighting parameters, accumulate user gaze duration, and obtain focus score and level after pupil standard deviation correction; collect interactive event data and complete serialization storage, and use incremental hash algorithm to complete data verification and deduplication; the teacher retrieves log files, reconstructs three-dimensional scene, uses encryption technology to ensure data security, and supports trajectory playback, breakpoint continuation learning and encrypted sharing links and restricted log distribution.

2. The method according to claim 1, characterized in that... The initial calibration subdivision described in step (1) specifically includes: (1-1) Initialize the eye tracking channel, start the XR-enhanced teaching application, and configure the eye recognition threshold, effective eye detection area, and eye pose calculation benchmark parameters in sequence; calibrate the user's corneal curvature, pupil size, anterior chamber depth, and Kappa angle eye parameters; refer to the publicly available standard eye dataset, fit and generate a user-specific eye mathematical model, and encapsulate the model parameters into a local configuration file; (1-2) Initialization of the hand tracking channel: Based on the anatomical parameters of the human hand, determine the rotation axis direction, motion freedom, and physiological motion limits of the left and right palmar interphalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints of the user in front of the camera; with the wrist as the topological root node, construct the topological structure of the five fingers bones using hierarchical chain kinematic rules, and calculate and generate the hand kinematic constraint matrix and the initial posture data of all joints; (1-3) Eye-hand coordination reference calibration: Zhang Zhengyou calibration method is used to uniformly solve the lens distortion parameters of the devices including infrared eye-tracking camera, visible light RGB camera, depth sensor and IMU; cross-device coordinate system transformation is completed through hand-eye joint calibration algorithm, and all sensing devices are uniformly mapped to the XR head center reference coordinate system; the sampling time sequence of multiple devices is aligned with the hardware synchronization unit, and external parameter transformation matrix and timestamp configuration file are generated to construct a unified spatiotemporal reference for multimodal data acquisition and pose calculation.

3. The method according to claim 1, characterized in that... The multimodal data acquisition in step (2) specifically includes: (2-1) Eye-tracking data acquisition: Based on the configuration parameters, the time-slice timing scheduling algorithm is used to drive the infrared eye-tracking unit to alternately acquire long and short exposure infrared images. Long exposure images are used to capture the iris and pupil boundary, and short exposure images are used to locate the infrared reflection spot on the corneal surface. The global clock is used to mark the timestamp frame by frame, and the time interpolation frame registration algorithm is used to complete the temporal alignment of images with different frame rates, generating a sequence of long and short exposure images with a unified timestamp. (2-2) Hand data acquisition: A visible light RGB camera is used to continuously acquire images of the user's hand, experimental table and experimental equipment in the virtual scene; U-Net3+ full-scale semantic segmentation network is used to distinguish the hand foreground from the table and experimental equipment background; a global clock is used to complete frame timing synchronization and generate timing-aligned hand region image sequence and environmental background image sequence. (2-3) Head motion data acquisition: IMU is used to continuously acquire raw inertial data of the user's head in three axes of acceleration and angular velocity; hardware edge-triggered synchronization algorithm is applied to bind sampling timestamps to each set of inertial data; according to the IMU-visual timestamp mapping protocol, cross-modal temporal matching of inertial data with eye movement and hand image data is completed to generate a time-unified head inertial data sequence and corresponding timestamps.

4. The method according to claim 1, characterized in that... The feature extraction described in step (3) specifically includes: (3-1) Eye feature extraction: The weighted least squares ellipse fitting algorithm is used to process long exposure images, segment the pupil and iris boundaries, and calculate the pixel coordinates of the pupil center; the four-neighbor Gaussian subpixel fitting algorithm is used to process short exposure images, and the peak value of the surface is taken as the coordinates of the corneal reflection point; after the feature calculation is completed, the image cache is uniformly recycled and the running memory is released to generate the inter-frame temporally aligned sequence of pupil center and spot center coordinates. (3-2) Hand feature extraction and chirality determination: The YOLOv7-m lightweight target detection model was used to detect the hand region image and calculate the two-dimensional coordinates of the palm and the bounding box parameters of the hand orientation. After image cropping and pose correction based on OBB, the FastMETRO hand pose estimation model was used to extract the 3D coordinates of 21 hand joints and the pose legality was verified by combining the hand kinematic constraint matrix. Spatial reference vectors were constructed based on the metacarpophalangeal joints of the index and middle fingers. The left and right hand nature was determined by the phalangeal angle sign statistical method, and the three-dimensional coordinate sequence of hand joints and the chirality label sequence were generated. (3-3) Head pose calculation: The SuperVINS visual inertial odometry processing pipeline is used to process RGB images and IMU inertial data. The SuperPoint feature extraction network is called to obtain sparse corner points and edge features of the image. The LightGlue feature matching network is used to complete the pairing, and the LK optical flow algorithm is used to track feature motion. Standard closed pre-integration is performed on the IMU data. The nonlinear equation is constructed by combining visual reprojection error and IMU pre-integration residual to solve iteratively and obtain the 6DoF pose sequence in the XR head-mounted display coordinate system.

5. The method according to claim 1, characterized in that... The motion trend prediction mentioned in step (4) specifically includes: (4-1) Fixation of gaze direction: Based on the spatial position of the infrared light source, the coordinates of the pupil and light spot and the 6DoF pose sequence of the head, the three-dimensional coordinates of the corneal curvature center are solved using the corneal spherical reflection optical model; the optical axis of the eyeball is obtained by connecting the corneal center and the pupil center, and the optical axis deviation is compensated by using the Rodriguez rotation formula in combination with the Kappa angle parameter, so as to obtain the user's real-time gaze direction vector. (4-2) Future gaze prediction: NEKF is used to process the user's continuous multi-frame real-time gaze direction sequence, adaptively correct process noise, optimize eye movement parameters, and obtain the current gaze state and movement rate; the sliding window linear extrapolation method is used to extrapolate the gaze angle at future moments and obtain the predicted gaze direction. (4-3) Hand movement prediction: Based on the three-dimensional coordinates of the hand joints and the limit constraints of each joint's movement, the inverse kinematics algorithm is used to solve the joint rotation angle; the joint angular velocity and angular acceleration are calculated by the inter-frame difference method, and the joint angle of future frames is inferred by the motion time-series prediction model; the forward kinematics algorithm is used to map along the skeletal chain step by step to generate the predicted three-dimensional coordinates of the fingertip and each phalanx, and to predict the hand movements that the user is about to perform.

6. The method according to claim 1, characterized in that... The interactive control described in step (5) specifically includes: (5-1) Intent object determination: Based on the predicted gaze direction and combined with the fingertip predicted three-dimensional coordinates obtained in step (4), a spatiotemporal reference sequence is constructed; the directed OBB intersection detection algorithm is used to determine the intersection state of the gaze, fingertip and the object in the enhanced teaching process; if the intersection point points to the same object stably for multiple consecutive frames and the number of dwell frames reaches the preset threshold, the object is determined to be the user's target interaction object, and the interaction object ID, spatial coordinates and interaction priority are determined; (5-2) Dynamic threshold adjustment: Based on the selected object ID and interaction priority parameters, an adaptive dynamic distance threshold algorithm is used to reduce the activation threshold of the selected object to adapt to the fine-tuning, rotation, knob and button click operations of the virtual scene; the trigger threshold of non-selected objects is simultaneously amplified to avoid false triggers caused by the user's gaze scanning or slight hand tremors, and the list of effective activation thresholds of interactive objects is updated. (5-3) Double confirmation trigger: Based on the predicted three-dimensional coordinates of the fingertip, the dynamic activation threshold of the object, and the ID of the selected object, calculate the predicted Euclidean distance between the fingertip and the center of the bounding box of the selected object frame by frame; when the distance is less than the dynamic threshold of the object, and the ID of the object is consistent with the interactive object ID jointly determined by the gaze and hand, the interaction condition is determined to be met, and the preset action instructions of clicking, sliding, and rotating are executed to complete the double confirmation interaction of eye and hand, and generate interaction event data.

7. The method according to claim 1, characterized in that... The visual output of the image in step (6) specifically includes: (6-1) Gaze-based rendering: Combining the predicted gaze direction with XR head-mounted display parameters, the rendering center point of the entire image is determined; gaze-driven ROI multi-level resolution rendering technology is adopted to divide the image into a gaze core area, a teaching content area, and an outer background area, and match high, medium, and basic resolutions respectively; a dynamic field of view radius adjustment algorithm is used to adaptively adjust the high-definition area range according to the user's gaze speed, render the virtual objects that the user is focused on in high definition, and generate a layered composite image; (6-2) Visual enhancement: Based on the predicted user gaze direction, locate the target area of ​​the screen. If the pixel size of the area is lower than the preset recognizable threshold, the screen contrast is insufficient, or there is foreground object occlusion, the enhancement logic is automatically triggered. Pixel-level local super-resolution, adaptive contrast enhancement, and foreground and background blurring are executed in sequence to highlight key teaching information and generate enhanced screen frames. (6-3) Safety warning: Based on the predicted hand coordinates and predicted gaze direction, a sweeping collision detection algorithm is used to predict the collision with the pre-marked danger zone surrounding the virtual teaching scene. If it is determined that the fingertip is about to enter the danger zone and the gaze direction is not aligned with the area, a high-frequency flashing warning icon is added to the center of the gaze point through real-time layer overlay rendering technology. The interactive response freeze algorithm is enabled to restrict hand operation and realize real-time safety protection of the virtual experimental scene.

8. The method according to claim 1, characterized in that... The teaching data management described in step (7) specifically includes: (7-1) Attention assessment: In the virtual experimental scene, ROIs are pre-divided and differentiated weight parameters are configured. Based on the real-time gaze direction frame by frame, the three-dimensional coordinates of the gaze point are obtained through scene intersection operation. The gaze point is matched to the corresponding ROI, and the effective gaze duration of each region is accumulated. The weighted correction is completed by combining the standard deviation of pupil diameter. The normalized attention score and attention level label are obtained to analyze the student's learning status. (7-2) Interaction log storage: When an eye-hand coordination interaction event is triggered, the event timestamp, target object ID, predicted gaze direction, hand joint angle and chirality label, and current teaching segment code are collected; the data is structured and serialized in Protocol Buffers format, written to a local flash log file and asynchronously uploaded to a cloud storage server; each record uses an incremental hash algorithm to generate a check code to achieve data integrity verification and duplicate data filtering; (7-3) Teaching Replay and Resume Learning from Breakpoints: Teachers can pull log files from the cloud server to reconstruct a 3D virtual scene locally, restore the user's gaze trajectory and hand movements along the original timeline, and support playback operations such as pause, speed adjustment, and keyframe jump; local logs are stored using the AES-256 advanced encryption standard, and the cloud transmission link uses the TLS protocol to ensure the security of teaching data; when students exit the course, a breakpoint snapshot is automatically generated, and users can resume their learning status by logging in again; teachers can create encrypted sharing links with passwords and expiration dates to achieve restricted distribution of logs.

9. An enhanced teaching interaction system based on eye-hand coordination prediction, characterized in that, The method for implementing the method in any one of claims 1 to 8 includes an initialization calibration subdivision module, a multimodal data acquisition module, a feature extraction module, a motion trend prediction module, an interactive control module, a visual output module, and a teaching data management module. The initialization calibration subdivision module completes the calibration of eye-tracking, hand, and head sensor parameters, constructs a dedicated model and skeletal topology, and unifies the coordinate system and sampling timing of multiple devices; The multimodal data acquisition module acquires eye-tracking images, hand images, and head inertial data, completes the temporal alignment of multi-source data, and generates a standard data sequence with timestamps; The feature extraction module extracts the coordinate information of the pupil and corneal reflection points, calculates the three-dimensional coordinates of the hand joints and determines the hand nature, and fuses the data to solve the 6DoF pose of the XR head-mounted display; The motion trend prediction module combines an optical model to calculate the real-time gaze direction, relies on a time-series model to predict the hand movement posture, and uses a filtering algorithm to deduce the subsequent gaze angle. The interaction control module determines the interaction object and classifies the interaction priority, adaptively adjusts the interaction trigger threshold, verifies the information, and completes the eye-hand coordination interaction trigger. The image visual output module performs multi-level rendering based on the gaze area, performs super-resolution and contrast enhancement on the image, detects dangerous areas, and achieves experimental safety protection. The teaching data management module counts gaze duration to assess learning focus, verifies, encrypts, and stores interaction logs, and enables trajectory playback, breakpoint continuation of learning, and log sharing.