Methods and systems for automatic adaptation of teaching atmosphere in immersive learning environments

Abstract

This invention belongs to the field of virtual reality teaching applications, and provides a method and system for automatically adapting the teaching atmosphere in an immersive teaching environment. The method includes: eye-tracking image acquisition; gaze point recognition; ROI tracking; ROI boundary fusion; object adaptive optimization; ROI brightness adjustment; and gaze object interaction. This invention can help improve the immersion and interactivity of future classrooms, enhance the user experience of immersive virtual environments, and promote the deep integration of intelligent teaching environments with users' own cognition.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality teaching applications, specifically relating to a method and system for automatically adapting the teaching atmosphere in an immersive teaching environment. Background Technology

[0002] With the iterative development of cutting-edge information technologies such as generative artificial intelligence, virtual reality, and mixed reality, the teaching environment is rapidly transforming towards intelligence and immersion. This not only expands the immersive experience and natural interaction functions of traditional teaching but also requires the deep integration of multimodal perception, intelligent computing, and educational context. By establishing a dynamic mapping between physical and virtual spaces, and accurately collecting and analyzing learners' visual trajectories and interactive behaviors, we can capture learners' attention distribution and cognitive preferences, driving education from experience-guided to data-driven. Therefore, breakthroughs in key technologies such as accurate eye-tracking data collection, real-time gaze behavior response, multi-object management, and hierarchical feedback scheduling can fully unleash the educational value of immersive teaching, significantly improving learning efficiency and achieving personalized instruction.

[0003] However, existing immersive teaching systems still have many bottlenecks: (1) In complex lighting or multi-person interactive teaching scenarios, eye-tracking data is easily affected by noise, and the lack of a stable acquisition and preprocessing mechanism leads to insufficient accuracy error in the three-dimensional mapping of gaze points; (2) The linkage between virtual and real scenes relies heavily on preset scripts, and no dynamic response logic based on the user's real-time gaze area and points of interest has been established, making it impossible to achieve accurate tracking of attention and atmosphere adaptation; (3) The efficiency of recognizing and managing multiple objects in the teaching environment is low, especially when facing new teaching tools, lacking the ability to quickly collect samples, adapt models, and schedule states; (4) The feedback mechanism lacks clear priority hierarchy, and feedback conflicts or delays are prone to occur when multiple users interact, which damages the immersive teaching experience and the continuity of the process. These problems restrict the implementation effect of automatic adaptation of immersive teaching atmosphere. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and system for automatic adaptation of teaching atmosphere in immersive teaching environment, which provides a new, intelligent and systematic interactive method for immersive teaching and can help improve the immersion and interactivity of future classrooms.

[0005] The objective of this invention is achieved through the following technical measures.

[0006] This invention provides a method for automatically adapting the teaching atmosphere in an immersive teaching environment, comprising the following steps:

[0007] (1) Eye-tracking image acquisition: The user's eye images are captured using the built-in binocular camera of the head-mounted display. The pupil center is calibrated using a coordinate regression algorithm, and the coordinates of the left and right pupils are calculated. Median filtering is used to remove the jump noise of the two-dimensional state vector of the pupil, and Kalman filtering is used to smooth the gaze point trajectory. Based on the pre-trained DLT model, the pixel coordinates of the user's gaze point are converted into a three-dimensional gaze direction vector.

[0008] (2) Gaze point recognition; collect images of the user's field of vision of the head-mounted display, input them into the lightweight object detection model, assign the teaching object with the highest probability instance category to the confidence level higher than the threshold; generate a rectangular window perpendicular to the user's gaze direction, and adaptively adjust the window size according to the head position and posture; calculate the priority of the gaze object, correct the object's coordinates and visibility status, and prioritize the detection and rendering of high-level objects.

[0009] (3) ROI (Region of Interest) tracking: Based on the user's gaze duration and frequency, filter points of interest and write their coordinates, posture and attention area into the behavior analysis database; obtain the instance category of the virtual object where the point of interest is located, cluster the points of interest with high similarity, and use circles to represent ROI; based on the correlation of the teaching knowledge graph, search for virtual objects of the same instance category in the virtual scene and recommend the ROI where the above virtual objects are located to the user.

[0010] (4) ROI boundary fusion: Extract image blocks of the gaze region, use the Cannny operator to detect image edges, extract effective contour information, and write it into the ROI boundary cache; analyze the boundary cache of effective ROIs, intelligently merge ROIs with high correlation; count the cumulative gaze duration and frequency of users, generate a heat map, and weight and superimpose it onto the ROI, dynamically adjust the ROI boundary radius along the user gaze direction.

[0011] (5) Object adaptive optimization: detect the confidence of objects in the image, re-acquire object images with confidence below the threshold, and record their acquisition angle and lighting parameters; use window detection algorithm to reset the position and pose of virtual objects in the scene, and fully present the structure and appearance of the objects of interest; use fading operation to process the original position of virtual objects, correct scale and rotation deviations, and perform smooth transition optimization.

[0012] (6) ROI brightness adjustment: Extract the average brightness value within the ROI, and enter the global or local brightness adjustment mode based on the comparison result with the ideal brightness range; adjust the ROI exposure time and photosensitive gain according to the relationship with the upper and lower limits of the ideal range until the average brightness value enters the ideal range; adopt a phased transition strategy to dynamically adjust the global exposure time and gain coefficient to correct the global brightness.

[0013] (7) Gaze object interaction; detect the collision between the user's gaze and objects in the virtual scene, use a quadruple to record valid gazes with a gaze duration exceeding the threshold; find the interaction type label based on the gaze object ID, and trigger different interaction instructions based on the gaze duration; add the interaction instructions to the global queue in chronological order, schedule feedback according to priority, and execute the interaction.

[0014] The present invention also provides an automatic adaptation system for teaching atmosphere in an immersive teaching environment, including: an eye-tracking image acquisition module, a gaze point recognition module, a ROI tracking module, a ROI boundary fusion module, an object adaptive optimization module, a ROI brightness adjustment module, and a gaze object interaction module.

[0015] The eye-tracking image acquisition module is used to acquire the user's pupil coordinates, a smooth and stable gaze point trajectory, and output a three-dimensional gaze direction vector.

[0016] The gaze point recognition module is used to detect the instance category to which the object belongs, adaptively generate virtual windows, and calculate and render window objects with high priority.

[0017] The ROI tracking module is used to extract users' points of interest, cluster interest regions with high similarity, and recommend potentially relevant regions using knowledge graphs.

[0018] The ROI boundary fusion module is used to detect edges and thermal distribution, intelligently merge ROIs with high correlation, and adaptively correct ROI boundaries.

[0019] The object adaptive optimization module is used to acquire object images with low confidence, reset the position and pose of the occluded virtual object, and correct its scale and rotation deviation.

[0020] The ROI brightness adjustment module is used to determine the brightness deviation of the ROI, adjust the exposure time and photosensitivity gain, and correct local or global brightness.

[0021] The gaze object interaction module is used to detect collisions between the user's gaze and virtual objects, trigger corresponding collision feedback, and schedule high-priority collision events.

[0022] The beneficial results of this invention are as follows: Eye-tracking technology is used to capture eye images of the user viewing a head-mounted display; Kalman filtering is applied to smooth the gaze trajectory; the category of the teaching object focused on by the gaze point is identified; points of interest are selected based on the user's gaze duration and frequency, and ROIs are represented by circles; effective contour information of the ROI is extracted; the user's cumulative gaze duration and frequency are statistically analyzed to generate a heat map, which is then weighted and superimposed onto the ROI; the position and posture of the virtual object are reset to fully present the structure and appearance of the object of interest; the average brightness within the ROI is extracted; based on the comparison with the ideal brightness range, the average brightness of the ROI is adjusted using global or local brightness adjustment methods; collisions between the user's gaze and objects in the virtual scene are detected; interaction type tags are found according to the category of the gazed object; feedback is scheduled according to priority, and corresponding interactions are executed. With the widespread application of generative artificial intelligence and immersive technology in the education field, immersive teaching environments integrating multiple technologies are constantly emerging, and the need to provide more intelligent functions for virtual teaching environments is becoming increasingly urgent. This invention improves the user experience of immersive virtual environments and promotes the deep integration of intelligent teaching environments with the user's own cognition. Attached Figure Description

[0023] Figure 1 This is a flowchart of the framework of the automatic adaptation system for teaching atmosphere in an immersive teaching environment in this embodiment of the invention.

[0024] Figure 2 This is a schematic diagram of the user's gaze point in an embodiment of the present invention. 201 - Pupil gaze point in frame t-2, 202 - Pupil gaze point in frame t-1, 203 - Pupil gaze point in frame t, 204 - Pupil gaze point in frame t+1, 205 - User gaze point.

[0025] Figure 3 This is a schematic diagram of the teaching object detection frame in an embodiment of the present invention, 301-blackboard detection frame, 302-podium detection frame, 303-desk detection frame, 304-computer detection frame.

[0026] Figure 4 This is a schematic diagram of dynamic adjustment of the gaze point ROI in a virtual-real fusion teaching environment in this embodiment of the invention. 401 - Initial blackboard ROI, 402 - Initial podium ROI, 403 - Adjusted blackboard ROI, 404 - Adjusted podium ROI.

[0027] Figure 5 This is a schematic diagram of ROI image block extraction and region labeling in an embodiment of the present invention. 501-Blackboard ROI, 502-Platform ROI, 503-Table and chair ROI, 504-Aggregated ROI of blackboard and podium, 505-Aggregated ROI of podium and table and chair.

[0028] Figure 6This is a schematic diagram of virtual object reset and smooth transition in an embodiment of the present invention, 601-input layer, 602-feature extraction layer, 603-lightweight optimization layer, 604-output mapping layer.

[0029] Figure 7 This is a schematic diagram of local brightness adjustment in an embodiment of the present invention. 701 - Classroom view before brightness adjustment, 702 - Classroom view after the brightness of the podium is balanced.

[0030] Figure 8 This is a schematic diagram of collision detection between the gaze ray and the virtual object in an embodiment of the present invention. Ray 801 enters the three-dimensional bounding box of the blackboard, ray 802 exits the three-dimensional bounding box of the blackboard, ray 803 enters the three-dimensional bounding box of the podium, and ray 804 exits the three-dimensional bounding box of the podium. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0032] like Figure 1 As shown, this embodiment provides an automatic adaptation system for teaching atmosphere in an immersive teaching environment, including: an eye-tracking image acquisition module, a gaze point recognition module, an ROI tracking module, an ROI boundary fusion module, an object adaptive optimization module, an ROI brightness adjustment module, and a gaze object interaction module;

[0033] The eye-tracking image acquisition module is used to acquire the user's pupil coordinates, a smooth and stable gaze point trajectory, and output a three-dimensional gaze direction vector.

[0034] The gaze point recognition module is used to detect the instance category to which the object belongs, adaptively generate virtual windows, and calculate and render window objects with high priority.

[0035] The ROI tracking module is used to extract users' points of interest, cluster interest regions with high similarity, and recommend potentially related regions using knowledge graphs.

[0036] The ROI boundary fusion module is used to detect edges and thermal distribution, intelligently merge ROIs with high correlation, and adaptively correct ROI boundaries.

[0037] The object adaptive optimization module is used to acquire object images with low confidence, reset the position and pose of the occluded virtual object, and correct its scale and rotation deviation.

[0038] The ROI brightness adjustment module is used to determine the brightness deviation of the ROI, adjust the exposure time and photosensitivity gain, and correct local or global brightness.

[0039] The gaze object interaction module is used to detect collisions between the user's gaze and virtual objects, trigger corresponding collision feedback, and schedule high-priority collision events.

[0040] In addition, this embodiment also provides a method for automatically adapting the teaching atmosphere in an immersive teaching environment, including the following steps:

[0041] (1) Eye-tracking image acquisition. The user's eye images are captured using the built-in binocular camera of the head-mounted display. The pupil center is calibrated using a coordinate regression algorithm, and the coordinates of the left and right pupils are calculated. Median filtering is used to remove the jump noise of the two-dimensional state vector of the pupil, and Kalman filtering is used to smooth the gaze point trajectory. Based on the pre-trained DLT model, the pixel coordinates of the user's gaze point are converted into a three-dimensional gaze direction vector.

[0042] (1-1) Binocular Image Acquisition. The built-in camera of the head-mounted display device is activated at a frequency of 25Hz, and exposure compensation is enabled simultaneously to correct reflective areas. The frame synchronization interface is called to overlay a 1μs precision timestamp on each frame of the image. The format is <Device ID>_<Absolute Time>_<Frame Sequence>, for example, CamPair01_20250915T142045.017283_000025. The left and right image sequences are aligned according to the timestamp results, and the brightness parameters of the images are checked frame by frame. An image queue is constructed every 5 frames, and the image is input into a lightweight convolutional neural network to extract eye features. The pupil center is calibrated using a coordinate regression algorithm, and the coordinates of the left and right pupils are calculated.

[0043] (1-2) Gaze Point Trajectory Output. Based on the pupil coordinate changes, a two-dimensional state vector of the user's gaze position and head movement velocity is calculated. A median filter with a filtering window size of 3 frames is used to remove jump noise. The processed vector is then input into a Kalman filter model, where state prediction, covariance update, measurement residual calculation, and optimal estimation correction are performed sequentially to output the smoothed gaze point trajectory. The specific steps of gaze point processing are as follows:

[0044] I: Such as Figure 2 As shown, the pupil fixation points in frames t-2, t-1, t, and t+1 are 201, 202, 203, and 204, respectively, and are defined as P. t-2 (x t-2 y t-2 ), P t-1 (x t-1 y t-1 ), P t (x t y t ) and P t+1 (xt+1 y t+1 ), based on the two-dimensional state vector s of position and velocity t =[p t ,v t ] T The linear coordinates are transformed into spatial coordinates.

[0045] II: P t-1 P t The positional difference between them is denoted as Δx t and Δy t Formula 1 is used to represent the original state of the user's gaze point in frame t;

[0046] s t =[x t ,y t ,Δx t ,Δy t ] T (Formula 1)

[0047] III: Based on the filter window size, set the original state vector sequence for frame t.

[0048] S t ={s t-1 ,s t ,s t+1 Apply a mean filter independently to each state component (x, y, Δx, Δy) and output the result. As a measurement of Kalman filtering;

[0049] IV: Calculate the predicted transition state using Formula 2

[0050]

[0051] Where F represents the state transition matrix, which is set here to...

[0052] Combined process noise covariance matrix The prediction covariance matrix P of the (t-1)th frame t-1 Calculate the prediction covariance of frame t, as shown in Formula 3:

[0053] P t =FP t-1 F T +Q (Formula 3)

[0054] V: Based on the predicted covariance P t Calculate the Kalman gain using Formula 4:

[0055] K t =P t HT HP t H T +R) -1 (Formula 4)

[0056] Where R is the measurement noise covariance matrix, used to suppress the influence of unreliable measurements, and is set here as... H represents the observation matrix, set as follows: Mapping the four-dimensional state to the two-dimensional observation space;

[0057] VI: Update state S' using Kalman gain t As shown in Formula 5:

[0058]

[0059] Among them, z t The first two dimensions of the filtered state vector are... Extract the latest state S' t The first two components output the gaze point spatial coordinates P(x, y) of gaze point 205.

[0060] (1-3) Pixel coordinate back projection. Based on the spatial coordinates of the user's gaze point, a pre-trained Direct Linear Transformation (DLT) parameter model is input to generate the intrinsic and extrinsic parameter matrices of the built-in camera; the pixel coordinates are converted into direction vectors in the camera coordinate system using the perspective projection formula, and then transformed to a unified spatial coordinate system using the coordinate transformation matrix to obtain the three-dimensional gaze direction vectors originating from the centers of the left and right pupils.

[0061] (2) Gaze point recognition. Images of the user's field of view are acquired and fed into a lightweight object detection model. Teaching objects with confidence scores higher than the threshold are assigned the instance category with the highest probability. A rectangular window perpendicular to the user's gaze direction is generated and the window size is adaptively adjusted according to the head position and posture. The priority of the gazed object is calculated, the coordinates and visibility status of the object are corrected, and high-level objects are detected and rendered first.

[0062] (2-1) Testing of Teaching Subjects. Collect the six degrees of freedom (6DoF) states of the head-mounted display, such as... Figure 3 As shown, the image within the user's field of view is input into the YOLOv7-tiny network to obtain the coordinates and class probabilities of the detection boxes for the blackboard, podium, desk, and computer. Based on the objects within the detection boxes with confidence scores higher than a threshold, the instance class ICAT with the highest class probability is assigned (e.g., blackboard, podium, desk, and computer corresponding to 301, 302, 303, and 304 respectively). The same object class is maintained throughout the movement. The specific object detection steps are as follows:

[0063] I: Let DB be the set of detection boxes captured in the t-th frame. t Detection box d i The minimum and maximum values ​​of its horizontal and vertical axes, and the object confidence score Pobj i (Value range [0,1]), category probability vector p i This is represented as [x_ min y_ min , x_ max y_ max ,Pobj i p i The instance category ICAT of the detection box is determined by p. i The maximum component value is determined. Figure 3 In the diagram, 301, 302, 303, and 304 represent the blackboard, podium, desk, and computer detection boxes, respectively, denoted as d1, d2, d3, and d4; the confidence scores of the detected objects are Pobj1, Pobj2, Pobj3, and Pobj4, and the category probability vectors are p1, p2, p3, and p4.

[0064] II: Using a YOLOv7-tiny network trained on classroom scene images from ImageNet, predict the coordinates of each bounding box, object confidence (Pobj), and class probability vector (p). i ;

[0065] III: Calculate the confidence S for each detection box using Formula 6. i Obtain the category determined by the detection box:

[0066] S i =P obj ×max(p i )(Formula 6)

[0067] Where max(p) i ) represents the probability that the bounding box object is most likely to have an ICat; confidence level S i The default threshold θ is 0.25;

[0068] IV: Retain S i Detection boxes ≥ θ and assign ICat.

[0069] (2-2) Adaptive Object Window. When a user selects an object in a virtual scene using the gaze interaction of the head-mounted display, the vector relationship between the virtual object and the head position is calculated, a rectangular virtual window perpendicular to the user's gaze direction is generated, and the orientation and position are updated according to the 6DoF posture of the head-mounted display; the initial size of the window is adaptively adjusted according to the object distance, and the width and height are proportional to the object distance.

[0070] (2-3) Object window rendering. Based on the ICAT, 3D coordinates, and confidence field of the gazed object, calculate its priority score and store it in the state dictionary; when the position of the gazed object is displaced or partially occluded, the dictionary immediately corrects the coordinates and visibility state; and calls the recurrent neural network (RNN) to estimate the position of the next frame, sorting objects according to the gaze priority score, and prioritizing the detection and rendering of high-level objects.

[0071] (3) ROI tracking. Based on gaze duration and repetition frequency, points of interest are selected, and their coordinates, poses and attention areas are written into the behavior analysis database; the instance category of the virtual object where the point of interest is located is obtained, and points of interest with high similarity are clustered and represented by circles; based on the relevance of the teaching knowledge graph, virtual objects of the same instance category in the virtual scene are searched, and the ROIs of these objects are recommended to the user.

[0072] (3-1) Interest Point Extraction. When the user's gaze on an object in the virtual scene is detected to last for more than 600ms, this gaze point is marked as a candidate interest point. If the same object is triggered 4 or more times within 5 consecutive seconds, the teaching knowledge graph is automatically queried to calculate the semantic weight value of the gazed object. If the weight value exceeds the specified threshold, the candidate point is marked as an interest point, and its coordinates, posture and attention area are written into the behavior analysis database.

[0073] (3-2) ROI Tracking. The coordinates of the points of interest are mapped to the virtual scene using inverse projection transformation to obtain the instance category ICat of the virtual object where the points of interest are located; the similarity of ICat is quantified, and points of interest with a similarity greater than 0.6 are grouped into the same cluster to obtain multiple high-density cluster centers; based on the spatial radiality of user gaze behavior, each cluster center is mapped to the virtual scene coordinate system, and each ROI is represented by a circle.

[0074] (3-3) Similar Region Recommendation. Extract the ICat of the core teaching objects in the ROI. Based on the correlation of their teaching knowledge graph, search for other ICat objects in the entire virtual scene and recommend ROIs represented by the inscribed circle of the object's bounding rectangle to the user; calculate the kernel density ratio and boundary density ratio of different ROIs, and dynamically update the boundaries of the ROIs. Figure 4 Taking ROIs 401 and 402 as examples of exceeding the boundary, the specific steps for dynamically adjusting the ROI boundary are as follows:

[0075] I: The spatial coordinates of the point of interest are p k (x k y k Initial ROI i The radius is r i Set the initial ROI1 and ROI2 of the blackboard and podium corresponding to 401 and 402, respectively, and set the radii to r1 and r2.

[0076] II: Calculate ROI using Formula 7 i The kernel density ratio ρi and the boundary density ratio ∈ i :

[0077]

[0078] Where, N i Indicates the ROI i Intrinsic Interest Points (NI) i Indicates ROI i Number of ICAT-associated points of interest, NO i This indicates that the area extends beyond the circular boundary and is related to the ROI. i The number of interest points associated with ICAT;

[0079] III: Calculate the updated decision factor δ using Formula 8. i Dynamically adjust the radius of the ROI;

[0080]

[0081] IV: Calculate the ROI radius using Formula 9. The adjusted ROI boundaries are shown in Figures 403 and 404.

[0082]

[0083] (4) ROI boundary fusion. Extract image blocks of the gaze region, use the Cannny operator to detect image edges, extract effective contour information, and write it into the ROI boundary cache; analyze the boundary cache of effective ROIs, intelligently merge ROIs with high correlation; count the cumulative gaze duration and frequency of users, generate a heat map, and weight and superimpose it onto the ROI boundary, dynamically adjust the ROI boundary radius along the user gaze direction.

[0084] (4-1) Edge perception optimization. Extract the image patch of the teaching object in the gaze region, use the Cannny operator to detect the image edge, obtain the initial contour set, and calculate the contour area and aspect ratio; include the initial contours with a contour area greater than 200 pixels, aspect ratio range [0.5, 2.0], and depth standard deviation less than 0.2m into the effective contours, connect the adjacent contours, and write their cluster center and radius into the ROI boundary cache.

[0085] (4-2) Merging Teaching Areas. Analyze effective ROI boundary caches and intelligently merge adjacent teaching areas; extract HOG texture feature vectors and LBP feature matrices from multiple ROIs, transform them to the Fourier domain, and use density clustering algorithms to aggregate the space; set neighborhood radii and minimum clustering thresholds, integrate ROIs with high semantic relevance in teaching, and construct semantically coherent teaching focus units. The specific steps for ROI merging are as follows:

[0086] I: Such as Figure 5 In the diagram, 501, 502, and 503 correspond to the blackboard, podium, and desks and chairs, respectively, and the effective ROIs are represented as ROI1, ROI2, and ROI3.

[0087] II: Extract ROI using Formula 10 i HOG texture feature vector h i and LBP characteristic matrix I i ,

[0088]

[0089] III: Flatten the HOG vector and LBP matrix, then concatenate them into the ROI. i fusion feature vector Performing a one-dimensional Fast Fourier Transform (FFT) yields a complex-valued vector F. i =FFT(f i ), and its modulus ||F i || is used as the value of the new feature vector;

[0090] IV: Set a threshold ε = 0.5. If any two ROIs i If the Euclidean distance between the feature vector values ​​is less than a threshold, then they are merged. Figure 5 The Euclidean distance d(||F1||,||F2||) between the blackboard ROI and the podium ROI is less than 0.5, so they are merged into a new aggregated ROI, 504 and 505.

[0091] (4-3) ROI Boundary Correction. The cumulative user gaze duration and frequency of the merged ROI are statistically analyzed, and a heat map is generated using the Gaussian kernel density function. The heat map is aligned with the ROI boundary and then superimposed with weights. The pixels in the overlapping part of the boundary are normalized. If the heat gradient along the user gaze direction continues to increase, the ROI boundary radius is dynamically expanded along that direction; otherwise, the boundary range is shrunk.

[0092] (5) Object Adaptive Optimization. Detect the confidence level of objects in the image, re-acquire images of objects with lower confidence levels, and record their acquisition angle and lighting parameters; use a window detection algorithm to reset the position and pose of virtual objects in the scene, fully presenting the structure and appearance of the objects of interest; use fading operation to process the original position of virtual objects, correct scale and rotation deviations, and perform smooth transition optimization.

[0093] (5-1) Focus on object extraction. Using the YOLOv7-tiny basic detection model, when the object recognition confidence is lower than 0.55, a voice prompt is triggered to re-acquire the object image, requiring the user to acquire high-definition images of the object from multiple angles. If the object moves out of the user's field of view, 20 frames of background images are continuously acquired, and the acquisition angle and illumination parameters of the background and high-definition images are recorded, and background and object labels are added to them.

[0094] (5-2) Virtual Object Resetting. If a virtual object in the user's ROI is occluded by an object in the real teaching environment, the topological relationship between the virtual object and other objects in the scene is maintained. A window detection algorithm is used to find areas in the background image that do not obstruct the user's view. The position and pose of the virtual object are then reset in the virtual scene to fully reveal its structure and appearance. The specific steps of the region search are as follows:

[0095] I: Define the virtual object and the real teaching environment as P and N, respectively. Based on the user's head-mounted display pose and gaze direction, determine the projected image of the virtual object. Select a set of candidate windows W from the background image generated by the real environment around the object. i The window size is limited to 0.8-1.2 times the original window size.

[0096] II: Calculate the occlusion rate of the candidate window using Formula 11:

[0097]

[0098] Among them, Ω i Z(p) represents the set of pixels in the window, and Z(p) is the depth value of the pixel in the depth map. t (p) represents the target desired depth, and ∈ represents the occlusion tolerance. If the occlusion rate o(W) i Less than the threshold τ occ If the value is 0.2, it is determined to be an unobstructed area.

[0099] III: In all unobstructed candidate windows, select the occlusion rate o(W) i Illumination uniformity ΔL(W) i ) and scale factor s(W i Factors such as illumination difference threshold τ are also introduced. L =Δμ+kΔσ, where Δμ and Δσ represent the mean and standard deviation of the brightness difference between the virtual object and the real teaching environment, respectively, and k is an adjustment factor. When k<1, it is very sensitive to light fluctuations; when k>1.5, it causes overly dark or overly bright areas to be incorrectly accepted. Therefore, normally k∈[1,1.5], construct the scoring function as shown in Formula 12, and the window W with the highest score is the object reset position.

[0100]

[0101] Among them, w o w l w s These are the weighting coefficients, with default values ​​of 0.4, 0.3, and 0.3.

[0102] IV: Map the object's anchor point to the new window W, and obtain the object's new rotation matrix and translation vector (R,T). Use formula 13 for smoothing exponent to avoid jitter after resetting:

[0103] T i =αT i +(1-α)T i-1 (Formula 13)

[0104] Among them, T i T represents the pose matrix of the current frame. i-1 The matrix corresponds to the previous frame, and α is a smoothing factor, typically set to [0.7, 0.9], used to control the weight ratio of the old and new virtual objects. The larger α is, the better the smoothing effect, but the response speed of tracking the object's position and pose will decrease.

[0105] V: If the occlusion rate or lighting difference of all candidate windows does not meet the conditions, keep the object in place and increase its transparency, such as displaying it in a semi-transparent manner, until a new unoccluded window appears, and then restore it to full display.

[0106] (5-3) Enhanced adaptation results. For example... Figure 6 As shown, when a virtual object is moved to a new position, a fading process needs to be performed on the original position. The virtual object at the new position is then input into a lightweight detection model to extract its edge contours, textures, and depth. The extracted features are compared with the template to correct its scale and rotation deviations, optimize the smooth transition of the virtual object, and synchronize it to the virtual environment in real time.

[0107] (6) ROI brightness adjustment. Extract the average brightness value within the ROI, and based on the comparison with the ideal brightness range, enter the global or local brightness adjustment mode; adjust the ROI exposure time and photosensitive gain according to the relationship with the upper and lower limits of the ideal range until the average brightness value enters the ideal range; adopt a phased transition strategy to dynamically adjust the global exposure time and global gain coefficient to correct the global brightness.

[0108] (6-1) Brightness Deviation Judgment. Extract the mean brightness L within the i-th ROI. m Then compare it with the preset ideal brightness range [80, 180]. If the average brightness falls within the range, it is considered normal brightness. If the average brightness exceeds the range, calculate the brightness deviation amplitude ΔL relative to the nearest boundary according to formula 14. When ΔL≥20, it is determined that the brightness deviation is significant and enters the global brightness adjustment mode; otherwise, it enters the local brightness adjustment mode.

[0109] ΔL=min(|L m -80|,|L m -180|) (Formula 14)

[0110] (6-2) Local brightness adjustment. Determine the average brightness of the detection area. If L m <80, increase the exposure time within the area by a fixed ratio in each frame, and adjust the photosensitivity gain synchronously; if L m If the value is greater than 180, then adjust the exposure time and photosensitive gain in the opposite direction. After adjusting the exposure time and photosensitive gain, recalculate the average ROI brightness until the ROI brightness is within the ideal brightness range. Figure 7 The brightness of the 701 lecture hall and the background of the desks and chairs is similar, but significantly lower than the ideal brightness range. The specific steps for adjusting the brightness are as follows:

[0111] I: Calculate the average luminance of the ROI of podium 701 using formula 15:

[0112]

[0113] Where L(x,y) represents the brightness value of a pixel in the ROI, |ω| is the total number of pixels in the ROI, and ω represents the condition that (x-x0) is satisfied. 2 +(y-y0) 2 ≤r 2 All pixels.

[0114] II: If the average ROI brightness is 72, then the deviation ΔL from the ideal brightness range is 8 (80-72=8), indicating a small deviation that does not exceed the significant threshold of 20. Therefore, a small increment is used to fine-tune the exposure time and gain coefficient within the ROI. According to the camera imaging model, the ROI brightness value is proportional to the exposure time T and the gain coefficient G. Formula 16 is used to fine-tune T and G to gradually increase or decrease the brightness:

[0115]

[0116] Where s is the direction coefficient, with positive and negative values ​​representing brightening or darkening respectively; T is the original exposure time; G is the original gain coefficient; and Δ is the adjustment ratio, which defaults to 0.05.

[0117] If the current exposure time is T = 5ms, the gain is G = 1.0, and the orientation coefficient is set to s = 1, then the adjusted parameters are T' = 5.25ms and G' = 1.05, which increases the brightness of the area.

[0118] III: After adjusting the brightness, recalculate using formula 15. If the average value falls within the ideal range, stop adjusting; otherwise, repeat step II until the requirements are met. The adjusted effect is as follows: Figure 7 As shown in Figure 702.

[0119] (6-3) Global brightness adjustment. When the average brightness of ROI in 15 consecutive frames is found to deviate from the ideal brightness range by ΔL≥20, global brightness adjustment is performed; if the overall brightness is too low, the ROI brightness parameter is kept unchanged, and the global exposure time is dynamically adjusted to the ideal brightness range; if the overall brightness is too high, the global gain coefficient is reduced until it meets the ideal brightness range; a phased transition strategy is adopted to gradually correct the global brightness.

[0120] (7) Gaze object interaction. Detect collision between the user's gaze and objects in the virtual scene, and use a four-tuple to record valid gazes with a gaze duration exceeding a threshold; find the interaction type label based on the gaze object ID, and trigger different interaction instructions based on the gaze duration; add the interaction instructions to the global queue in chronological order, schedule feedback according to priority, and execute the interaction.

[0121] (7-1) Collision detection. For example... Figure 8 As shown, the system detects collisions between the user's line of sight and the 3D bounding boxes of objects such as the blackboard and podium in the virtual scene. When a collision occurs, the system records the virtual object's ID and the timestamps t for the collision occurrence and termination. in t out And its coordinates, and calculate the fixation duration Δt=t out -t in Set a valid gaze duration threshold of 1 second to avoid misclassifying brief saccades as valid gazes; if the gaze ray lingers on a virtual object for longer than the time threshold, it is marked as a valid gaze event and stored as a quadruple:

[0122] E = (ID, Δt, C, T)

[0123] Where ID is the virtual object identifier, Δt is the gaze duration, C is the coordinates of the gaze center, and T is the timestamp of the collision.

[0124] (7-2) Collision Event Mapping. After a valid gaze occurs, the interaction logic is triggered based on the gaze object's ID. The specific steps are as follows:

[0125] I: Locate the corresponding interaction type tag based on the object identifier ID: There are three types: interpretable text, operable tools, and focusable areas. If it is "interpretable text", the information display logic is triggered, and a prompt box or explanation pops up; if it is "operable tools", the interactive tool is selected or launched; if it is "focusable area", highlighting or outline lighting effects are triggered.

[0126] II: Obtain the trigger condition based on the gaze duration Δt:

[0127]

[0128] Among them, Rb Indicates shallow feedback, R d This indicates deep interaction. When 1s ≤ Δt < 2s, R is executed. b Highlight the object the user is looking at; when Δt ≥ 2s, execute R. d This will bring up an interactive data panel or activate the operation function.

[0129] (7-3) Interaction Feedback Scheduling. Events triggered by interaction commands are added to the global execution queue in chronological order. If multiple interaction events exist within the same frame, interaction priorities are assigned to objects using Formula 17, taking into account gaze duration, weight parameters, and static priority.

[0130] P = wΔt + L (Formula 17)

[0131] Where Δt is the gaze duration of the object, w is the weight parameter, and L is the static priority of the object (blackboard = 2, podium = 1, background = 0); feedback is executed sequentially from high to low priority P. After the interaction is completed, the queue record in the event pool is cleared and the cache is released.

[0132] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0133] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically adapting the teaching atmosphere in an immersive teaching environment, characterized in that... The method includes the following steps: (1) Eye-tracking image acquisition: The user's eye images are captured using the built-in binocular camera of the head-mounted display. The pupil center is calibrated using the coordinate regression algorithm, and the coordinates of the left and right pupils are calculated. Median filtering is used to remove the jump noise of the two-dimensional state vector of the pupil, and Kalman filtering is used to smooth the gaze point trajectory. According to the pre-trained DLT model, the pixel coordinates of the user's gaze point are converted into a three-dimensional gaze direction vector. (2) Gaze point recognition; collect images of the user's field of vision of the head display, input them into the lightweight object detection model, assign the teaching object with the highest probability instance category to the confidence level higher than the threshold; generate a rectangular window perpendicular to the user's gaze direction, and adaptively adjust the window size according to the head position and posture; calculate the priority of the gaze object, correct the object's coordinates and visibility status, and prioritize the detection and rendering of high-level objects. (3) ROI tracking; Based on the user's gaze duration and frequency, filter points of interest and write their coordinates, posture and attention area into the behavior analysis database; obtain the instance category of the virtual object where the point of interest is located, cluster the points of interest with high similarity, and use circles to represent ROI; Based on the correlation of the teaching knowledge graph, search for virtual objects of the same instance category in the virtual scene and recommend the ROI where the above virtual objects are located to the user; (4) ROI boundary fusion: Extract image blocks of the gaze region, use the Cannny operator to detect image edges, extract effective contour information, and write it into the ROI boundary cache; analyze the boundary cache of effective ROIs, intelligently merge ROIs with high correlation; count the cumulative gaze duration and frequency of users, generate a heat map, and weight and overlay it onto the ROI, dynamically adjust the ROI boundary radius along the user gaze direction; (5) Object adaptive optimization; detect the confidence of objects in the image, re-acquire images of objects with confidence below the threshold, and record their acquisition angle and lighting parameters; use window detection algorithm to reset the position and pose of virtual objects in the scene, and fully present the structure and appearance of the objects of interest; use fading operation to process the original position of virtual objects, correct scale and rotation deviations, and perform smooth transition optimization; (6) ROI brightness adjustment; extract the average brightness value within the ROI, and enter the global or local brightness adjustment mode based on the comparison result with the ideal brightness range; adjust the ROI exposure time and photosensitive gain according to the relationship with the upper and lower limits of the ideal range until the average brightness value enters the ideal range; adopt a phased transition strategy to dynamically adjust the global exposure time and gain coefficient to correct the global brightness. (7) Gaze object interaction; detect the collision between the user's gaze and objects in the virtual scene, use a quadruple to record valid gazes with a gaze duration exceeding the threshold; find the interaction type label based on the gaze object ID, and trigger different interaction instructions based on the gaze duration; add the interaction instructions to the global queue in chronological order, schedule feedback according to priority, and execute the interaction.

2. The method for automatically adapting the teaching atmosphere in an immersive teaching environment according to claim 1, characterized in that... The eye-tracking image acquisition in step (1) specifically includes: (1-1) Binocular image acquisition: Start the built-in left and right cameras of the head-mounted display device at a constant frequency, and simultaneously turn on exposure compensation to correct the reflective area; call the frame synchronization interface to overlay a timestamp on each frame of image, align the left and right image sequences according to the timestamp, and verify the brightness parameters of the image frame by frame; and build an image queue with a fixed number of frames, input it into a lightweight convolutional neural network, extract eye features, use a coordinate regression algorithm to calibrate the pupil center, and calculate the coordinates of the left and right pupils; (1-2) Gaze point trajectory output: Calculate the two-dimensional state vector of the user's gaze position and head movement speed according to the pupil coordinate change, use median filtering with a fixed window size to remove jump noise, and then input the processed vector into the Kalman filter model, and perform state prediction, covariance update, measurement residual calculation and optimal estimation correction in sequence to output the smoothed gaze point trajectory; (1-3) Pixel coordinate back projection; based on the spatial coordinates of the user's gaze point, input the pre-trained direct linear transformation parameter model to generate the intrinsic and extrinsic parameter matrices of the built-in camera; use the perspective projection formula to convert the pixel coordinates into direction vectors in the camera coordinate system, and then use the coordinate transformation matrix to convert them to a unified spatial coordinate system to obtain the three-dimensional gaze direction vectors starting from the center of the left and right pupils.

3. The method for automatically adapting the teaching atmosphere in an immersive teaching environment according to claim 1, characterized in that... The gaze point identification described in step (2) specifically includes: (2-1) Detection of teaching objects; collect the six degrees of freedom of the head-mounted display, input the image within the user's field of view into the YOLOv7-tiny network, obtain the coordinates and class probabilities of the object detection box in the real teaching environment, assign the instance class with the highest probability based on the object in the detection box with a confidence level higher than the threshold, and keep the class of the same object unchanged during the movement. (2-2) Adaptive object window: When the user selects an object in the virtual scene using the gaze interaction of the head-mounted display, the vector relationship between the virtual object and the head position is calculated, a rectangular virtual window perpendicular to the user's gaze direction is generated, and the orientation and position are updated according to the 6DoF posture of the head-mounted display; the initial size of the window is adaptively adjusted according to the object distance, and the width and height are proportional to the object distance; (2-3) Object window rendering: Calculate the priority score of the object being gazed upon based on its instance category, 3D coordinates, and confidence field, and store it in the state dictionary; when the position of the object being gazed upon is displaced or partially occluded, the dictionary immediately corrects the coordinates and visibility status; and calls the recurrent neural network to estimate the position of the next frame, sorts the objects according to the gaze priority score, and prioritizes the detection and rendering of high-level objects.

4. The method for automatically adapting the teaching atmosphere in an immersive teaching environment according to claim 1, characterized in that... The ROI tracking described in step (3) specifically includes: (3-1) Interest point extraction: When it is detected that the user's gaze on a virtual object exceeds a certain time threshold, the gaze point is marked as a candidate interest point; if the same object is triggered multiple times within a continuous period of time, the teaching knowledge graph associated with the object is automatically queried, the semantic weight value of the gazed object is calculated, and if the weight value exceeds the specified threshold, the candidate point is marked as an interest point, and its coordinates, posture and attention area are written into the behavior analysis database. (3-2) ROI tracking; using inverse projection transformation to map the coordinates of interest points to the virtual scene to obtain the instance category of the virtual object where the interest point is located; quantifying the similarity of instance categories, grouping interest points with similarity greater than the threshold into the same cluster to obtain multiple high-density cluster centers; based on the spatial radiality of user gaze behavior, using circles to represent each ROI; (3-3) Similar region recommendation: Extract the instance category of the core teaching object in the ROI, and search for other virtual teaching objects with the same instance in the entire virtual scene based on the correlation of their teaching knowledge graph. Recommend the ROI represented by the inscribed circle of the bounding rectangle of each object to the user; calculate the kernel density ratio and boundary density ratio of different ROIs, and dynamically update the boundary of the ROI.

5. The method for automatically adapting the teaching atmosphere in an immersive teaching environment according to claim 1, characterized in that... The ROI boundary fusion described in step (4) specifically includes: (4-1) Edge perception optimization: Extract the image patch of the teaching object in the gaze region, use the Cannny operator to detect the image edge, obtain the initial contour set, calculate the contour area and aspect ratio; include the initial contours whose contour area, aspect ratio range, and depth standard deviation are within the corresponding threshold into the effective contours, connect the adjacent contours, and write their cluster center and radius into the ROI boundary cache. (4-2) Merging teaching areas; Analyze effective ROI boundary caches and intelligently merge adjacent teaching areas; Extract HOG texture feature vectors and LBP feature matrices of multiple ROIs, transform them to the Fourier domain, and call density clustering algorithm to aggregate the space; Set neighborhood radius and minimum clustering threshold, integrate ROIs with high teaching semantic relevance, and construct semantically coherent teaching focus units. (4-3) ROI boundary correction: Statistically calculate the cumulative user gaze duration and frequency of the merged ROI, and generate a heat map using the Gaussian kernel density function; After aligning the heat map with the ROI boundary, perform weighted superposition and normalize the pixels in the overlapping part of the boundary; If the heat gradient along the user gaze direction continues to rise, dynamically expand the ROI boundary radius along that direction; otherwise, shrink the boundary range.

6. The method for automatically adapting the teaching atmosphere in an immersive teaching environment according to claim 1, characterized in that... The adaptive optimization of the object mentioned in step (5) specifically includes: (5-1) Focus on object extraction; using the YOLOv7-tiny basic detection model, when the object recognition confidence is lower than the threshold, a voice prompt is triggered to re-acquire the object image, requiring the user to acquire high-definition images of the object from multiple angles; if the object moves out of the user's head-mounted display field of view, multiple frames of background images are continuously acquired, the acquisition angle and illumination parameters of the background and high-definition images are recorded, and background and object labels are added to these acquired parameters; (5-2) Virtual object reset: If a virtual object in the ROI that the user is looking at is occluded by an object in the real teaching environment, the virtual object is kept in topological relationship with other objects in the scene. The window detection algorithm is used to find the area in the background environment image that does not obstruct the user's view. The position and posture of the virtual object are reset in the virtual scene to fully display its structure and appearance. (5-3) Adaptation result enhancement: When the virtual object is moved to a new position, the original position needs to be faded. The virtual object at the new position is input into the lightweight detection model to extract its edge contour, texture and depth. The extracted features are compared with the template to correct its scale and rotation deviation, optimize the smooth transition of the virtual object, and synchronize it to the virtual environment in real time.

7. The method for automatically adapting the teaching atmosphere in an immersive teaching environment according to claim 1, characterized in that... The ROI brightness adjustment in step (6) specifically includes: (6-1) Brightness deviation judgment: Extract the average brightness within the ROI and compare it with the preset ideal brightness range; if the average brightness falls within the range, it is considered normal brightness; if the average brightness exceeds the range, calculate the brightness deviation amplitude relative to the nearest boundary. When the deviation amplitude is greater than the threshold, it is determined that the brightness deviation is significant and enters the global brightness adjustment mode; otherwise, it enters the local brightness adjustment mode. (6-2) Local brightness adjustment; Determine the average brightness of the detection area. If the average brightness deviates from the ideal range by less than the threshold, increase the exposure time in the area by a fixed ratio in each frame and adjust the photosensitive gain synchronously. If the average brightness deviates from the ideal range by more than the threshold, adjust the exposure time and photosensitive gain in the opposite direction. Adjust the exposure time and photosensitive gain, and then recalculate the average brightness of the ROI until the brightness is in the ideal brightness range. (6-3) Global brightness adjustment: When the average ROI brightness of consecutive frames deviates from the ideal brightness range by more than a threshold, global brightness adjustment is performed; if the overall brightness is lower than the threshold, the global exposure time is dynamically adjusted to the ideal brightness range; if the overall brightness is too bright, the global gain coefficient is reduced until it meets the ideal brightness range; a phased transition strategy is adopted to gradually correct the global brightness.

8. The method for automatically adapting the teaching atmosphere in an immersive teaching environment according to claim 1, characterized in that... The gaze object interaction described in step (7) specifically includes: (7-1) Collision detection: Detect the collision between the user's line of sight and objects in the virtual scene. When the collision ends, record the ID of the virtual object, the timestamp of the collision occurrence and end, and its coordinates, and calculate the gaze duration. Set a minimum threshold for effective gaze duration to avoid misjudging a brief scan as a valid gaze. If the gaze ray stays on the virtual object for a longer time than the threshold, it is marked as a valid gaze event and stored as a quadruple. (7-2) Collision event mapping; After a valid gaze occurs, the corresponding interaction type label is found according to the object identifier ID, which is divided into interpretable text, operable tools and focusable area types; The trigger condition is obtained according to the gaze duration. When the gaze duration is less than the threshold, shallow feedback is executed and the object being gazed at by the user is highlighted; When the gaze duration is greater than the threshold, deep interaction is executed and the interaction data panel pops up or the operation function is activated. (7-3) Interactive feedback scheduling: Add interactive instruction trigger events to the global execution queue in chronological order; if there are multiple interactive events in the same frame, assign interactive priority to the object by combining gaze duration, weight parameters and static priority; execute feedback in order of priority from high to low, and after completing the interactive execution, clear the queue record of the event pool and release the cache.

9. An automatic adaptation system for teaching atmosphere in an immersive teaching environment, characterized in that: The method for implementing the method according to any one of claims 1 to 8 includes: an eye-tracking image acquisition module, a gaze point recognition module, an ROI tracking module, an ROI boundary fusion module, an object adaptive optimization module, an ROI brightness adjustment module, and a gaze object interaction module; The eye-tracking image acquisition module is used to acquire the user's pupil coordinates, a smooth and stable gaze point trajectory, and output a three-dimensional gaze direction vector. The gaze point recognition module is used to detect the instance category to which the object belongs, adaptively generate virtual windows, and calculate and render window objects with high priority. The ROI tracking module is used to extract users' points of interest, cluster interest regions with high similarity, and recommend potentially related regions using knowledge graphs. The ROI boundary fusion module is used to detect edges and thermal distribution, intelligently merge ROIs with high correlation, and adaptively correct ROI boundaries. The object adaptive optimization module is used to acquire object images with low confidence, reset the position and pose of the occluded virtual object, and correct its scale and rotation deviation. The ROI brightness adjustment module is used to determine the brightness deviation of the ROI, adjust the exposure time and photosensitivity gain, and correct local or global brightness. The gaze object interaction module is used to detect collisions between the user's gaze and virtual objects, trigger corresponding collision feedback, and schedule high-priority collision events.

Images