A camera-based real-time line-of-sight interaction and visual analysis system and method

Abstract

This invention discloses a real-time gaze interaction and visualization analysis system and method based on a camera, belonging to the field of gaze tracking. It includes a gaze information acquisition module, a gaze mapping module, a calibration module, a detection module, an intervention module, and a visualization analysis module. The gaze information acquisition module acquires a real-time video stream of the user's face, extracts three-dimensional facial key points using a facial key point detection model, locates the face, and acquires head posture and eye region images. The eye region images are then input into a gaze estimation model. This invention uses a common camera and achieves high-precision, non-invasive gaze tracking by fusing facial key points, head posture, and the gaze model, combined with dynamic calibration. Simultaneously, it links gaze analysis with teaching content, providing graded intervention based on the degree of deviation to effectively guide attention. Finally, it outputs structured individual and group learning data, supporting personalized feedback and courseware optimization, thus contributing to teaching improvement.

Description

Technical Field

[0001] This invention belongs to the field of eye tracking, specifically, it is a real-time eye interaction and visualization analysis system and method based on a camera. Background Technology

[0002] The eyes are the windows to the soul, and eye-tracking allows us to capture information conveyed by the eyes, enabling human-computer interaction and research in fields such as communication studies, psychology, and online learning behavior analysis. Eye-tracking is a technology that uses various detection methods to acquire a gaze point and continuously track it over a period of time. This technology is widely used in human-computer interaction, virtual reality, and user experience evaluation. Existing eye-tracking devices and technologies are mainly divided into two categories: 3D eye-tracking and 2D eye-tracking. 3D cameras use dual or multi-camera arrays, combined with structured light, Time-of-Flight (ToF) and other technologies, to acquire depth information and coordinate data of objects in three-dimensional space. In eye-tracking scenarios, it can capture the three-dimensional structure of the eyes, accurately locating the spatial position of the eyeball, rotation angle, and the three-dimensional coordinates of the pupil. 2D cameras, on the other hand, rely on single-frame planar images to complete visual information acquisition, capturing features such as the outline, color, and texture of objects in two-dimensional space. In the field of eye-tracking, it identifies changes in the planar position of feature points such as the pupil and iris to determine the gaze direction and gaze point. It boasts advantages such as low deployment cost, mature algorithms, and compact hardware size, making it compatible with lightweight devices such as ordinary computers and mobile terminals. It is widely used in scenarios such as basic human-computer interaction, preliminary analysis of online learning behavior, and low-cost user experience evaluation.

[0003] Chinese patent CN202111361427.0 discloses an online learning attention tracking method and its application based on gaze estimation. The method includes: acquiring facial images, eye images, and binocular infrared images of the subject to be detected, as well as a scene image of the learning environment in which the subject is located; the scene image includes an image of the learning device interacting with the subject; inputting the facial image, eye image, and binocular infrared image into a gaze estimation recognition model to obtain the three-dimensional gaze direction of the subject's eyes in the camera coordinate system; converting the three-dimensional gaze direction in the camera coordinate system into a two-dimensional gaze point in the screen coordinate system of the learning device; and generating the current attention detection result of the subject based on the positional relationship between the two-dimensional gaze point and the learning area in the scene image. This invention achieves complementarity between head posture and two types of eye features, improving the accuracy of gaze estimation in complex backgrounds and providing objective supporting data for improving students' online learning attention.

[0004] However, the above solutions still have several problems. First, they fail to consider that most users' computers and other electronic devices are only equipped with ordinary cameras. Some solutions require very high camera accuracy, which some ordinary cameras on the market cannot meet. Furthermore, they do not consider the impact of different users' behavioral habits on eye tracking, resulting in significant eye tracking errors. Second, the display results of human-computer interaction are limited and cannot meet the needs of users who are distracted during online learning through eye tracking. They also cannot provide data for user experience evaluation, online learning behavior analysis, or for teachers to obtain feedback from courseware.

[0005] Therefore, there is an urgent need for a low-cost learning tool that can be adapted to ordinary 2D webcams, provide learning intervention for users' online learning, improve learning efficiency, and analyze and improve users' online learning behavior. Summary of the Invention

[0006] This invention aims to provide a real-time gaze interaction and visualization analysis system and method based on a camera, to solve the problems mentioned in the background section. To achieve the above objective, this invention adopts the following technical solution:

[0007] A camera-based real-time gaze interaction and visualization analysis system includes: a gaze information acquisition module, a gaze mapping module, a calibration module, a detection module, an intervention module, and a visualization analysis module;

[0008] The gaze information acquisition module is used to acquire real-time video stream of the user's face, extract three-dimensional facial key points using a facial key point detection model, locate the face and acquire head posture and eye region images, input the eye region images into the gaze estimation model, and output a unit gaze vector representing the user's current gaze direction.

[0009] The gaze mapping module is used to divide the user's screen into multiple grid-like screen blocks, and calculate the intersection point between the gaze and the screen through a spatial geometric projection model based on the head posture and unit gaze vector. Taking the screen block where the intersection point is located as the center, the screen blocks directly adjacent to it in the upper, lower, left and right directions are selected together as the initial mapping gaze area. The screen blocks serve as the basic spatial units for gaze positioning, and their size and number are dynamically adjusted according to the application scenario.

[0010] The calibration module is used to dynamically calibrate the initial mapped gaze area. The dynamic calibration is based on the user's real-time head posture changes and the offset of the initial mapped gaze area, and adjusts the calibration parameters in real time to output the set of screen blocks corresponding to the user's current real gaze, as the final gaze area.

[0011] The detection module is used to determine whether the user has an invalid interaction state or attention deviation based on the final gaze area and the focus area associated with the current teaching content. The focus area is set by the teacher according to the teaching content of the current teaching period, and represents the set of screen blocks that the user should gaze at. The focus area consists of multiple adjacent screen blocks and can be dynamically adjusted as the teaching content is switched.

[0012] The intervention module is used to trigger corresponding levels of visual prompts based on the degree of invalid interaction or attention deviation when an invalid interaction state or attention deviation is detected.

[0013] The visualization analysis module is used to generate learning data based on the matching of the final gaze area and the focus area, and upload the learning data to the teacher's end. The learning data includes at least the matching degree of the final gaze area and the focus area, the trigger information of invalid interactions and attention deviations, and the trigger record of graded prompts.

[0014] Optionally, the calibration module includes an initial calibration unit and individual calibration units;

[0015] The initial calibration unit is used to establish global correction parameters through a center point guidance method to perform preliminary compensation on the initial mapped gaze area. The center point guidance method involves displaying a calibration prompt point in the center of the screen and guiding the user to gaze at the calibration prompt point. The global correction parameters are generated based on the spatial offset between the initial mapped gaze area and the set of screen blocks corresponding to the calibration prompt point.

[0016] The individual calibration unit is used to construct a set of user-specific stable gaze blocks corresponding to multiple calibration points through multiple rounds of sampling after the initial calibration is completed, as gaze stability data. In the subsequent gaze estimation process, the final gaze area is determined from the corresponding gaze stability data based on the spatial proximity relationship between the current initial mapped gaze area and each calibration point.

[0017] Optionally, the initial calibration unit includes:

[0018] The center calibration point display subunit is used to display the initial calibration points of multiple adjacent screen blocks at the center of the screen.

[0019] The offset calculation subunit guides the user to gaze at the initial calibration point, collects the user's initial mapped gaze area, and calculates the spatial offset between the user and the set of screen blocks corresponding to the initial calibration point.

[0020] A global correction parameter generation subunit is used to establish global correction parameters based on the spatial offset;

[0021] The compensation output subunit is used to compensate the subsequently acquired initial mapped gaze region using the global correction parameters and output the initial corrected gaze region.

[0022] Optionally, the individual calibration unit includes:

[0023] The multi-round sampling control subunit distributes calibration points in the edge and center areas of the screen, repeatedly performs a preset number of sampling rounds, and displays all calibration points in sequence in each round;

[0024] The gaze area acquisition subunit is used to acquire the user's initial corrected gaze area when displaying each calibration point;

[0025] The frequency statistics subunit is used to merge the initial correction gaze area collected in all rounds for each calibration point and count the frequency of each screen block being covered.

[0026] A stable set construction subunit is used to retain screen blocks whose frequency reaches a preset frequency threshold, forming a stable block set corresponding to the calibration point;

[0027] The nearest calibration point matching subunit is used to calculate the Euclidean distance between the center coordinates of the initial corrected gaze area and each calibration point, match the initial corrected gaze area to the nearest calibration point, and output the final gaze area based on the set of stable blocks of the calibration point, wherein the center coordinates of the initial corrected gaze area are the geometric center of the screen blocks it contains.

[0028] Optionally, the detection module includes an interaction state discrimination unit and an attention shift discrimination unit;

[0029] The interaction state determination unit determines whether the user is in a valid interaction state. When no valid face area is detected within a consecutive preset number of frames, or the calculated line of sight has no intersection with the screen, it is determined to be an invalid interaction state and an invalid interaction state signal is output.

[0030] The attention shift discrimination unit compares the current final gaze area with the focus area corresponding to the current teaching period when the user is not in an invalid interaction state. If the final gaze area and the focus area corresponding to the current teaching period do not overlap, the attention shift is classified according to the duration of the attention shift. When the shift duration is ≥3 seconds and <10 seconds, it is judged as a mild shift; when the shift duration is ≥10 seconds, it is judged as a severe shift.

[0031] Optionally, the intervention module performs tiered prompts based on the type of the received signal:

[0032] The first-level prompt unit is used to trigger a first-level prompt when a slight deviation is detected. At this time, a dynamic light bar is displayed on the edge of the user's screen to guide the user's gaze back to the focus area.

[0033] The secondary prompt unit is used to trigger a secondary prompt when a severe offset is determined. At this time, the non-focus area of ​​the user terminal is blacked out, and only the content of the focus area is displayed. The non-focus area refers to all user terminal screen blocks outside the focus area of ​​the current teaching period.

[0034] The three-level prompt unit is used to trigger a three-level prompt when the interaction is determined to be invalid, and to play a voice prompt through the speaker to remind the user to return to the focus area;

[0035] The prompt-to-close unit is used to close all triggered hierarchical visual cues after the user's gaze returns to the focus area and remains there for 2 seconds.

[0036] Optionally, the first-level prompting unit specifically involves: determining the direction of the user's gaze relative to the focus area based on the relative position of the final gaze area center and the current focus area; displaying a gradient light bar on the right side of the screen when the gaze is to the left, on the left side of the screen when the gaze is to the right, at the bottom of the screen when the gaze is to the top, and at the top of the screen when the gaze is to the bottom; and guiding the user's gaze back through a dynamic light bar that is opposite to the direction of the gaze shift.

[0037] Optionally, the visualization analysis module includes:

[0038] The learning progress report generation unit is used to synchronously record the trigger time, trigger type, duration and release time of the graded prompts. Combined with the real-time matching degree between the user's final gaze area and focus area, the frequency and duration of invalid interaction states, it generates a structured digital learning progress report containing user individual identifiers, teaching time periods and corresponding teaching content tags, and uploads it to the teacher's end.

[0039] The individual feedback generation unit analyzes the user's attention concentration periods, high-frequency deviation intervals, and prompt triggering patterns based on data from a single user's final gaze area, and generates a personalized review report with suggestions for attention improvement.

[0040] The group feedback generation unit, based on the distribution data of the final gaze areas of multiple users under the same courseware, statistically analyzes the average gaze duration and the percentage of people gazing in each area of ​​the courseware, while identifying high-incidence nodes of attention deviation during the courseware playback process. Combining the correspondence between the display position of the courseware content and the group gaze and deviation data, it generates the visual attention feature analysis results of the courseware content.

[0041] Optionally, the gaze mapping module is also used to obtain the image resolution captured by the user's camera, and dynamically adjust the size and number of the screen blocks according to the image resolution. The higher the image resolution, the smaller the screen block size and the more the total number; the lower the resolution, the larger the screen block size and the fewer the total number.

[0042] A camera-based real-time gaze interaction and visualization analysis method, applied to the camera-based real-time gaze interaction and visualization analysis system as described in any one of claims 1-9, includes the following steps:

[0043] S101, gaze information acquisition: acquire the real-time video stream of the user's face, extract three-dimensional facial key points using the facial key point detection model, locate the face and acquire head posture and eye region images, input the eye region images into the gaze estimation model, and output a unit gaze vector representing the user's current gaze direction.

[0044] S102, gaze mapping, divides the user's screen into multiple grid-like screen blocks, and calculates the intersection point between the gaze and the screen through a spatial geometric projection model based on the head posture and unit gaze vector. Taking the screen block where the intersection point is located as the center, selects the screen blocks directly adjacent to it in the upper, lower, left and right directions, and together they serve as the initial mapped gaze area. The screen blocks serve as the basic spatial units for gaze positioning, and their size and number are dynamically adjusted according to the application scenario and can also be dynamically adjusted according to the resolution of the images captured by the camera.

[0045] S103, Dynamic calibration, dynamically calibrating the initial mapped gaze area, the dynamic calibration is based on the user's real-time head posture changes and the offset of the initial mapped gaze area, adjusting the calibration parameters in real time, and outputting the set of screen blocks corresponding to the user's current real gaze as the final gaze area.

[0046] S104, State detection: Based on the final gaze area and the focus area associated with the current teaching content, determine whether the user has an invalid interaction state or attention deviation. The focus area is set by the teacher according to the teaching content of the current teaching period, representing the set of screen blocks that the user should gaze at. The focus area consists of multiple adjacent screen blocks and can be dynamically adjusted as the teaching content is switched.

[0047] S105, graded intervention, triggers corresponding level of visual prompts based on the degree of invalid interaction or attention deviation when an invalid interaction state or attention deviation is detected;

[0048] S106, Visual analysis: Based on the matching of the final gaze area and the focus area, generate learning data and upload the learning data to the teacher's end. The learning data includes at least the matching degree of the final gaze area and the focus area, invalid interaction and attention deviation trigger information, and graded prompt trigger records.

[0049] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: First, it achieves adaptive, non-invasive gaze tracking, relying only on ordinary 2D cameras. By fusing 3D facial key points, head posture, and gaze estimation models, and combining dynamic screen block division, initial calibration, and individualized multi-point stable gaze modeling, it achieves real-time gaze positioning under the user's natural movement state, improving accuracy without requiring dedicated hardware. Second, it associates gaze analysis with teaching content and implements progressive intervention based on the degree of deviation, effectively guiding the user's attention back. Finally, by collecting and structured outputting individual and group learning data, it not only generates personalized attention review reports but also supports courseware-level visual attention heatmap analysis and attention deviation node identification, providing teachers with data-driven teaching improvement basis.

[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0051] The present invention can be further illustrated by the non-limiting embodiments given in the accompanying drawings;

[0052] Figure 1 This is an architecture block diagram of a camera-based real-time gaze interaction and visualization analysis system.

[0053] Figure 2 This is a flowchart of the method of the present invention;

[0054] Figure 3 This is a schematic diagram of a 9-point calibration according to an embodiment of the present invention;

[0055] Figure 4 This is a heatmap of attention density according to an embodiment of the present invention; Detailed Implementation

[0056] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the one or more embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. The various technical features described in the present invention (such as "preset duration," "preset threshold," "multiple calibration points," etc.) represent a class of configurable parameters or schemes, and their specific values ​​or implementation forms can be flexibly adjusted according to actual application scenarios, hardware performance, and user needs. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0057] Eye tracking is now being used in many fields. For example, in online learning and education, the user's eye gaze is an important indicator for assessing learning focus. By collecting and quantifying learning behaviors, dynamically intervening in attention, and providing personalized learning guidance, users' learning efficiency can be greatly improved. At the same time, abstract eye trajectories can be transformed into intuitive distribution charts, providing teachers with a basis for understanding student learning and optimizing intervention strategies.

[0058] Currently, eye-tracking devices used in online learning generally employ technologies such as VR devices, 3D cameras, or infrared devices. However, due to their high price, limited application scenarios, and significant differences in compatibility between different brands of cameras, these hardware devices are difficult to promote and use on a large scale on commonly used terminal devices such as mobile phones, tablets, and laptops. This results in limited coverage of eye-tracking data collection, failing to provide sufficient data foundation for the visualization and analysis of group learning behaviors such as classes, grades, and online classes. It also fails to meet the needs of individual users switching between multiple terminals for learning and cannot be personalized to enhance the experience based on the individual characteristics of different users.

[0059] On the other hand, online learning today often involves one teacher facing numerous users, making it difficult for the teacher to monitor each user's learning progress in real time. Similarly, with pre-recorded lessons, teachers cannot provide timely reminders to inattentive users. Therefore, what's needed is real-time analysis during learning, providing guidance to promptly address and distract unfocused learners, rather than simply providing post-learning analysis results.

[0060] Therefore, there is an urgent need for a low-cost and high-precision eye-tracking solution that is compatible with ordinary terminal devices to meet the actual needs of online learning and education scenarios. This solution can accurately identify the area of ​​the user's gaze on the screen and generate intuitive and accurate visualization reports and trajectory charts of attention distribution. It can provide educators with a visual learning analysis tool and also provide targeted prompts and guidance to users whose attention has shifted based on the visualization results, thereby helping to improve the efficiency of online learning.

[0061] The user-side configuration is as follows: For terminal devices using ordinary 2D cameras, the camera resolution is determined by the camera hardware, commonly 1920×1080px (30fps). This also adapts to scenarios with lower device performance and limited camera hardware specifications, such as 1280×720px (25fps). Terminal devices include laptops, tablets, etc., where the 1920×1080px screen physical size is W... p =34.5cm, perpendicular to H p =19.5cm, 1280×720px screen physical dimensions are horizontal W p =30cm, perpendicular to H p=17cm; The terminal is equipped with a speaker (for voice playback), and the built-in camera can be used directly.

[0062] like Figures 1 to 4 As shown, the specific modules included are as follows to solve the above problems:

[0063] The gaze information acquisition module, in order to acquire the real-time video stream of the user's face, uses a facial landmark detection model (such as MediaPipe Face Landmarker) to extract 3D facial landmarks, locate the face, and acquire head pose and binocular region images. The binocular region images are then input into a gaze estimation model (such as 3DGazeNet), outputting a unit gaze vector representing the user's current gaze direction. Simultaneously, it quantizes and outputs head pose parameters α and β, and the vertical distance d between the user and the screen, for use by subsequent modules. The specific implementation and calculation are as follows:

[0064] 1. Real-time video stream acquisition: Using the terminal's built-in camera, a real-time video stream of the user's face is acquired. Conventional image processing methods are used to remove ambient light interference to ensure a clear video stream.

[0065] 2. Acquisition of 3D facial key points, head pose, and eye region images: The MediaPipe FaceLandmarker model is used to extract 3D key points (including depth information) on the user's face. The face region is located based on the extracted key points, and head pose information is acquired simultaneously. At the same time, based on the eye boundary key points in the facial key points, the eye region images are located and cropped. After preprocessing, the contrast of the pupils and irises is enhanced and then input into the gaze estimation model.

[0066] 3. Calculation of the vertical distance d between the user and the screen: Based on the spatial relationship between the extracted facial 3D key points (with the 3D coordinates of the eye center point as the core reference) and the screen plane, the screen plane is set as the z=0 plane in the 3D coordinate system, and the 3D coordinates of the eye center point are... (Parameter description:) The horizontal coordinates of the center point of the eye are in cm. The vertical coordinates of the center point of the eye, in cm; The depth value of the eye relative to the screen (in cm) is the vertical distance, which is the vertical distance from the center point of the eye to the screen plane. The calculation formula is:

[0067] (d is the vertical distance between the user and the screen, in cm. The absolute value is used to ensure that the distance is positive.)

[0068] Calculation process: Assume the three-dimensional coordinates of the eye center point extracted from the model are as follows: (Right now , , Substituting into the formula, we get:

[0069]

[0070] Output d=60cm for subsequent calculation of line-of-sight intersection points.

[0071] 4. Calculation of unit gaze vector g: The unit gaze vector g needs to accurately represent the user's current gaze direction. It is defined in the head coordinate system and consists of a horizontal component. Vertical component Forward component (Parameter description:) The horizontal component is positive to the right and its value ranges from [-1, 1]. The vertical component is positive downwards, and its value ranges from [-1, 1]. This is the forward component, positive when facing the screen, with a value range of (0,1]. Since it is a unit vector, it must satisfy the constraint that the vector magnitude is 1. The calculation formula is:

[0072]

[0073] The unit constraint formula is:

[0074]

[0075] Parameter value description: , , Based on the dynamic changes in the user's actual gaze direction, the gaze estimation model was used to calculate the result using binocular region images; the model's output parameters were selected for this calculation. , , (Adapted for common scenarios where users are looking directly at the screen and are slightly tilted to the lower right).

[0076] Verification calculation: Select the parameters ( , , Substituting into the unit vector constraint formula, the calculation process is as follows:

[0077]

[0078] The unit gaze vector was successfully output, meeting the unit vector constraint requirements. .

[0079] 5. Calculation of head attitude parameters α and β: Among the head attitude parameters, pitch angle α represents the vertical rotation angle of the head, and yaw angle β represents the horizontal rotation angle of the head (α and β are both angle values ​​in degrees). Both are derived from the three components of the unit gaze vector g. The vector components are converted into angle values ​​using the arctangent function. The calculation formula is as follows:

[0080]

[0081]

[0082] Parameter description: It is the arctangent function, and π is the value of pi (approximately 3.1416). , , Reusing the values ​​from the previous text, we substitute them into the formula to calculate:

[0083]

[0084]

[0085] The outputs α≈4.6° and β≈6.9° for use by the subsequent gaze mapping module.

[0086] The gaze mapping module divides the user's screen into multiple grid-like screen blocks. Based on the head pose and unit gaze vector output by the gaze information acquisition module, it calculates the intersection point between the gaze and the screen using a spatial geometric projection model. Using the screen block containing this intersection point as the center, it selects the directly adjacent screen blocks above, below, left, and right as the initial mapped gaze area. The screen blocks serve as the basic spatial units for gaze localization, and their size and number are dynamically adjusted according to the application scenario, while also flexibly adapting to the image resolution captured by the camera, as detailed below:

[0087] 1. Screen block size and quantity calculation: Screen block size (horizontal width) Vertical height ) and quantity (number of horizontal columns) , number of vertical rows The result is calculated based on dynamic adjustment of the camera resolution and the total number of pixels on the screen. The horizontal width of the block, in pixels (px). The vertical height of the block, in pixels (px). This is the number of horizontal columns, without units; (This refers to the number of vertical rows, without units). When the camera resolution is high, more columns and rows are selected to make the block size smaller; when the camera resolution is low, fewer columns and rows are selected to make the block size larger. The calculation formula is as follows:

[0088]

[0089]

[0090]

[0091] This represents the total number of horizontal pixels on the screen (in pixels, obtained by collecting data from the device screen parameters). This represents the total number of vertical pixels on the screen (in pixels, obtained by collecting data from the device screen parameters). This represents the total number of screen blocks.

[0092] This calculation uses , (1920×1080px high-resolution scene); When the camera resolution is 1920×1080px, select... , It is compatible with high-precision line-of-sight positioning; when the camera resolution is 1280×720px (low resolution), select... , To avoid excessive positioning errors due to insufficient resolution and to ensure smooth system operation, high-resolution scene parameters were selected for this calculation.

[0093] Calculation process: , , , Substituting into the formula, the calculation yields:

[0094]

[0095]

[0096]

[0097] If it is a low-resolution scene ( , ), Substitute , The calculation shows that:

[0098]

[0099]

[0100]

[0101] Output high resolution scene , This is for use in subsequent calculations.

[0102] 2. Calculation of the coordinates of the intersection point between the gaze and the screen: Based on the head pitch angle α and yaw angle β output by the gaze information acquisition module, the vertical distance d between the user and the screen, as well as the screen's physical size and resolution, the gaze direction in the head coordinate system is projected onto the screen plane using a spatial geometric projection model to obtain the pixel coordinates of the intersection point. ( The horizontal pixel coordinates of the intersection point, in pixels (px). The vertical pixel coordinates of the intersection point (in pixels) are calculated using the following formula:

[0103]

[0104]

[0105] , The screen's horizontal and vertical physical dimensions (in cm); α≈4.6°, β≈6.9°, d=60cm, calculated using the gaze information acquisition module; , .

[0106] Calculation process: First calculate the tangent value, , Then, substituting all the parameters into the formula, the result is:

[0107]

[0108]

[0109] Output the coordinates of the intersection point .

[0110] 3. Initial Mapped Gaze Area Determination: Taking the screen block where the line of sight intersects with the screen as the center, select the screen blocks directly adjacent to it in the top, bottom, left, and right directions to form the initial mapped gaze area; first, calculate the index of the block where the intersection point is located, the calculation formula is ( (This is a floor function, ensuring the index is an integer).

[0111]

[0112]

[0113] Parameter description: For block-level indexes (no unit). Vertical index for blocks (unitless); , Let these be the coordinates of the intersection point. , This refers to the block size.

[0114] Calculation process: Substitute the above parameters into the formula to calculate:

[0115]

[0116]

[0117] The intersection point is located in the block index (64,44). Taking this block as the center, select the blocks that are directly adjacent to it in the top, bottom, left and right (the indices are (64,43), (64,45), (63,44), and (65,44) respectively). Together with the central block, they form the initial mapping gaze area. The block indices contained in the initial mapping gaze area are (63,44), (64,43), (64,44), (64,45), and (65,44). Output the initial mapping gaze area for use by the subsequent calibration module.

[0118] The calibration module dynamically calibrates the initial mapped gaze area output by the gaze mapping module. This dynamic calibration is based on real-time changes in the user's head posture and the offset of the initial mapped gaze area, adjusting calibration parameters in real time and outputting a set of screen blocks corresponding to the user's current actual gaze, which serves as the final gaze area. This module comprises two sub-units: an initial calibration unit and an individual calibration unit. The specific implementation and calculation of each sub-unit are as follows:

[0119] The initial calibration unit is used to establish global correction parameters through a center-point guidance method, and to perform preliminary compensation on the initial mapped gaze area. The center-point guidance method involves displaying a unique calibration prompt point only in the center of the screen and guiding the user to gaze at this prompt point. The global correction parameters are directly generated based on the spatial offset between the initial mapped gaze area corresponding to this single calibration prompt point and the set of screen blocks corresponding to that prompt point. It includes four sub-functions: center calibration point display, offset calculation, global correction parameter generation, and compensation output. The specific calculations are as follows:

[0120] (1) Determination of the set of screen blocks corresponding to the center calibration point: combined with the screen block division parameters ( , The central calibration point corresponds to the central block index (45,30), and the corresponding block set is (44,30), (45,29), (45,30), (45,31), (46,30).

[0121] (2) Spatial offset calculation: Guide the user to gaze at the center calibration point, collect the user's initial mapped gaze area, and calculate the center coordinate offset between the block set corresponding to the center calibration point and the initial mapped gaze area; first, calculate the block center coordinates, the calculation formula is:

[0122]

[0123]

[0124] Parameter description: The horizontal center coordinates of the block (in pixels). The vertical center coordinates of the block (in pixels). , For block index; , This refers to the block size.

[0125] The formula for calculating the offset of a single calibration point (center point) is:

[0126]

[0127]

[0128] Parameter description: This is the horizontal offset (in pixels). This is the vertical offset (in pixels). , The average coordinates of the center of the block set corresponding to the center calibration point (in pixels). , The average coordinates (in pixels) of the center of the initial mapped gaze area corresponding to the center calibration point.

[0129] Calculation process: Assume the average coordinates of the center of the initial mapped gaze area corresponding to the collected center calibration point are (930px, 544.33px) (i.e. , The average center coordinates of the block set corresponding to the center calibration point are (950px, 535px) (i.e.) , Substituting into the formula, we get:

[0130]

[0131]

[0132] (Keep two decimal places to improve calibration accuracy)

[0133] (3) Global correction parameter generation and initial correction gaze area output: The global correction parameters are determined based on the center single-point offset, and the correction parameters are as follows: , (No scaling bias, only offset compensation) , The formula for correcting the center coordinates of each block in the initial mapping gaze area is:

[0134]

[0135]

[0136] Parameter description: The horizontal center coordinates (in pixels) of the corrected block. The vertical center coordinates of the corrected block (in pixels). , To correct the center coordinates of the block before correction; , To correct the offset.

[0137] Based on the corrected center coordinates Determine the initial correction gaze area and output this initial correction gaze area for use by the individual calibration unit.

[0138] The individual calibration unit, after initial calibration, constructs a user-specific stable gaze region set for each calibration point through multiple rounds of sampling. This set serves as gaze stability data. In subsequent gaze estimation, the final gaze region is determined from the corresponding gaze stability data based on the spatial proximity between the current initially calibrated gaze region and each calibration point. The unit comprises five sub-functions: multi-round sampling control, gaze region acquisition, frequency statistics, stable set construction, and nearest calibration point matching. The specific implementation and calculations are as follows:

[0139] (1) Frequency statistics and construction of stable block set: The preset sampling rounds are 3 rounds. Multiple calibration points are distributed in the screen edge and center area (nine evenly distributed positions: four corners, midpoints of four sides, and center). In each round, all calibration points are displayed in sequence, and the initial correction gaze area corresponding to each calibration point is collected (nine calibration points are selected, which can also be dynamically adjusted according to the distortion of the camera. If the distortion is high, the number of collection points can be increased). The central block index and block set corresponding to the nine calibration points are as follows:

[0140] Top left calibration point: central block index (10,10), corresponding block set (9,10), (10,9), (10,10), (10,11), (11,10);

[0141] Top right calibration point: central block index (80,10), corresponding block sets (79,10), (80,9), (80,10), (80,11), (81,10);

[0142] Bottom left calibration point: central block index (10,50), corresponding block sets (9,50), (10,49), (10,50), (10,51), (11,50);

[0143] Bottom right calibration point: central block index (80,50), corresponding block sets (79,50), (80,49), (80,50), (80,51), (81,50);

[0144] Midpoint calibration point of the upper edge: central block index (45,10), corresponding block set (44,10), (45,9), (45,10), (45,11), (46,10);

[0145] Lower edge midpoint calibration point: center block index (45,50), corresponding block set (44,50), (45,49), (45,50), (45,51), (46,50);

[0146] Left edge midpoint calibration point: central block index (10,30), corresponding block sets (9,30), (10,29), (10,30), (10,31), (11,30);

[0147] Right edge midpoint calibration point: central block index (80,30), corresponding block set (79,30), (80,29), (80,30), (80,31), (81,30);

[0148] Central calibration point: central block index (45,30), corresponding block set (44,30), (45,29), (45,30), (45,31), (46,30).

[0149] Taking the central calibration point (45,30) as an example, the initial correction gaze areas collected in 3 rounds are merged, and the coverage frequency of each block is counted (parameter description: frequency is the number of times a block is gazed upon, without unit; the preset frequency threshold is 2 times, that is, blocks with a frequency ≥ 2 times are retained). Blocks with a frequency ≥ 2 are retained to form a stable block set (assumed to be (44,30), (45,29), (45,30), (45,31), (46,30)). The other eight calibration points are constructed according to the same logic, and their respective stable block sets are constructed, finally forming stable gaze data exclusive to nine calibration points.

[0150] (2) Euclidean distance calculation and nearest calibration point matching: Calculate the Euclidean distance between the center coordinates of the current initial correction gaze area and the nine calibration points, match the initial correction gaze area to the nearest calibration point, and output the final gaze area based on the stable block set of that calibration point; wherein, the center coordinates of the initial correction gaze area are the geometric center of the screen blocks it contains, and the Euclidean distance calculation formula is:

[0151]

[0152] Parameter description: The distance is Euclidean (in pixels), k=1,2,...,9 (corresponding to 9 calibration points); , The initial coordinates of the center of the gaze area (in pixels). , The coordinates of the center of a single calibration point (in pixels).

[0153] Calculation process: Assume the center coordinates of the current initial correction gaze area are (950px, 535px) (i.e. , The center coordinates of the center calibration point (45, 30) are (950px, 535px) (i.e. , Substituting into the formula, we get:

[0154]

[0155] This distance is the smallest among the nine calibration points (assuming that the Euclidean distance between the other calibration points and the initial corrected gaze area is ≥15px). It is matched to the center calibration point. Based on the set of stable blocks at this point, the final gaze area is output as (44,30), (45,29), (45,30), (45,31), (46,30), for use by the subsequent detection module.

[0156] The detection module, based on the final gaze area output by the calibration module and the focus area associated with the current teaching content, determines whether the user is in an invalid interaction state or has a deviated attention. The focus area is set by the teacher based on the teaching content of the current teaching session; it represents the set of screen blocks the user should be focusing on, consisting of multiple adjacent screen blocks, and can be dynamically adjusted as the teaching content changes. This module includes two sub-units: an interaction state discrimination unit and an attention deviation discrimination unit. The details of each sub-unit are as follows:

[0157] The interaction state determination unit is used to determine whether the user is in a valid interaction state. When no valid face area is detected within a consecutive preset number of frames, or when the calculated line of sight does not intersect with the screen, it is determined to be an invalid interaction state and an invalid interaction state signal is output. The preset consecutive frame number threshold can be set to 30 frames (combined with the camera frame rate of 30fps, corresponding to 1 second, the specific value can be determined according to the camera status. If the camera is slow to respond due to quality or usage time, it can be appropriately increased, such as setting the preset consecutive frame number threshold to 80 frames). When no face information is collected for 30 consecutive frames, it is determined to be an invalid interaction state and an invalid interaction state signal is output.

[0158] The attention shift detection unit compares the current final gaze area with the focus area corresponding to the current teaching session, provided the user is not in an invalid interaction state. If the final gaze area and the focus area do not overlap, the attention shift is graded based on its duration. Seconds and Seconds, determined to be a slight offset; when The time interval is considered a severe offset; the core calculations are as follows:

[0159] (1) Comparison of final gaze area and focus area: The focus area is set by the teacher (assuming the focus area block index of the current teaching period is (35,25), (35,26), (36,25), (36,26), (37,25)). The final gaze area output by the calibration module is (44,30), (45,29), (45,30), (45,31), (46,30). The two block sets are compared and there are no overlapping blocks, which is determined to be attention shift.

[0160] (2) Calculation and classification of the duration of the attention shift: Based on the camera frame rate (30fps), the number of consecutive frames of attention shift is counted. (Parameter description:) The number of consecutive offset frames; For camera frame rate; The duration of the offset (in seconds) is calculated using the following formula:

[0161]

[0162] Parameter description: The preset slight offset threshold is 3 seconds (corresponding to 90 frames, i.e.) The severe offset threshold is 10 seconds (corresponding to 300 frames, i.e.) ).

[0163] Calculation process 1 (slight offset): Assume the continuous offset is 120 frames (i.e. Substituting into the formula, we get:

[0164]

[0165] because It was determined to be a slight deviation.

[0166] Calculation process 2 (severe offset): If the number of consecutive offset frames is 330 (i.e. ), the calculation yielded:

[0167]

[0168] because If the deviation is determined to be severe, the corresponding deviation classification result is output for use by the intervention module.

[0169] The intervention module, upon detecting invalid interaction states or attention deviations, triggers corresponding levels of visual prompts based on the severity of the invalid interaction state or deviation. It comprises four sub-units: a first-level prompt unit, a second-level prompt unit, a third-level prompt unit, and a prompt closure unit. The specific implementation of each sub-unit is as follows:

[0170] The first-level prompt unit triggers a prompt when a slight gaze deviation is detected, displaying a dynamic light bar at the edge of the user's screen to guide the user's gaze back to the focus area. Specifically, it determines the direction of the user's gaze deviation relative to the focus area based on the relative position of the final gaze area center and the current focus area. The formula for calculating the relative deviation direction is as follows:

[0171]

[0172]

[0173] Parameter description: This is the horizontal relative offset (in pixels). This is the vertical relative offset (in pixels). , The final average coordinates of the center of the gaze area (in pixels). , This represents the average coordinates (in pixels) of the center of the focus area.

[0174] Calculation process: Assume the final gaze area center coordinates are (950px, 535px) (i.e. , The center coordinates of the focus area are (750px, 480px) (i.e.) , Substituting into the formula, we get:

[0175]

[0176]

[0177] because For positive values If the value is positive, it indicates that the gaze is to the right or down relative to the focus area, triggering a level one prompt, displaying a gradient light bar on the left side of the screen (corresponding to a rightward offset) and a gradient light bar at the top of the screen (corresponding to a downward offset).

[0178] The secondary prompt unit is used to trigger a secondary prompt when a severe offset is detected, and to perform a black screen processing on the user terminal for the non-focus area, retaining only the content of the focus area for display; where the non-focus area is all user terminal screen blocks outside the focus area of ​​the current teaching period.

[0179] The three-level prompt unit is used to trigger a three-level prompt when an invalid interaction state is determined, and plays a voice reminder through the speaker to remind the user to return to the focus area.

[0180] The prompt-to-close unit is used to close all triggered tiered visual prompts after the user's gaze returns to the focused area and remains there for 2 seconds; the core criterion is: the preset threshold for the duration of gaze returning to the focused area is 2 seconds. (seconds, a duration threshold), when the user's gaze is detected to return to the focus area and remain there for a period of time. At a certain time, all triggered hierarchical visual prompts will be turned off.

[0181] The visualization analysis module generates learning data based on the matching results of the final gaze area and focus area output by the calibration module, and uploads this data to the teacher's end. The learning data includes at least the matching degree between the final gaze area and focus area, trigger information for ineffective interactions and attention deviations, and tiered prompt trigger records. This module comprises three sub-units: a learning report generation unit, an individual feedback generation unit, and a group feedback generation unit. The specific implementation and calculation of each sub-unit are as follows:

[0182] The learning progress report generation unit synchronously records the trigger time, trigger type, duration, and release time of tiered prompts. Combined with the real-time matching degree between the user's final gaze area and focus area, and the frequency and duration of invalid interaction states, it generates a structured digital learning progress report containing individual user identifiers, teaching time periods, and corresponding teaching content tags, and uploads it to the teacher's end. The core calculations are as follows:

[0183] Matching degree is used to characterize the degree of overlap between the user's gaze area and focus area, and the calculation formula is:

[0184]

[0185] Parameter description: For matching degree; The number of overlapping blocks between the final gaze area and the focus area; This refers to the total number of blocks in the focus area.

[0186] Calculation process: Assume the total number of blocks in the focus area The number of overlapping blocks between the final gaze area and the focus area. Substituting the values ​​into the formula yields:

[0187]

[0188] Synchronously record invalid interactions, offset prompts, and other information, generate a structural chemistry situation report, and upload it to the teacher's end.

[0189] The individual feedback generation unit analyzes a user's final gaze area data, including periods of focused attention, high-frequency deviation intervals, and cue triggering patterns, to generate a personalized review report with suggestions for attention improvement. The core calculation (high-frequency deviation interval analysis) is as follows:

[0190] Based on the final gaze area data of a single user throughout the entire process, the gaze duration of each block is calculated using the following formula:

[0191]

[0192] Parameter description: The duration of gaze on a single block (in seconds); The number of frames for a single block; The camera frame rate is used to determine the non-focused area blocks with the longest gaze duration and identify high-frequency deviation intervals.

[0193] Calculation process: Assume that the number of frames observed in block (80, 50) is 450 (i.e. Substituting into the formula, we get:

[0194]

[0195] This duration represents the longest non-focused gaze duration. This area and its surrounding regions are identified as high-frequency deviation zones. Based on the trigger patterns of the prompts, a personalized review report is generated.

[0196] The group feedback generation unit, based on the final gaze area distribution data of multiple users under the same courseware, statistically analyzes the average gaze duration and the percentage of users gazing at each area of ​​the courseware, while also identifying high-incidence nodes of attention deviation during courseware playback, generating visual attention feature analysis results for the courseware content; the core calculations are as follows:

[0197] Assuming there are 30 users under the same courseware ( (Total number of users), the formulas for calculating the average gaze duration and the percentage of users gazing at the same group are as follows:

[0198]

[0199]

[0200] The average gaze duration of the group (in seconds); the sum of the gaze durations of 30 users in the target block, that is, the total value (in seconds) of the gaze durations of all users (k=1,2,...,30) in the block. The percentage of people who are watching; The number of users who have viewed the target block; This represents the total number of users.

[0201] The average gaze duration of the group (in seconds); the sum of the gaze durations of 30 users in the target block, that is, the total value (in seconds) of the gaze durations of all users (k=1,2,...,30) in the block. The percentage of people who are watching; The number of users who have viewed the target block; This represents the total number of users.

[0202] Calculation process: Taking block (45,30) as an example, the total gaze duration of 30 users in this block is 450 seconds. Substituting this into the formula for average gaze duration, we get:

[0203]

[0204] The number of users who viewed this block was 27 (i.e. Substituting into the proportion formula, we get:

[0205]

[0206] By combining the high-frequency deviation nodes, the visual attention feature analysis results of the courseware are generated, providing a reference for teachers to optimize the courseware.

[0207] The group feedback unit synchronously generates a heatmap of group gaze (i.e., attached). Figure 4 The heatmap shown uses the entire screen as a background and integrates all gaze data from 30 users. Blue represents the area with the highest gaze frequency, and the shades of red correspond to the average gaze duration and the percentage of people gazing at the corresponding screen area. The darker the color, the longer the average gaze duration and the higher the percentage of people gazing at the screen, and vice versa. The attached heatmap can intuitively show the focus of attention of the group on each area of ​​the courseware, as well as the areas where the group's attention is most likely to deviate. It helps teachers quickly identify the most attractive and easily noticed content in the courseware, and at the same time discover the parts of the courseware that are easy to distract students and have low attention. Teachers can adjust the courseware content in these parts based on this data.

[0208] The above provides a detailed description of a real-time gaze interaction and visualization analysis system and method based on a camera, as provided by the present invention. The specific embodiments are described only to aid in understanding the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A real-time gaze interaction and visualization analysis system based on a camera, characterized in that, include: The system includes a gaze information acquisition module, a gaze mapping module, a calibration module, a detection module, an intervention module, and a visualization analysis module. The gaze information acquisition module is used to acquire real-time video stream of the user's face, extract three-dimensional facial key points using a facial key point detection model, locate the face and acquire head posture and eye region images, input the eye region images into the gaze estimation model, and output a unit gaze vector representing the user's current gaze direction. The gaze mapping module is used to divide the user's screen into multiple grid-like screen blocks, and calculate the intersection point between the gaze and the screen through a spatial geometric projection model based on the head posture and unit gaze vector. Taking the screen block where the intersection point is located as the center, the screen blocks directly adjacent to it in the upper, lower, left and right directions are selected together as the initial mapping gaze area. The screen blocks serve as the basic spatial units for gaze positioning, and their size and number are dynamically adjusted according to the application scenario. The calibration module is used to dynamically calibrate the initial mapped gaze area. The dynamic calibration is based on the user's real-time head posture changes and the offset of the initial mapped gaze area, and adjusts the calibration parameters in real time to output the set of screen blocks corresponding to the user's current real gaze, as the final gaze area. The detection module is used to determine whether the user has an invalid interaction state or attention deviation based on the final gaze area and the focus area associated with the current teaching content. The focus area is set by the teacher according to the teaching content of the current teaching period, and represents the set of screen blocks that the user should gaze at. The focus area consists of multiple adjacent screen blocks and can be dynamically adjusted as the teaching content is switched. The intervention module is used to trigger corresponding levels of visual prompts based on the degree of invalid interaction or attention deviation when an invalid interaction state or attention deviation is detected. The visualization analysis module is used to generate learning data based on the matching of the final gaze area and the focus area, and upload the learning data to the teacher's end. The learning data includes at least the matching degree of the final gaze area and the focus area, the trigger information of invalid interactions and attention deviations, and the trigger record of graded prompts.

2. The real-time gaze interaction and visualization analysis system based on a camera according to claim 1, characterized in that, The calibration module includes an initial calibration unit and individual calibration units; The initial calibration unit is used to establish global correction parameters through a center point guidance method to perform preliminary compensation on the initial mapped gaze area. The center point guidance method involves displaying a calibration prompt point in the center of the screen and guiding the user to gaze at the calibration prompt point. The global correction parameters are generated based on the spatial offset between the initial mapped gaze area and the set of screen blocks corresponding to the calibration prompt point. The individual calibration unit is used to construct a set of user-specific stable gaze blocks corresponding to multiple calibration points through multiple rounds of sampling after the initial calibration is completed, as gaze stability data. In the subsequent gaze estimation process, the final gaze area is determined from the corresponding gaze stability data based on the spatial proximity relationship between the current initial mapped gaze area and each calibration point.

3. The real-time gaze interaction and visualization analysis system based on a camera according to claim 2, characterized in that, The initial calibration unit includes: The center calibration point display subunit is used to display the initial calibration points of multiple adjacent screen blocks at the center of the screen. The offset calculation subunit guides the user to gaze at the initial calibration point, collects the user's initial mapped gaze area, and calculates the spatial offset between the user and the set of screen blocks corresponding to the initial calibration point. A global correction parameter generation subunit is used to establish global correction parameters based on the spatial offset; The compensation output subunit is used to compensate the subsequently acquired initial mapped gaze region using the global correction parameters and output the initial corrected gaze region.

4. The real-time gaze interaction and visualization analysis system based on a camera according to claim 3, characterized in that, The individual calibration unit includes: The multi-round sampling control subunit distributes calibration points in the edge and center areas of the screen, repeatedly performs a preset number of sampling rounds, and displays all calibration points in sequence in each round; The gaze area acquisition subunit is used to acquire the user's initial corrected gaze area when displaying each calibration point; The frequency statistics subunit is used to merge the initial correction gaze area collected in all rounds for each calibration point and count the frequency of each screen block being covered. A stable set construction subunit is used to retain screen blocks whose frequency reaches a preset frequency threshold, forming a stable block set corresponding to the calibration point; The nearest calibration point matching subunit is used to calculate the Euclidean distance between the center coordinates of the initial corrected gaze area and each calibration point, match the initial corrected gaze area to the nearest calibration point, and output the final gaze area based on the set of stable blocks of the calibration point, wherein the center coordinates of the initial corrected gaze area are the geometric center of the screen blocks it contains.

5. The real-time gaze interaction and visualization analysis system based on a camera according to claim 1, characterized in that, The detection module includes an interaction state discrimination unit and an attention shift discrimination unit; The interaction state determination unit determines whether the user is in a valid interaction state. When no valid face area is detected within a consecutive preset number of frames, or the calculated line of sight has no intersection with the screen, it is determined to be an invalid interaction state and an invalid interaction state signal is output. The attention shift discrimination unit compares the current final gaze area with the focus area corresponding to the current teaching period when the user is not in an invalid interaction state. If the final gaze area and the focus area corresponding to the current teaching period do not overlap, the attention shift is classified according to the duration of the attention shift. When the shift duration is ≥3 seconds and <10 seconds, it is judged as a mild shift; when the shift duration is ≥10 seconds, it is judged as a severe shift.

6. The real-time gaze interaction and visualization analysis system based on a camera according to claim 5, characterized in that, The intervention module provides tiered prompts based on the type of the received signal: The first-level prompt unit is used to trigger a first-level prompt when a slight deviation is detected. At this time, a dynamic light bar is displayed on the edge of the user's screen to guide the user's gaze back to the focus area. The secondary prompt unit is used to trigger a secondary prompt when a severe offset is determined. At this time, the non-focus area of ​​the user terminal is blacked out, and only the content of the focus area is displayed. The non-focus area refers to all user terminal screen blocks outside the focus area of ​​the current teaching period. The three-level prompt unit is used to trigger a three-level prompt when the interaction is determined to be invalid, and to play a voice prompt through the speaker to remind the user to return to the focus area; The prompt-to-close unit is used to close all triggered hierarchical visual cues after the user's gaze returns to the focus area and remains there for 2 seconds.

7. A real-time gaze interaction and visualization analysis system based on a camera according to claim 6, characterized in that, The first-level prompt unit specifically determines the direction of the user's gaze relative to the focus area based on the relative position of the final gaze area center and the current focus area. When the gaze is to the left, a gradient light bar is displayed on the right side of the screen; when the gaze is to the right, a gradient light bar is displayed on the left side of the screen; when the gaze is to the top, a gradient light bar is displayed at the bottom of the screen; and when the gaze is to the bottom, a gradient light bar is displayed at the top of the screen. The user's gaze is guided back through the dynamic light bar that is opposite to the direction of the gaze shift.

8. The real-time gaze interaction and visualization analysis system based on a camera according to claim 1, characterized in that, The visualization analysis module includes: The learning progress report generation unit is used to synchronously record the trigger time, trigger type, duration and release time of the graded prompts. Combined with the real-time matching degree between the user's final gaze area and focus area, the frequency and duration of invalid interaction states, it generates a structured digital learning progress report containing user individual identifiers, teaching time periods and corresponding teaching content tags, and uploads it to the teacher's end. The individual feedback generation unit analyzes the user's attention concentration periods, high-frequency deviation intervals, and prompt triggering patterns based on data from a single user's final gaze area, and generates a personalized review report with suggestions for attention improvement. The group feedback generation unit, based on the distribution data of the final gaze areas of multiple users under the same courseware, statistically analyzes the average gaze duration and the percentage of people gazing in each area of ​​the courseware, while identifying high-incidence nodes of attention deviation during the courseware playback process. Combining the correspondence between the display position of the courseware content and the group gaze and deviation data, it generates the visual attention feature analysis results of the courseware content.

9. A real-time gaze interaction and visualization analysis system based on a camera according to claim 1, characterized in that, The gaze mapping module is also used to obtain the image resolution captured by the user's camera, and dynamically adjust the size and number of the screen blocks according to the image resolution. The higher the image resolution, the smaller the screen block size and the greater the total number. The lower the resolution, the larger the screen block size and the fewer the total number of blocks.

10. A real-time gaze interaction and visualization analysis method based on a camera, characterized in that, The system applied to the camera-based real-time gaze interaction and visualization analysis system as described in any one of claims 1-9 includes the following steps: S101, gaze information acquisition: acquire the real-time video stream of the user's face, extract three-dimensional facial key points using the facial key point detection model, locate the face and acquire head posture and eye region images, input the eye region images into the gaze estimation model, and output a unit gaze vector representing the user's current gaze direction. S102, gaze mapping, divides the user's screen into multiple grid-like screen blocks, and calculates the intersection point between the gaze and the screen through a spatial geometric projection model based on the head posture and unit gaze vector. Taking the screen block where the intersection point is located as the center, selects the screen blocks directly adjacent to it in the upper, lower, left and right directions, and together they serve as the initial mapped gaze area. The screen blocks serve as the basic spatial units for gaze positioning, and their size and number are dynamically adjusted according to the application scenario and can also be dynamically adjusted according to the resolution of the images captured by the camera. S103, Dynamic calibration, dynamically calibrating the initial mapped gaze area, the dynamic calibration is based on the user's real-time head posture changes and the offset of the initial mapped gaze area, adjusting the calibration parameters in real time, and outputting the set of screen blocks corresponding to the user's current real gaze as the final gaze area. S104, State detection: Based on the final gaze area and the focus area associated with the current teaching content, determine whether the user has an invalid interaction state or attention deviation. The focus area is set by the teacher according to the teaching content of the current teaching period, representing the set of screen blocks that the user should gaze at. The focus area consists of multiple adjacent screen blocks and can be dynamically adjusted as the teaching content is switched. S105, graded intervention, triggers corresponding level of visual prompts based on the degree of invalid interaction or attention deviation when an invalid interaction state or attention deviation is detected; S106, Visual analysis: Based on the matching of the final gaze area and the focus area, generate learning data and upload the learning data to the teacher's end. The learning data includes at least the matching degree of the final gaze area and the focus area, invalid interaction and attention deviation trigger information, and graded prompt trigger records.

Images