Fatigue detection and learning content adaptation system based on eye movement tracking

Abstract

The invention provides a fatigue detection and learning content adaptation system based on eye movement tracking, and belongs to the field of biometric feature recognition technology, computer program design and content adaptation algorithms. Eye movement data of a user is captured through a camera, the fatigue degree is judged in combination with an attention model, automatic adjustment or rest reminding of learning content is achieved, and user experience is improved. And the learning efficiency and experience of the user are improved.

Description

Technical Field

[0001] The present invention relates to the fields of biometric recognition technology, computer programming, and content adaptation algorithms, and in particular to a fatigue detection and learning content adaptation system based on eye tracking, which can be applied to scenarios such as online course platforms and e-readers to improve users' learning efficiency. Background Art

[0002] In today's era of digital learning, learning tools such as online course platforms and e-readers have been widely used. However, when users use these tools for learning for a long time, they are prone to fatigue, resulting in reduced learning efficiency. Currently, although there are some fatigue detection methods, most of them have problems such as inaccurate detection, significant interference to users, or insufficient intelligent content adaptation. For example, some methods detect fatigue by monitoring the user's facial expressions and head posture, but these methods are greatly affected by external factors such as light and user posture; some methods require users to wear additional equipment, such as EEG sensors, which increases the cost and inconvenience of use. Therefore, there is an urgent need for a method that can accurately detect the user's fatigue state and automatically adapt the learning content according to the fatigue state. Summary of the Invention

[0003] To address the above technical issues, this paper provides a fatigue detection and learning content adaptation system based on eye tracking. By capturing user eye movement data with a camera and combining it with an attention model to determine fatigue levels, the system automatically adjusts learning content or provides break reminders, thereby improving user learning efficiency and experience.

[0004] The technical solution of the present invention is:

[0005] A fatigue detection and learning content adaptation system based on eye tracking, comprising:

[0006] Eye movement data acquisition module: It consists of a camera and an image recognition processing unit. The camera is used to collect user eye images. The image recognition processing unit processes the images, extracts eye movement data such as blink frequency and gaze dwell time, and transmits the data to the fatigue detection module.

[0007] Fatigue detection module: contains an attention model, which is used to receive eye movement data transmitted by the eye movement data acquisition module, calculate the user's fatigue level according to the preset fatigue judgment rules and fatigue level calculation formula, and send the result to the learning content adaptation module.

[0008] Learning content adaptation module: Based on the fatigue level results sent by the fatigue detection module, the learning content is adapted and adjusted according to the preset adjustment rules, such as controlling the video playback speed, adding interactive questions, and triggering rest reminders.

[0009] Further,

[0010] Eye movement data collection

[0011] The user's eye images are collected in real time through the camera installed on the learning device; the eye images are processed using image processing algorithms to extract eye movement data, including blink frequency and gaze dwell time, and the data is transmitted to the fatigue detection module.

[0012] Image processing algorithms:

[0013] The eye area is detected and located using a cascade classifier based on Haar features. The specific steps are as follows:

[0014] Grayscale the input image to reduce the amount of image data and improve processing speed;

[0015] Use Gaussian filter to smooth the grayscale image and remove noise interference;

[0016] The Haar feature extraction operator is used to extract the features of the eye area, and the eye area is detected and located through a cascade classifier to obtain the coordinate position and range of the eye.

[0017] After locating the eye area, the open and closed states of the eyes are further judged to calculate the blink frequency.

[0018] A threshold-based method is used to binarize the grayscale image of the eye area, and the ratio of the height when the eyes are open to the height when they are closed is calculated. When the ratio is less than the preset threshold, it is judged to be in the eyes closed state.

[0019] To calculate the gaze dwell time, the different areas of learning content are first determined; by locating the eye area and detecting the pupil center, the coordinate position of the pupil on the screen is determined to determine whether the pupil stays in a certain learning content area; when the pupil stays in the area for a preset time interval, the gaze dwell time is calculated, that is, the time interval from the pupil entering the area to leaving the area.

[0020] Further,

[0021] Fatigue detection

[0022] Combined with the attention model, the collected eye movement data is used to determine the user's fatigue level. The attention model uses recurrent neural networks or long short-term memory networks in deep learning to capture the time series characteristics of eye movement data.

[0023] Specifically include

[0024] 1) Attention model construction

[0025] Take the blink frequency f and the gaze fixation time T as input features. After normalization, they are input into the attention model. The structure of the attention model includes an input layer, a hidden layer, and an output layer. The input layer receives the normalized eye movement data. The hidden layer consists of several LSTM units, which are used to learn the time series features and long-term dependencies of the eye movement data. The output layer outputs the probability value of the user's fatigue level through a fully connected layer and an activation function. The fatigue level is divided into several grades.

[0026] 2) Fatigue level judgment algorithm

[0027] Let the output of the attention model be the fatigue level probability vector P = [p1, p2, p3], where p1, p2, and p3 represent the probabilities of mild fatigue, moderate fatigue, and severe fatigue respectively. When p2 or p3 is greater than the preset threshold, it is judged that the user is in a fatigue state, and the fatigue level is determined according to the probability value.

[0028] It is also possible to make a comprehensive judgment by combining factors such as the user's learning time and the difficulty of the learning content. Let the learning time be t_s and the difficulty of the learning content be d. Then the comprehensive judgment formula for fatigue detection is:

[0029] F = α * f + β * (1 / T) + γ * t_s + δ * d

[0030] Among them, α, β, γ, and δ are weight coefficients, which are determined by training with experimental data. The larger the F value, the higher the user's fatigue level.

[0031] Furthermore,

[0032] Learning content adaptation

[0033] According to the results of fatigue detection, automatically adjust the learning content or trigger a rest reminder, specifically including:

[0034] 1) Learning content adjustment

[0035] When the user is mildly fatigued, reduce the video playback speed. Let the original video playback speed be v0 and the adjusted playback speed be v1. Then v1 = v0 * k, where k is the speed adjustment coefficient, 0 < k < 1, and the specific value is determined according to the fatigue level.

[0036] When the user is moderately fatigued, increase the number and frequency of interactive questions. The content of the interactive questions is generated according to the current learned knowledge points, ensuring relevance to the learning content. Let the original interval between the appearance of interactive questions be t0 and the adjusted interval be t1. Then t1 = t0 / m, where m is the interactive question frequency adjustment coefficient, m > 1, and is determined according to the fatigue level.

[0037] When the user is severely fatigued, trigger a rest reminder.

[0038] 2) Rest reminder mechanism

[0039] The length of the break reminder is determined by the fatigue level; after the break, the user can choose to continue studying or extend the break.

[0040] The beneficial effects of the present invention are

[0041] The present invention captures the user's eye movement data through a camera, without the user having to wear additional equipment, and is non-contact and non-interference; it combines attention models and complex algorithms to judge the degree of fatigue, thereby improving the accuracy of fatigue detection; it automatically adjusts learning content or triggers rest reminders based on fatigue detection results, which can effectively improve the user's learning efficiency and experience, and has broad application prospects and market value. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] The present invention provides a fatigue detection and learning content adaptation system based on eye tracking, comprising:

[0044] 1. Eye movement data acquisition module: It consists of a camera and an image recognition processing unit. The camera is used to collect user eye images. The image recognition processing unit processes the images, extracts eye movement data such as blink frequency and gaze dwell time, and transmits the data to the fatigue detection module.

[0045] 2. Fatigue detection module: Contains an attention model, which is used to receive eye movement data transmitted by the eye movement data acquisition module, calculate the user's fatigue level according to the preset fatigue judgment rules and fatigue level calculation formula, and send the result to the learning content adaptation module.

[0046] 3. Learning content adaptation module: Based on the fatigue level results sent by the fatigue detection module, the learning content is adapted according to the preset adjustment rules, such as controlling the video playback speed, adding interactive questions, and triggering break reminders.

[0047] The details are as follows:

[0048] (1) Eye movement data collection

[0049] The user's eye images are collected in real time through the camera installed on the learning device (such as a computer, tablet, e-reader, etc.). The eye images are processed using image processing algorithms to extract eye movement data, including blink frequency and gaze dwell time, and the data is transmitted to the fatigue detection module.

[0050] Image processing algorithms:

[0051] The eye area is detected and located using a cascade classifier based on Haar features. The specific steps are as follows:

[0052] Grayscale the input image to reduce the amount of image data and improve processing speed;

[0053] Use Gaussian filter to smooth the grayscale image and remove noise interference;

[0054] Use the Haar feature extraction operator to extract the features of the eye area, and use the cascade classifier to detect and locate the eye area to obtain the coordinate position and range of the eye.

[0055] After locating the eye area, the eye's open and closed states are further determined to calculate the blink frequency. Using a threshold-based method, the grayscale image of the eye area is binarized and the ratio of the eye's height when open to the eye's height when closed is calculated. When this ratio is less than a preset threshold, the eye is considered closed. Let N be the number of eye closures within a period of time (e.g., t seconds), then the blink frequency f = N / t.

[0056] To calculate gaze dwell time, we first identify the different areas of learning content (such as the video area, text area, and interactive question area in online courses). By locating the eye area and detecting the pupil center, we determine the pupil's coordinate position on the screen and determine whether the pupil is within a specific learning content area. When the pupil stays in the area for a preset time interval (such as Δt), we begin calculating the gaze dwell time T, which is the time interval from the pupil entering the area to leaving it.

[0057] (2) Fatigue detection

[0058] Combined with an attention model, the collected eye movement data (blink frequency and gaze dwell time) is used to determine the user's fatigue level. The attention model uses recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) in deep learning to capture the time series characteristics of eye movement data.

[0059] ① Attention model construction

[0060] The blink frequency f and the fixation time T are used as input features. After normalization, they are input into the attention model. The structure of the attention model includes an input layer, a hidden layer, and an output layer. The input layer receives the normalized eye movement data; the hidden layer consists of multiple LSTM units, which are used to learn the time series features and long-term dependencies of the eye movement data; the output layer outputs the probability value of the user's fatigue level through a fully connected layer and an activation function. The fatigue level is divided into multiple levels, such as mild fatigue, moderate fatigue, and severe fatigue.

[0061] ② Fatigue level judgment algorithm

[0062] Let the output of the attention model be the fatigue level probability vector P = [p1, p2, p3], where p1, p2, and p3 represent the probabilities of mild fatigue, moderate fatigue, and severe fatigue respectively. When p2 or p3 is greater than the preset threshold, it is judged that the user is in a fatigue state, and the fatigue level is determined according to the probability value.

[0063] At the same time, in order to improve the accuracy of fatigue detection, factors such as the user's learning time and the difficulty of the learning content can also be combined for comprehensive judgment. Let the learning time be t_s and the difficulty of the learning content be d (which can be quantified by indicators such as content complexity and the number of knowledge points), then the comprehensive judgment formula for fatigue detection is:

[0064] F = α * f + β * (1 / T) + γ * t_s + δ * d

[0065] Among them, α, β, γ, and δ are weight coefficients, which are determined by training with experimental data. The larger the F value, the higher the fatigue level of the user.

[0066] (3) Learning content adaptation

[0067] According to the results of fatigue detection, the learning content is automatically adjusted or a rest reminder is triggered.

[0068] ① Learning content adjustment

[0069] When the user is in mild fatigue, the video playback speed is reduced. Let the original video playback speed be v0 and the adjusted playback speed be v1, then v1 = v0 * k, where k is the speed adjustment coefficient, 0 < k < 1, and the specific value is determined according to the fatigue level.

[0070] When the user is in moderate fatigue, the number and frequency of interactive questions are increased. The content of the interactive questions is generated according to the current learned knowledge points, ensuring relevance to the learning content. Let the original interval between the appearance of interactive questions be t0 and the adjusted interval be t1, then t1 = t0 / m, where m is the interactive question frequency adjustment coefficient, m > 1, and it is determined according to the fatigue level.

[0071] When the user is severely fatigued, a break reminder is triggered, such as a pop-up break reminder box, prompting the user to take a break for a while and pausing the playback of the learning content.

[0072] ② Rest reminder mechanism

[0073] The duration of the break reminder is determined by the fatigue level: 5-10 minutes for mild fatigue, 10-15 minutes for moderate fatigue, and 15-20 minutes for severe fatigue. After the break, users can choose to continue studying or extend the break.

[0074] The above description is only a preferred embodiment of the present invention and is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A fatigue detection and learning content adaptation system based on eye tracking, characterized in that: include: Eye movement data acquisition module: This module consists of a camera and an image recognition processing unit. The camera is used to collect user eye images. The image recognition processing unit processes the images, extracts eye movement data such as blink frequency and gaze dwell time, and transmits the data to the fatigue detection module. Fatigue detection module: This module includes an attention model and is used to receive eye movement data transmitted by the eye movement data acquisition module, calculate the user's fatigue level based on preset fatigue judgment rules and fatigue level calculation formula, and send the result to the learning content adaptation module; Learning content adaptation module: Based on the fatigue level results sent by the fatigue detection module, the learning content is adapted and adjusted according to the preset adjustment rules.

2. The system according to claim 1, wherein: Eye movement data collection The user's eye images are collected in real time through the camera installed on the learning device; the eye images are processed using image processing algorithms to extract eye movement data, including blink frequency and gaze dwell time, and the data is transmitted to the fatigue detection module.

3. The system according to claim 2, characterized in that Image processing algorithms: The eye area is detected and located using a cascade classifier based on Haar features. The specific steps are as follows: Grayscale the input image to reduce the amount of image data and improve processing speed; Use Gaussian filter to smooth the grayscale image and remove noise interference; The Haar feature extraction operator is used to extract the features of the eye area, and the eye area is detected and located through a cascade classifier to obtain the coordinate position and range of the eye.

4. The system according to claim 3, characterized in that After locating the eye area, the open and closed states of the eyes are further judged to calculate the blink frequency.

5. The system according to claim 4, characterized in that A threshold-based method is used to binarize the grayscale image of the eye area, and the ratio of the height when the eyes are open to the height when they are closed is calculated. When the ratio is less than the preset threshold, it is judged to be in the eyes closed state.

6. The system according to claim 5, characterized in that To calculate the gaze dwell time, we first identify the different areas of the learning content. By locating the eye area and detecting the pupil center, we determine the pupil's coordinate position on the screen and judge whether the pupil stays in a certain learning content area. When the pupil stays in the area for a preset time interval, the gaze dwell time is calculated, that is, the time interval from the pupil entering the area to leaving the area.

7. The system according to claim 1, wherein: Fatigue detection Combined with the attention model, the collected eye movement data is used to judge the user's fatigue level; the attention model uses the recurrent neural network or long short-term memory network in deep learning to capture the time series characteristics of eye movement data.

8. The system according to claim 7, characterized in that Specifically include 1) Attention model construction The blink frequency f and the gaze fixation time T are used as input features. After normalization, they are input into the attention model. The structure of the attention model includes an input layer, a hidden layer, and an output layer. The input layer receives the normalized eye movement data. The hidden layer consists of several LSTM units, which are used to learn the time series features and long-term dependencies of the eye movement data. The output layer outputs the probability value of the user's fatigue level through a fully connected layer and an activation function. The fatigue level is divided into several grades. 2) Fatigue level judgment algorithm Let the output of the attention model be the fatigue level probability vector P = [p1, p2, p3], where p1, p2, and p3 represent the probabilities of mild fatigue, moderate fatigue, and severe fatigue respectively. When p2 or p3 is greater than a preset threshold, it is determined that the user is in a fatigue state, and the fatigue level is determined according to the probability value.

9. The system according to claim 8, wherein It is also possible to make a comprehensive judgment by combining factors such as the user's learning time and the difficulty of the learning content. Let the learning time be t_s and the difficulty of the learning content be d. Then the comprehensive judgment formula for fatigue detection is: F = α * f + β * (1 / T) + γ * t_s + δ * d Where α, β, γ, and δ are weight coefficients, which are determined by training with experimental data. The larger the F value, the higher the fatigue level of the user.

10. The system according to claim 1, wherein Learning content adaptation According to the results of fatigue detection, the learning content is automatically adjusted or a rest reminder is triggered, specifically including: 1) Learning content adjustment When the user is in a state of mild fatigue, the video playback speed is reduced. Let the original video playback speed be v0 and the adjusted playback speed be v1. Then v1 = v0 * k, where k is the speed adjustment coefficient, 0 < k < 1, and the specific value is determined according to the fatigue level. When the user is in a state of moderate fatigue, the number and frequency of interactive questions are increased. The content of the interactive questions is generated according to the current learned knowledge points to ensure relevance to the learning content. Let the original interval between the appearance of interactive questions be t0 and the adjusted interval be t1. Then t1 = t0 / m, where m is the interactive question frequency adjustment coefficient, m > 1, and is determined according to the fatigue level. When the user is in a state of severe fatigue, a rest reminder is triggered. 2) Rest reminder mechanism The duration of the rest reminder is determined according to the fatigue level. After the rest time ends, the user can choose to continue learning or extend the rest time.