Student concentration degree detection method based on facial expression recognition

Abstract

The invention relates to the cross technical field of artificial intelligence technology, computer vision technology and educational informatization technology, in particular to a student concentration degree detection method based on facial expression recognition, which comprises the following steps of multi-modal data acquisition, image preprocessing, key feature extraction, abnormal data processing and concentration degree calculation. The multi-modal data acquisition comprises visual data acquisition, behavior data acquisition and data fusion storage, and the method takes data-driven concentration assessment as a core and is implemented according to a progressive logic of acquisition, preprocessing, extraction, correction and calculation; comprehensive data is acquired through multi-modal acquisition, preprocessing interference elimination, feature extraction sacrifice the basis of evaluation, anomaly correction is accurate, and finally a result is obtained through weighted fusion; the problem that manual evaluation is subjective and low in efficiency is solved, efficient multi-scene adaptation and data desensitization are processed, privacy risks are reduced, teaching adjustment and differentiation strategies can be supported, and education digitization requirements are met.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence technology, computer vision technology, and educational informatization technology, specifically to a method for detecting student attention based on facial expression recognition. Background Technology

[0002] Traditional student attention assessment relies on teachers' subjective observation, resulting in low efficiency, inconsistent standards, and large errors. Existing detection technologies often depend on single indicators, have weak correlation with true attention levels, lack effective fusion of multimodal data, and have fixed feature weights. Furthermore, the collection of student facial and eye-tracking data lacks standardized anonymization mechanisms, posing significant privacy compliance risks and failing to meet the accurate learning monitoring needs of blended learning and MOOC platforms. Therefore, it is necessary to propose a student attention detection method based on facial expression recognition. Summary of the Invention

[0003] To address the problems in existing technologies, this invention provides a method for detecting student attention based on facial expression recognition.

[0004] The technical solution adopted by this invention to solve its technical problem is: a student attention detection method based on facial expression recognition, comprising the following steps: 1) Multimodal data acquisition: Facial video streams are acquired via computer webcam as visual data, and mouse movement and click events are acquired using PyAutoGUI tool as behavioral data; 2) Image preprocessing: The acquired facial video stream frames are sequentially subjected to grayscale conversion, normalization, and ROI cropping operations; the grayscale conversion is performed using the formula... accomplish Component weighted transformation; the normalization is achieved through the formula The grayscale values ​​are mapped to a distribution with a mean of 0 and a standard deviation of 1; the ROI extraction determines the boundary of the face region by using the coordinates of facial key points, and extracts the effective facial region; 3) Key Feature Extraction: Based on the dlib68 keypoint detection model, keypoints of the eyes (keypoints 36-41 in the left eye and keypoints 42-47 in the right eye) and facial features are extracted, and the coordinates of the gaze center and the variance of gaze stability are calculated. Facial expression feature vectors are obtained through the HOG feature extraction algorithm, which includes detection window division, gradient calculation, 9-interval orientation histogram statistics, and L2 normalization. Mouse displacement is calculated based on the Euclidean distance formula to determine the screen activity state. 4) Abnormal data handling: Nearest neighbor interpolation is used to fill in missing data when face detection is interrupted; a sliding window is constructed based on timestamps to align the time dimensions of expression, gaze, and screen activity data; 5) Attention Calculation: An SVM classifier is used to classify the HOG facial expression feature vectors and output attentive / distracted facial expression labels; a multimodal feature weighted fusion model is constructed to calculate the attention score with weights of 0.4 for facial expression attention, 0.3 for gaze stability, and 0.3 for screen activity, and the final attention level is output through a threshold decision function.

[0005] Specifically, the facial video stream acquisition parameters in the multimodal data acquisition are set to 640×480 resolution, 30fps frame rate, mouse displacement detection threshold is set to 5 pixels, and sampling frequency is set to 10Hz.

[0006] Specifically, during the multimodal data acquisition process, eye coordinates, mouse position, and facial feature data at the same moment are fused into a multidimensional vector and stored using timestamps to avoid data misalignment.

[0007] Specifically, the line-of-sight stability calculation is performed using the formula... Implementation, in which , The number of sampling frames, For the first Frame view center coordinates, The mean of the coordinates. A higher value indicates a more stable line of sight.

[0008] Specifically, the hyperplane equation of the SVM classifier is: The classification decision function is ;when When, it is judged as a focused expression, when At that time, it was judged as a distracted expression.

[0009] Specifically, the focus scoring formula for the multimodal feature weighted fusion model is as follows: ,in Rate focus level The quantitative value corresponding to the expression classification result. This is a calculated value for line-of-sight stability. This is the quantized value corresponding to the screen activity state.

[0010] The beneficial effects of this invention are as follows: The student attention detection method based on facial expression recognition described in this invention takes "data-driven attention assessment" as its core and is implemented according to the progressive logic of "data collection → preprocessing → feature extraction → anomaly correction → comprehensive calculation," which has many beneficial effects: multimodal collection ensures the comprehensiveness of the original data; preprocessing eliminates interference and purifies the data; feature extraction provides accurate basis for assessment; anomaly correction ensures data accuracy; and finally, weighted fusion outputs the results. It overcomes the subjective inefficiency of manual assessment, improving the objectivity and accuracy of detection; data processing is efficient and adaptable to multiple teaching scenarios; data anonymization reduces privacy risks; and it can also support teachers in adjusting teaching and formulating differentiated strategies in real time, meeting the needs of digital education and helping to improve teaching quality and student monitoring levels. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0012] Figure 1 A flowchart of a student attention detection method based on facial expression recognition provided by the present invention; Figure 2 This is a schematic diagram of a multimodal feature fusion model for a student attention detection method based on facial expression recognition provided by the present invention. Detailed Implementation

[0013] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0014] like Figures 1-2 As shown, the present invention provides the following technical solution: Example: A method for detecting student attention based on facial expression recognition, comprising the following steps: 1) Multimodal data acquisition: Facial video streams are acquired via computer webcam as visual data, and mouse movement and click events are acquired using PyAutoGUI tool as behavioral data; 2) Image preprocessing: The acquired facial video stream frames are sequentially subjected to grayscale conversion, normalization, and ROI cropping operations; grayscale conversion is performed using the formula... accomplish Component weighting transformation; normalization via formula The grayscale values ​​are mapped to a distribution with a mean of 0 and a standard deviation of 1; the ROI is extracted by determining the boundary of the face region using the coordinates of facial key points, and the effective facial region is extracted. 3) Key Feature Extraction: Based on the dlib68 keypoint detection model, keypoints of the eyes (keypoints 36-41 in the left eye and keypoints 42-47 in the right eye) and facial features are extracted, and the coordinates of the gaze center and the variance of gaze stability are calculated. Facial expression feature vectors are obtained through the HOG feature extraction algorithm, which includes detection window division, gradient calculation, 9-interval orientation histogram statistics, and L2 normalization. Mouse displacement is calculated based on the Euclidean distance formula to determine the screen activity state. 4) Abnormal data handling: Nearest neighbor interpolation is used to fill in missing data when face detection is interrupted; a sliding window is constructed based on timestamps to align the time dimensions of expression, gaze, and screen activity data; 5) Attention Calculation: An SVM classifier is used to classify the HOG facial expression feature vectors and output attentive / distracted facial expression labels; a multimodal feature weighted fusion model is constructed to calculate the attention score with weights of 0.4 for facial expression attention, 0.3 for gaze stability, and 0.3 for screen activity, and the final attention level is output through a threshold decision function.

[0015] Among them, the facial video stream acquisition parameters in multimodal data acquisition are set to 640×480 resolution, 30fps frame rate, mouse displacement detection threshold is set to 5 pixels, and sampling frequency is set to 10Hz.

[0016] During the multimodal data acquisition process, eye coordinates, mouse position, and facial feature data at the same moment are fused into a multidimensional vector and stored using timestamps to avoid data misalignment.

[0017] Among them, line-of-sight stability is calculated using the formula Implementation, in which , The number of sampling frames, For the first Frame view center coordinates, The mean of the coordinates. A higher value indicates a more stable line of sight.

[0018] The hyperplane equation of the SVM classifier is: The classification decision function is ;when When, it is judged as a focused expression, when At that time, it was judged as a distracted expression.

[0019] The focus scoring formula for the multimodal feature weighted fusion model is as follows: ,in Rate focus level The quantitative value corresponding to the expression classification result. This is a calculated value for line-of-sight stability. This is the quantized value corresponding to the screen activity state.

[0020] When using it, the following steps are included: Step 1: Multimodal Data Acquisition – The “Data Input Layer” of the Detection Process. This involves constructing a visual-behavioral dual-dimensional data acquisition framework to obtain the raw data required for attention assessment. A. Visual Data Acquisition: A computer webcam supporting multiple platforms such as Windows / Linux / macOS is used to acquire facial video streams at a resolution of 640×480 and a frame rate of 30fps; the face is located using the dlib68 key point detection model. If no face is detected, an abnormal prompt is triggered (such as camera obstruction or dim lighting). If the detection is successful, the coordinates of the eye area (key points 36-41, 42-47), the face (key points 17-21 of the eyebrows, forehead, and cheeks), and eye movement data are extracted. B. Behavioral data collection: PyAutoGUI tool is used to listen to mouse events at a sampling frequency of 10Hz and record mouse coordinates and click data; a displacement threshold of 5 pixels is set (in accordance with the "static" definition of Microsoft HID specification). If the displacement exceeds the threshold, it is marked as screen active state (1), otherwise it is marked as static (0). Data acquisition equipment and parameters:

[0021] C. Data fusion and storage: Eye coordinates, mouse position, and facial feature data at the same time are fused into a multi-dimensional vector according to the timestamp to avoid data misalignment; The output "fused multimodal data" is directly used as the input for "Step 2: Image Preprocessing". The facial video stream frame images enter the image preprocessing stage, and the mouse behavior data is temporarily stored and will be synchronized with the feature data later. The second step, image preprocessing—the "data purification layer" of the detection process—eliminates environmental interference and simplifies data dimensions for the facial video stream frames acquired in the first step. The specific operations are as follows: D. Grayscale conversion: using the formula Converting RGB color images to grayscale images (adapting to the human eye's highest sensitivity to green and lowest sensitivity to blue) allows each pixel to be represented by a grayscale value of 0-255, reducing the amount of data and highlighting image brightness and contrast. E. Normalization: To address feature biases caused by differences in illumination, a formula is used. ( The average gray level of the image. The grayscale values ​​are standardized to a distribution with a mean of 0 and a standard deviation of 1 (mean 0, standard deviation 1) to eliminate interference from strong light and backlight. F. ROI Extraction: Based on the facial landmark coordinates extracted in step 1, determine the boundary of the facial region (ROI). For up and down coordinates, (using left and right column coordinates), extract the effective facial area and remove "junk data" such as hair and background; The output "normalized grayscale image after ROI cropping" is used as input for "HOG expression feature extraction" and "gaze feature extraction" in "Step 3: Key Feature Extraction", providing high-purity facial image data for feature extraction; The third step, key feature extraction—the "evaluation layer" of the detection process—is based on the image data preprocessed in the second step, combined with the mouse behavior data temporarily stored in the first step, to extract three core features: facial expression, gaze, and screen activity, as detailed below: G and HOG expression feature extraction: Divide the ROI image output from step 2 into 16×16 pixel cells and calculate the gradient direction and magnitude of each pixel; divide 0°-180° into 9 directional intervals and calculate the sum of gradient magnitudes in each interval to form a cell histogram; combine adjacent cells into blocks and perform L2 normalization (to eliminate illumination differences), gradient value truncation (to avoid strong edges dominating features), and secondary L2 normalization in sequence; concatenate the histograms of all blocks to form the HOG expression feature vector. H. Gait Feature Extraction: Based on the dlib keypoint coordinates (left eyes 36-41, right eyes 42-47) from step 1, calculate the coordinates of the centers of both eyes and take the average as the gaze center; then, using the formula... To prevent division by zero errors, The number of sampling frames, For the first Frame view center coordinates, Calculate line-of-sight stability using the coordinate mean. A higher value indicates a more stable line of sight; I. Screen Activity Feature Extraction: Retrieve the mouse data temporarily stored in step 1, and use the Euclidean distance formula... Calculate mouse displacement; mark 1 if the displacement exceeds a 5-pixel threshold, otherwise mark 0 to quantify screen activity state. The three types of feature data output, namely "HOG facial expression feature vector", "eye gaze stability value" and "screen activity state value", are directly input into "Step 4: Abnormal data processing" for data continuity and consistency correction. Step 4, Anomaly Handling – The “Data Correction Layer” of the detection process, addresses anomalies encountered during the data acquisition process for the three types of feature data extracted in Step 3, ensuring data quality: J. Face loss compensation: If face detection is interrupted in step 1 (e.g., students look down, turn their faces to the side, or occlude), resulting in step 3 being unable to extract expression / gazing features, the nearest neighbor interpolation method is used to fill the missing frame with the effective expression features and gaze stability values ​​of the previous frame to avoid data breakage. K. Data Synchronization and Alignment: Since the sampling frequencies of the three types of features are different (facial expression features correspond to 30Hz, gaze features correspond to 15Hz, and screen activity features correspond to 10Hz), a sliding window is constructed based on the timestamps in step 1 to allocate the three types of feature data to the corresponding time windows, thereby achieving time dimension alignment; The output "corrected synchronization feature data" (including aligned facial expressions, gaze, and screen activity features) serves as the core input for "step five, attention calculation," ensuring that there is no missing or misaligned data during the calculation. Step 5, Attention Calculation – The “Result Output Layer” of the testing process. Based on the synchronized feature data corrected in Step 4, the student's attention score and level are output through classification model and weighted fusion. L. Facial Expression Recognition: The HOG facial expression feature vector corrected in step 4 is input into a pre-trained SVM classifier; the SVM uses the optimal hyperplane (equation) The decision function is to classify the data into categories. ; It is judged as a focused expression (such as concentrated eyes and slightly furrowed brows). Distracted expressions (such as wandering eyes and frequent blinking) are identified and the classification results are quantified as an expression focus value of 0-1; M. Multimodal Weighted Fusion: Based on affective computing theory (facial expressions are direct physiological signals, weighted at 0.4), Tobii eye-tracking clinical research (gaze stability weighted at 0.3), and human-computer interaction theory (screen activity weighted at 0.3), using formulas... ( Rate focus level This represents the expression focus value. This is the line-of-sight stability value. Calculate an attention score (value 0-1) based on the screen activity status value. Weighting matrix:

[0022] N. Threshold Decision: Setting a scoring threshold — ≥0.7 indicates "focus", 0.3 < <0.7 indicates "general focus". A score of ≤0.3 indicates "lack of focus," and the final focus level is output to provide a basis for teaching intervention.

[0023] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting student attention based on facial expression recognition, characterized in that, Includes the following steps: 1) Multimodal data acquisition: Facial video streams are acquired via computer webcam as visual data, and mouse movement and click events are acquired using PyAutoGUI tool as behavioral data; 2) Image preprocessing: The acquired facial video stream frames are sequentially subjected to grayscale conversion, normalization, and ROI cropping operations; the grayscale conversion is performed using the formula... accomplish Component weighted transformation; the normalization is achieved through the formula The grayscale values ​​are mapped to a distribution with a mean of 0 and a standard deviation of 1; the ROI extraction determines the boundary of the face region by using the coordinates of facial key points, and extracts the effective facial region; 3) Key feature extraction: Based on the dlib68 key point detection model, extract eye and facial feature points, calculate the coordinates of the gaze center and the variance of gaze stability; obtain facial expression feature vectors through the HOG feature extraction algorithm, which includes detection window division, gradient calculation, 9-interval orientation histogram statistics and L2 normalization operation; calculate mouse displacement based on the Euclidean distance formula to determine the screen activity state. 4) Abnormal data handling: Nearest neighbor interpolation is used to fill in missing data when face detection is interrupted; A sliding window is built based on timestamps to align the time dimensions of facial expressions, gaze, and screen activity data. 5) Attention Calculation: An SVM classifier is used to classify the HOG facial expression feature vectors and output attentive / distracted facial expression labels; a multimodal feature weighted fusion model is constructed to calculate the attention score with weights of 0.4 for facial expression attention, 0.3 for gaze stability, and 0.3 for screen activity, and the final attention level is output through a threshold decision function.

2. The student attention detection method based on facial expression recognition according to claim 1, characterized in that: The facial video stream acquisition parameters in the multimodal data acquisition are set to 640×480 resolution, 30fps frame rate, mouse displacement detection threshold to 5 pixels, and sampling frequency to 10Hz.

3. The student attention detection method based on facial expression recognition according to claim 1, characterized in that: During the multimodal data acquisition process, eye coordinates, mouse position, and facial feature data at the same moment are fused into a multidimensional vector and stored using timestamps to avoid data misalignment.

4. The student attention detection method based on facial expression recognition according to claim 1, characterized in that: The line-of-sight stability calculation is performed using the formula Implementation, in which , The number of sampling frames, For the first Frame view center coordinates, The mean of the coordinates. A higher value indicates a more stable line of sight.

5. The student attention detection method based on facial expression recognition according to claim 1, characterized in that: The hyperplane equation of the SVM classifier is: The classification decision function is ;when When, it is judged as a focused expression, when At that time, it was judged as a distracted expression.

6. The student attention detection method based on facial expression recognition according to claim 1, characterized in that: The focus scoring formula for the multimodal feature weighted fusion model is as follows: ,in Rate focus level The quantitative value corresponding to the expression classification result. This is a calculated value for line-of-sight stability. This is the quantized value corresponding to the screen activity state.

Images