Learner doze detection algorithm based on multiple facial features

Abstract

The invention discloses a learner sleepiness state recognition algorithm based on multiple facial features. Firstly, based on a face key point model, the aspect ratio of eyes is calculated, the aspect ratio change rate is calculated through the distance between two wave crests, and a blink change state model is constructed. Secondly, an Euler angle is calculated through a Rodrigues rotation formula, motion features are expressed, meanwhile, a deviation angle is calculated, motion track data of the head in three dimensions are formed, and head motion time sequence features are obtained after fusion; and finally, segmenting a mouth region through the key points, calculating to obtain static area ratio and dynamic movement distance data of the mouth of the learner, and fusing to obtain a yawn feature model. According to input video data, feature data are subjected to weighted fusion and then are input into an XGBoost machine learning model, and a sleepiness state recognition model is obtained through parameter optimization. A test video data experiment shows that the recognition rate of the algorithm reaches 94%. According to the invention, real-time and accurate student in-class process data can be provided for teachers or educational administration supervision and management departments, and the method is of great significance to accurate management of classroom teaching and analysis of the learning efficiency of students.

Description

Technical Field

[0001] The present invention mainly relates to image processing technology, and in particular to a learner's sleepiness state detection algorithm based on multiple facial features. Background Art

[0002] In the traditional teaching model, in a real classroom environment, teachers can observe students' learning status in time and intervene in students' passive learning status in time, which is conducive to the development of teaching. Although the current reform of online teaching model has brought extremely high flexibility compared with traditional teaching, due to the separation of time and space, there is a lack of real-time feedback and interaction in online classrooms, the teacher's intervention in students' learning is weakened, and the interaction between students is poor, which is not conducive to the teacher's grasp of students' learning progress. In the case of learners' lack of self-management ability, the camera can be used to record the learners' relevant information in class. Through a specific in-class status recognition algorithm, learners can be reminded when they have negative classroom behavior. It is also convenient for the review of classroom teaching after class to achieve good teacher-student interaction.

[0003] Studies have shown that in educational scenarios, the learner's wakefulness is positively correlated with the learning effect. When students fall asleep in class, it means that they cannot concentrate at this time, which will greatly affect the learning effect. Common classroom sleepiness behaviors are erratic eyes, irregular eye blinking, that is, blinking in an attempt to wake oneself up, regular head nodding, and frequent yawning. Therefore, based on the analysis of learners' classroom sleepiness, this paper locates the eye area through the PFLD key point model, calculates the opening and closing behavior of the eyes over time, constructs a blink rate index, and determines whether the learner has blinked in a sleepy state; at the same time, the head posture estimation model is used to calculate the Euler angle that describes the head offset, and the head movement time series characteristics are constructed; and the learner's mouth characteristics in the two dimensions of movement speed and movement amplitude are calculated to conduct a comprehensive analysis of the learner's yawning behavior. Summary of the invention

[0004] Purpose of the invention: The present invention proposes an algorithm for identifying the sleepiness state of learners based on multiple facial features. The learner's sleepiness state is automatically annotated by using multiple facial regions, namely the blink rate characteristics of the eyes, the timing characteristics of the head's movement, and the yawning state characteristics of the mouth. In the invention, the entire implementation process is completed by the system. As long as a video is input, the algorithm can be used to perform relevant analysis and annotate the learner's sleepiness state in real time.

[0005] The technical solution adopted by the present invention is as follows:

[0006] Step 1: For the video frame image of the learner's online class, use PFLD to extract the key points of the face in the image and obtain 98 facial key points.

[0007] Step 2: Use the key points obtained in step 1 to locate the eye area, calculate the aspect ratio of the eye, and draw a curve showing the change of the degree of eye opening and closing over time.

[0008] Step 3: Define the ER interval based on the eye aspect ratio curve obtained in step 2. The interval represents the time between two adjacent peaks on the curve, that is, the time it takes for the learner to blink once. By measuring the time interval between each blink and the change in the time interval, we can obtain indicators for evaluating the frequency and regularity of blinking and build a blink change state model.

[0009] Step 4: Compare the blink rate index obtained in step 3 with the data of the previous and next frames to obtain the fluctuation of the blink rate and conduct further quantitative analysis of eye activity.

[0010] Step 5: Based on the 98 facial key points extracted in step 1, hundreds of face images with different head postures are trained to build a head posture recognition model.

[0011] Step 6: Input the image to be processed into the mathematical model fitted in step 3 to obtain the learner's Euler angles, which are pitch angle (pitch), yaw angle (yaw) and roll angle (roll), and process them accordingly.

[0012] Step 7, constructing a time series feature, i.e., the change of the head Euler angle in unit time, based on the Euler angle data obtained in step 4, so as to determine whether the learner has abnormal head behavior in a drowsy state.

[0013] Step 8, locate the mouth area according to the face model extracted in step 1, and use the position relationship of the corresponding key points of the mouth between frames to estimate the distance the mouth moves per unit time as one of the characteristics of the yawning state.

[0014] Step 9: In order to make the yawn state recognition more accurate, the mouth opening and closing ratio feature is added on the basis of step 8. The area of ​​the mouth when open and closed is obtained by calculating the number of pixels in the mouth area, and then calculating their ratio.

[0015] Step 10, by using methods such as variance analysis and gradient feature elimination, the previously acquired information such as eye blink rate features, head movement timing features, and yawn state features are processed to eliminate the collinearity between the attributes.

[0016] Step 11, after the data is processed in step 7, the constructed facial multi-feature data is input into the XGBoost model, and the final sleepy state label is obtained through parameter adjustment and model fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] Figure 1 This is a flow chart of the algorithm for detecting sleepiness based on facial multi-feature learners of the present invention;

[0019] Figure 2 This is a diagram of the specific implementation process of the sleepiness state detection algorithm based on multiple facial features;

[0020] Figure 3 Schematic diagram of facial key point recognition at different angles;

[0021] Figure 4 This is the comparison between the head posture algorithm in this paper and other algorithms; DETAILED DESCRIPTION

[0022] Step 1: Use Retinaface to extract the facial frame in the video frame image, and use PFLD to recognize facial key points to obtain 98 facial key points, as shown in the following formula 1.

[0023] F i = {P1,P2,...,P N};P i =(P x ,P y ),N=98,i=1,2,...,N (1)

[0024] Step 2: locate the eye area based on the key points of the face and calculate the eye aspect ratio. The eye aspect ratio is calculated as shown in formula (2). Draw a curve of the degree of eye opening and closing, and obtain an indicator for evaluating the blinking frequency and regularity based on the time interval between adjacent peaks.

[0025]

[0026] Step 3: In formulas (3) and (4), the blink rate ERt and the fluctuation of the blink rate ERt_avg can be obtained.

[0027]

[0028]

[0029] Step 4: Use the 98 facial key points obtained in step 1 to fit hundreds of face images with different head postures.

[0030] Step 5: Based on the head posture recognition model obtained in step 5, the learner's pitch angle, yaw angle, and roll angle are extracted, as shown in formulas (5), (6), and (7).

[0031] pitch=pitch k ×1.497+18.97 (5)

[0032] yaw=yaw k ×71.58+0.7037 (6)

[0033]

[0034] Step 6: Based on the obtained pitch angle, construct a pitch angle change angle feature with time series characteristics. Based on this feature, determine whether the learner's head has an abnormal movement state in the sleepy state. The feature construction is shown in formulas (8), (9), and (10).

[0035]

[0036]

[0037]

[0038] Step 7, in formula (9), ε is defined as the range of the learner's yaw angle between the normal state and the sleep state. If at time point t, the learner's yaw angle pitch value is between [pitch t -ε<pitch i <pitch t +ε], we regard it as the learner is in a normal state. If the value exceeds this interval, we believe that the learner shows a behavior of the head deviating from the normal position at time point t. After multiple experimental verifications, when ε is set to 40, the experimental results are the best.

[0039] Step 8: Accurately locate the mouth area through facial key point positioning technology. Then, use the key point positions of the previous and next frames to calculate the movement distance of the mouth. In addition, add the mouth area opening and closing ratio feature, calculate the number of pixels in the mouth area to obtain the area of ​​the mouth when open and closed, and calculate their ratio. The feature construction is shown in formulas (11), (12), and (13).

[0040]

[0041]

[0042]

[0043] Step 9, combine the two indicators to describe the movement of the mouth over time. When yawning, the range of movement of the mouth will expand and the speed will be slower. By analyzing the changes in the distance the mouth moves and the opening and closing of the mouth during this time period, it can be inferred whether the learner yawns and determine the sleepy state.

[0044] Step 10, by using methods such as variance analysis and gradient feature elimination, the previously acquired information such as eye blink rate features, head movement features and mouth movement features is processed to eliminate the collinearity between the attributes.

[0045] Step 11, after the data is processed in step 7, the constructed facial multi-feature data is input into the XGBoost model, and the final sleepy state label is obtained through parameter adjustment and model fusion.

Claims

1. A learner sleepiness state recognition algorithm based on facial multi-features, characterized in that: It includes two parts: facial multi-feature extraction and model training.

2. The feature extraction method according to claim 1, characterized in that: It includes feature extraction of eye blinking change state, head movement timing feature extraction, and yawning state feature extraction.

3. The eye feature extraction method according to claim 2, wherein: The eye area is located according to the key points of the face, and the eye aspect ratio is calculated to draw a curve of the change of the eye opening and closing degree over time. The time interval between adjacent peaks is used to obtain indicators for evaluating the frequency and regularity of blinking, and a blink change state model is obtained. F i ={P1,P2,...,P N };P i =(P x ,P y ),N=98,i=1,2,...,N (1) 4. The head posture feature extraction as claimed in claim 2, characterized in that: The 98 extracted facial key points are used to fit hundreds of facial images with different head postures. The obtained head posture recognition model extracts the learner's pitch angle, yaw angle, and roll angle. Based on the obtained pitch angle, a pitch angle change angle feature with time series characteristics is constructed, and this feature is used to determine whether the learner's head has an abnormal movement state when in a sleepy state. pitch=pitch k ×1.497+18.97 (5) yaw=yaw k ×71.58+0.7037 (6) Among them, pitch t is the index of the yaw angle in the standard state frame, N0=30, and the average value of the first 30 frames is taken as the standard for measuring the head deviation state. The best effect is obtained when ε is 40. A certain length of time is 150 frames, that is, N=150, as the unit time period for measuring the frequency of sleepy nodding.

5. The yawning state feature extraction method according to claim 2, characterized in that: The mouth area is accurately located through facial key point positioning technology, and the key point positions of the previous and next frames are used to calculate the distance the mouth moves. Then, the number of pixels in the mouth area is calculated to obtain the area of ​​the mouth when it is open and closed, and the ratio is calculated to obtain the mouth opening and closing degree index. These two indicators are used to describe the movement of the mouth over time. When drowsy, the range of movement of the mouth will expand and move slowly. By analyzing the changes in the distance and speed of the mouth movement, it can be inferred whether the learner is yawning and thus determine the drowsy state.

6. The model training method according to claim 2, characterized in that: Through methods such as variance analysis and gradient feature elimination, the previously acquired information such as eye blink change characteristics, head movement timing characteristics and yawning state characteristics are processed to eliminate the collinearity between attributes. The constructed facial multi-feature data is input into the XGBoost model, and the final sleepy state label is obtained through parameter adjustment and model fusion.

Images