Classroom student attention assessment method and system based on multi-source feature fusion

Abstract

The invention discloses a classroom student attention assessment method and system based on multi-source feature fusion. According to the invention, face recognition is carried out through the face detection unit of the system, occurrence frequencies of different emotional states of students are obtained through a trained student expression recognition neural network model containing four expression discrimination blocks in the expression recognition unit, and fatigue times of the students are obtained through the fatigue detection unit. The number of times of abnormal head postures of the students is obtained through the head posture estimation unit, finally attention evaluation parameters of the students are obtained through an extreme learning machine model of the attention evaluation unit, and then the attention states of the students in the class are evaluated and displayed. According to the method, a more efficient dynamic student expression recognition method can be provided, real-time and accurate classroom feedback is provided for teachers, optimization and implementation of personalized teaching strategies are supported, longitudinal extension of teaching of the teachers and transverse extension of learning modes of the students are facilitated, and new talents are expanded to cultivate a new normal form.

Description

Technical Field

[0001] The present invention relates to an attention assessment method, and to the field of artificial intelligence and educational information technology, and in particular to a classroom student attention assessment method and system based on multi-source feature fusion. Background Art

[0002] There are still several limitations in the current methods of evaluating students' classroom attention in teaching scenarios. First, in most teaching scenarios, since the teaching mode is one-to-many, the existence of small-scale, low-pixel, and multi-occluded student targets greatly increases the difficulty of monitoring students' attention and behavior, which in turn affects the generalization ability and practical application effect of existing models. Secondly, the singleness and low dimensionality of the evaluation method limit the comprehensive consideration of many factors affecting students' attention. Furthermore, the lack of an effective joint model between different evaluation criteria leads to excessive dependence of the evaluation results on a certain criterion, thus failing to effectively reduce the evaluation error. Finally, the lack of accuracy in the student expression recognition technology in the existing evaluation model also leads to deviations in the evaluation of students' attention. Therefore, it is urgent to build a comprehensive evaluation paradigm system with multi-dimensional cross-cutting and collaborative optimization to improve the accurate judgment of students' attention status, and then effectively support the deep embedding of personalized teaching strategies. Summary of the invention

[0003] In order to solve the problems existing in the background technology, the present invention provides a method and system for evaluating classroom student attention based on multi-source feature fusion. The method of the present invention provides a more efficient dynamic student expression recognition DSER (Dynamic Student Expression Recognition) method, provides teachers with real-time and accurate classroom feedback, supports the optimization and implementation of personalized teaching strategies, is conducive to the vertical extension of teachers' teaching and the horizontal expansion of students' learning methods, further opens up a new track for AI education, and expands the new paradigm of cultivating new quality talents.

[0004] The technical solution adopted by the present invention is: 1. A method for evaluating students' attention in classroom based on multi-source feature fusion: Step S1: Obtain several videos of students listening to lectures in class through the camera and extract the initial video frames, perform face detection preprocessing on each initial video frame, and then identify different emotional states of the face by locating facial key points and add expression labels, so as to obtain several facial expression images and construct a facial expression image dataset.

[0005] Step S2: Construct a student expression recognition neural network model SFaceNet containing four expression discrimination blocks, and train the student expression recognition neural network model SFaceNet using a facial expression image dataset.

[0006] Step S3: Obtain the classroom video of the student to be identified through the camera and extract the video frame to be identified, then perform face recognition, and then use the trained student expression recognition neural network model SFaceNet to identify the frequency of occurrence of different emotional states of each student, and identify the fatigue level and number of abnormal head postures of each student through facial key point positioning.

[0007] Step S4: For each student, the frequency of occurrence of different emotional states, fatigue level and number of abnormal head postures of the student are input into the pre-trained extreme learning machine model ELM (Extreme Learning Machine). After processing, the student's attention evaluation parameters are output. The attention status of the students in the class is evaluated and displayed according to the attention evaluation parameters of each student.

[0008] In the step S1, each initial video frame is subjected to face recognition through the face detection model RetinaFace to obtain an image of a detection frame with facial features of each student, and then data enhancement processing is performed, including random cropping, adjustment to a uniform size, random horizontal flipping and image normalization, so as to obtain a preprocessed video frame.

[0009] In the step S1, each pre-processed video frame is identified by the 468 facial key point positioning method based on machine learning Mediapipe and the Russell emotion model to identify four emotional states of the face including rejection expression, listening expression, confusion expression and excitement expression as typical classroom emotional expressions. For the face of the student in each detection frame of the pre-processed video frame, the four emotional states of the face are as follows: a) Rejection expression: To judge the degree of inclination of the mouth, when the angle formed by the line connecting the 62nd key point, the 292nd key point and the 17th key point in the face and the horizontal line is negative, it means that the mouth is tilted downward relative to the horizontal line, and the corners of the mouth on both sides are moved downward. Then, it is judged that the current emotional state of the face is a rejection expression, and there are no obvious features in the eyes.

[0010] b) Listen to the expression: To judge the degree of mouth tilt, when the line connecting the 62nd key point, the 292nd key point and the 17th key point in the face is approximately parallel to the horizontal line, the degree of mouth tilt is not obvious.

[0011] To judge the degree of eye openness, the height of the left eye in the eye feature is obtained by subtracting the ordinates of the 470th key point and the 472nd key point in the face, the eye width of the left eye in the eye feature is obtained by subtracting the abscissas of the 33rd key point and the 133rd key point in the face, the height of the right eye in the eye feature is obtained by subtracting the ordinates of the 475th key point and the 477th key point in the face, the eye width of the right eye in the eye feature is obtained by subtracting the abscissas of the 362nd key point and the 263rd key point in the face, if the ratio of the eye height to the eye width of the left eye and the ratio of the eye height to the eye width of the right eye are both greater than the ratio threshold, the degree of eye openness of the face is naturally open.

[0012] To judge the degree of eyebrow curvature, if the angles between the 70th key point and the 65th key point in the face and the 105th key point are greater than the angle threshold, and the angles between the 300th key point and the 295th key point in the face and the 334th key point are greater than the angle threshold, the eyebrow curvature of the face is naturally stretched.

[0013] When the inclination of the mouth of the human face is not obvious, the opening degree of the eyes of the human face is naturally open, and the curvature degree of the eyebrows of the human face is naturally stretched, it is determined that the current emotional state of the human face is a listening expression; c) Confused expression: To judge the degree of eyebrow curvature, if the angles between the 70th key point and the 65th key point and the 105th key point in the face are less than the angle threshold, and the angles between the 300th key point and the 295th key point and the 334th key point in the face are less than the angle threshold, the degree of eyebrow curvature of the face is frowning; When the inclination of the mouth of the human face is that the corners of the mouth on both sides move downward, and the curvature of the eyebrows of the human face is tightly wrinkled, it is determined that the current emotional state of the human face is a confused expression.

[0014] d) Excited expression: To judge the degree of inclination of the mouth, when the angle formed by the line connecting the 62nd key point, the 292nd key point and the 17th key point in the face and the horizontal line is a positive number, it indicates that the degree of inclination of the mouth of the face is that the corners of the mouth on both sides are upturned relative to the horizontal line.

[0015] When the inclination of the mouth of the face is that the corners of the mouth are upturned, the eyes are naturally open, the eyebrows are naturally stretched, and the Euclidean distance between the 12th key point and the 14th key point of the mouth of the face is greater than the preset distance threshold, which is set to 2 pixels in specific implementation, then it is determined that the current emotional state of the face is a listening expression.

[0016] An expression label of an emotional state is added to the face of each student in the detection frame.

[0017] In the step S2, the student expression recognition neural network model SFaceNet includes an input layer, a feature extraction and fusion layer, and an output layer connected in sequence, the input layer includes a first convolutional layer, a first batch normalization layer Batch Norm, a first activation function layer ReLu, and a maximum pooling layer Maxpooling connected in sequence, the feature extraction and fusion layer includes three expression discrimination blocks EmotionBlock V1, four expression discrimination blocks EmotionBlock V2, six expression discrimination blocks EmotionBlock V3, three expression discrimination blocks EmotionBlock V4, and an attention mechanism module CBAM connected in sequence, and also includes a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, the output of the maximum pooling layer Maxpooling is then input into the first expression discrimination block EmotionBlock V1, the maximum pooling layer Maxpooling and the output of the first expression discrimination block EmotionBlock V1 are added to obtain a first addition result, the first addition result is input into the second expression discrimination block EmotionBlock V1, the first addition result and the second expression discrimination block EmotionBlock V1 to obtain a second addition result, and the second addition result is input into the third expression discrimination block one EmotionBlock V1. The second addition result and the output of the third expression discrimination block one EmotionBlock V1 are added to obtain a third addition result, and the third addition result is input into the first expression discrimination block two EmotionBlock V2. The third addition result is processed by the second convolutional layer and added to the output of the first expression discrimination block two EmotionBlock V2 to obtain a fourth addition result. The fourth addition result is input into the second expression discrimination block two EmotionBlock V2. The fourth addition result and the output of the second expression discrimination block two EmotionBlock V2 are added to obtain a fifth addition result. The fifth addition result is input into the third expression discrimination block two EmotionBlock V2. The fifth addition result and the output of the third expression discrimination block two EmotionBlock V2 are added to obtain a sixth addition result. The sixth addition result is input into the fourth expression discrimination block two EmotionBlock V2. The sixth addition result and the fourth expression discrimination block two EmotionBlock V2 are added to obtain a sixth addition result. The outputs of V2 are added to obtain a seventh addition result, which is input into the first expression discrimination block three EmotionBlock V3. The seventh addition result is processed by the third convolutional layer and added to the output of the first expression discrimination block three EmotionBlock V3 to obtain an eighth addition result, which is input into the second expression discrimination block three EmotionBlock V3.The eighth addition result is added to the output of the second expression discrimination block three EmotionBlock V3 to obtain a ninth addition result, and the ninth addition result is input into the third expression discrimination block three EmotionBlock V3. The ninth addition result is added to the output of the third expression discrimination block three EmotionBlock V3 to obtain a tenth addition result, and the tenth addition result is input into the fourth expression discrimination block three EmotionBlock V3. The tenth addition result is added to the output of the fourth expression discrimination block three EmotionBlock V3 to obtain an eleventh addition result, and the eleventh addition result is input into the fifth expression discrimination block three EmotionBlock V3. The eleventh addition result is added to the output of the fifth expression discrimination block three EmotionBlock V3 to obtain a twelfth addition result, and the twelfth addition result is input into the sixth expression discrimination block three EmotionBlock V3. The twelfth addition result and the sixth expression discrimination block three EmotionBlock V3 are added together. The output of V3 is added to obtain the thirteenth addition result, the thirteenth addition result is processed by the fourth convolutional layer and added to the output of the first expression discrimination block four EmotionBlock V4 to obtain the fourteenth addition result, the fourteenth addition result is input into the second expression discrimination block four EmotionBlock V4, the fourteenth addition result and the output of the second expression discrimination block four EmotionBlock V4 are added to obtain the fifteenth addition result, the fifteenth addition result is input into the third expression discrimination block four EmotionBlock V4, the fifteenth addition result and the output of the third expression discrimination block four EmotionBlock V4 are added and then input into the attention mechanism module CBAM; the output layer includes the average pooling layer Avgpooling and the fully connected layer FC (Fully Connected) connected in sequence. ,

[0018] The expression discrimination block EmotionBlock V1, the expression discrimination block EmotionBlock V2, the expression discrimination block EmotionBlock V3 and the expression discrimination block EmotionBlock V4 include a second batch normalization layer Batch Norm, a second activation function layer ReLu, a fifth convolutional layer, a third batch normalization layer Batch Norm, a third activation function layer ReLu, a sixth convolutional layer, a contextual transformer module CoT (Contextual Transformer), a fourth batch normalization layer Batch Norm, a fourth activation function layer ReLu and a seventh convolutional layer connected in sequence.

[0019] In the step S3, for each video frame to be identified, each video frame to be identified is subjected to face recognition through the face detection model RetinaFace to obtain an image of the detection frame with the facial features of each student to be identified, and then the image of the detection frame with the facial features of each student to be identified is input into the trained student expression recognition neural network model SFaceNet for processing to obtain the emotional state of each student in the current video frame, and finally the frequency of occurrence of different emotional states of each student during the shooting time period of the classroom lecture video of the student to be identified is identified according to the images of the detection frame with the facial features of each student to be identified.

[0020] In the step S3, for each image with a detection frame of each student's facial features to be identified, it is first processed by a 468 facial key point positioning method based on machine learning Mediapipe to obtain 468 key point information of each student's face in the graphic. For each student, 6 key points are taken from each of the three parts of the student's face, namely the left eye, right eye and mouth, for a total of 18 key points. The eye aspect ratio EAR (Eye Aspect Ratio) is obtained based on the 6 key points of the left eye and the right eye of the student's face, and then the student's eye closing state is judged; the mouth opening degree MAR (Mouth Aspect Ratio) is obtained based on the 6 key points of the student's mouth, and then the student's yawning state is judged. Finally, according to each image with a detection frame of each student's facial features to be identified, the number of times each student closes his eyes and yawns during the shooting period of the classroom lecture video of the student to be identified is identified as the number of times each student is tired.

[0021] In the step S3, for each video frame to be identified, for each image to be identified with a detection frame of each student's facial features, firstly, the 468 facial key point positioning method based on machine learning Mediapipe is used for processing to obtain the two-dimensional coordinates of the 468 key points of each student's face in the image, and then the two-dimensional coordinates are normalized to avoid errors caused by size differences of different input images, so that subsequent calculations are more stable and universal, and then the normalized two-dimensional coordinates of each key point are mapped to the three-dimensional world coordinate system, and for each student, the pose estimation algorithm PnP (Perspective -n-Points) matches the three-dimensional coordinates and the two-dimensional coordinates of each key point of the student's face in the three-dimensional world coordinate system, so as to obtain the rotation and displacement of the camera, and finally obtain the three-degree-of-freedom information of the student's head, including the rotation Yaw around the vertical axis (z-axis), the rotation Pitch around the horizontal axis (y-axis), and the rotation Roll around the longitudinal axis (x-axis). The student's head-lowering state is judged according to the three-degree-of-freedom information of the student's head, and finally the number of times each student lowers his head during the shooting time period of the classroom listening video of the student to be identified is identified according to each image of the detection frame with the facial features of each student to be identified as the number of abnormal head postures of each student.

[0022] In step S4, the obtained seven-dimensional eigenvalue is used as a 1×7 input vector ( x 1: Number of times you bow your head; x 2: Number of yawns; x 3: Number of eye closures; x 4: Listen to the number of times the expression appears; x 5: The number of times the confused expression appears; x 6: The number of times the resistance expression appears; x 7: The number of times the excited expression appears) is input into the pre-trained extreme learning machine model ELM model to evaluate the students' attention, and finally obtain an attention evaluation parameter between 0 and 1; then the average value of the attention evaluation parameters of each student at each moment is used as the average attention evaluation parameter of the class at the current moment, and the time dimension is introduced to perform time series analysis. The preset time interval is 1min, and the video is recorded every 15s to obtain the average attention evaluation parameter of the class at a moment. The average value of the average attention evaluation parameters of the class at several moments in each preset time interval is used as the attention evaluation parameter of the class unit cycle, so as to judge the attention status of the students in the class, as follows: When 0≤Focus t When it is less than 0.4, the students in the class are in a very poor state of attention, that is, the teaching effect is inefficient. t It is the attention evaluation parameter of the class unit cycle at the current preset time interval; When 0.4≤Focus t When it is less than 0.6, the students’ attention in the class is poor, which means the teaching effect is not good.

[0023] When 0.6≤Focus t When it is <0.8, the students’ attention level in the class is average, which means the teaching effect is acceptable.

[0024] When 0.8≤Focus t When it is less than 1.0, the students in the class are paying very good attention, which means the class is effective.

[0025] 2. A classroom student attention assessment system based on multi-source feature fusion: The face detection unit uses the face detection model RetinaFace to perform face recognition on the video frames of the student's classroom lecture video obtained through the camera, and then outputs an image of the detection frame with each student's facial features as the face detection result. It can realize accurate detection of the student's face area and ensure that small-scale, low-pixel, and multi-occluded student faces can be effectively identified in complex classroom scenes.

[0026] The expression recognition unit constructs and trains the student expression recognition neural network model SFaceNet, which contains four expression discrimination blocks. The trained student expression recognition neural network model SFaceNet is used to identify the different emotional states of each student as the expression recognition result, and then the occurrence frequency of each student's different emotional states is obtained.

[0027] The fatigue detection unit locates the key points of the face and identifies the number of times each student closes his eyes and yawns as the number of fatigue levels.

[0028] The head posture estimation unit identifies the number of times each student lowers his head as the number of abnormal head postures by locating facial key points.

[0029] The attention assessment unit inputs the frequency of occurrence of students' different emotional states, fatigue levels, and the number of abnormal head postures into the pre-trained extreme learning machine model ELM. After processing, it outputs the students' attention assessment parameters. Based on the attention assessment parameters of each student, the attention status of the students in the class is evaluated and displayed on the monitor.

[0030] The beneficial effects of the present invention are: 1. The present invention improves the robustness and generalization ability in complex classroom environments: In teaching scenarios, the difficulty of obtaining facial information of students due to problems such as position, occlusion and low resolution has increased significantly. The present invention is based on the multi-task optimization scheme of the face detection RetinaFace neural network, which significantly enhances the recognition accuracy and stability of the system under small scale, low resolution and complex occlusion conditions, ensuring the accurate capture of students' facial features.

[0031] 2. The present invention integrates multi-source features to optimize attention analysis: by deeply extracting multi-source features such as students' expressions, fatigue and head posture, and comprehensively utilizing facial geometric key points and dynamic expression trajectories, the system's attention analysis capability is improved in a multi-dimensional cross-fusion manner, thereby overcoming the analysis limitations brought by traditional single-dimensional features and ensuring the fine-grained and three-dimensional nature of attention assessment.

[0032] 3. The present invention realizes an efficient collaborative evaluation paradigm: multi-source feature data is input into the extreme learning machine model ELM, and a deep collaborative optimization joint evaluation framework is constructed. Through the multi-dimensional feature space mapping and weight optimization of the model, the deviation caused by a single evaluation standard is significantly reduced, thereby enhancing the accuracy and consistency of the evaluation results, and providing an efficient paradigm for the objective quantitative evaluation of classroom performance.

[0033] 4. The present invention improves the accuracy and recognition rate of facial expression recognition: The SFaceNet neural network model for student facial expression recognition is built by self-made high-quality facial expression data sets and the introduction of the convolutional block attention mechanism CBAM module and the context transformer module CoT, which significantly improves the accuracy and recognition rate of student facial expression recognition. The efficient emotional state distinction and facial expression dynamics extraction enable the system to perform more detailed emotional analysis of students' classroom performance, improving the accuracy of the attention model under different emotional states.

[0034] 5. The present invention has refined fatigue and head posture analysis: based on Mediapipe key point detection and posture estimation PnP algorithm, the system realizes three-degree-of-freedom estimation of students' head posture by calculating the posture matrix, accurately monitors behavioral indicators such as the number of times students lower their heads, and accurately quantifies fatigue behaviors such as closing eyes and yawning, further improving the analysis accuracy of classroom performance.

[0035] 6. The present invention supports deep embedding of personalized teaching feedback: the system can provide teachers with detailed classroom feedback data through high-precision attention timing evaluation parameter generation, supporting the quantitative deployment of personalized teaching strategies. Intelligent evaluation driven by multi-source data enables teachers to grasp students' cognitive status in real time, providing objective support for precise teaching and differentiated teaching methods.

[0036] 7. The present invention meets the real-time response requirements in dynamic and complex environments: Due to the introduction of the efficient extreme learning machine model ELM, the system has been optimized in terms of data processing and feedback response speed, and can perform real-time monitoring and evaluation in a dynamic classroom environment, ensuring efficient feedback capabilities of attention states, thereby supporting real-time adjustment and optimization of the teaching process. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the method for the specific implementation of the present invention; Figure 2 This is a flow chart of the classification and recognition of expression data sets of the present invention; Figure 3 A network structure diagram of the student expression recognition network model SFaceNet of the present invention; Figure 4 A network structure diagram of the first expression recognition block EmotionBlock V1, the second expression recognition block EmotionBlock V2, the third expression recognition block EmotionBlock V3 and the fourth expression recognition block EmotionBlock V4 of the present invention; Figure 5 This is a fatigue detection flow chart of the present invention; Figure 6 A flow chart of student attention assessment of the present invention; Figure 7 It is a curve diagram of classroom student attention assessment parameters of the present invention. DETAILED DESCRIPTION

[0038] The following is a detailed description of a specific embodiment of the present invention, and is described with reference to the attached illustrations. It should be noted that the embodiments shown here are only a part of many possible implementations, not all. These descriptions should not be construed as limiting the scope of protection of the present invention. According to the spirit of the present invention, any other implementation that does not involve creative work and can be easily thought of by a person skilled in the art based on this embodiment should be deemed to be within the scope of protection of the present invention.

[0039] like Figure 1 As shown, the classroom student attention assessment method based on multi-source feature fusion of the present invention is specifically as follows: In the specific implementation of the present invention, the facial expression recognition dataset FER2013 (Facial Emotion Recognition 2013) is first obtained, and each facial expression image in the facial expression recognition dataset FER2013 is recalibrated, and the original label of each facial expression image is removed. Then, different emotional states of the face are identified and expression labels are added through facial key point positioning, that is, the facial expression recognition dataset FER2013 is used to identify four emotional states of the face including rejection, listening, confusion and excitement as typical classroom emotional expressions through the 468 facial key point positioning method based on machine learning Mediapipe and the Russell emotion model, and then artificial recalibration is performed to finally obtain the recalibrated facial expression recognition dataset FER2013. As shown in Table 1, after the 468-point coordinate information of a face is obtained by using the 468-point facial key point positioning method based on machine learning Mediapipe, the relevant parameters are calculated according to the theoretical characteristics given in Table 1, and the expression labels of the facial expression recognition dataset FER2013 are redefined for subsequent neural network training, such as Figure 2 As shown in the figure, for the student’s face in each detection frame of the preprocessed video frame, the four emotional states of the face are as follows: a) Rejection expression: To judge the degree of inclination of the mouth, when the angle formed by the line connecting the 62nd key point, the 292nd key point and the 17th key point in the face and the horizontal line is negative, it means that the mouth is tilted downward relative to the horizontal line, and the corners of the mouth on both sides are moved downward. Then, it is judged that the current emotional state of the face is a rejection expression, and there are no obvious features in the eyes.

[0040] b) Listen to the expression: To judge the degree of mouth tilt, when the line connecting the 62nd key point, the 292nd key point and the 17th key point in the face is approximately parallel to the horizontal line, the degree of mouth tilt is not obvious.

[0041] To judge the degree of eye openness, the height of the left eye in the eye feature is obtained by subtracting the ordinates of the 470th key point and the 472nd key point in the face, the eye width of the left eye in the eye feature is obtained by subtracting the abscissas of the 33rd key point and the 133rd key point in the face, the height of the right eye in the eye feature is obtained by subtracting the ordinates of the 475th key point and the 477th key point in the face, the eye width of the right eye in the eye feature is obtained by subtracting the abscissas of the 362nd key point and the 263rd key point in the face, if the ratio of the eye height to the eye width of the left eye and the ratio of the eye height to the eye width of the right eye are both greater than the ratio threshold, the degree of eye openness of the face is naturally open, and in specific implementation, the ratio threshold is set to 0.25.

[0042] To judge the degree of eyebrow curvature, the angles between the 70th key point and the 65th key point in the face and the 105th key point are greater than the angle threshold, and the angles between the 300th key point and the 295th key point in the face and the 334th key point are greater than the angle threshold. The degree of eyebrow curvature of the face is naturally stretched. In specific implementation, the angle threshold is 120°.

[0043] When the inclination of the mouth of the human face is not obvious, the opening degree of the eyes of the human face is naturally open, and the curvature degree of the eyebrows of the human face is naturally stretched, it is determined that the current emotional state of the human face is a listening expression; c) Confused expression: To judge the degree of eyebrow curvature, if the angles between the 70th key point and the 65th key point in the face and the 105th key point are less than the angle threshold, and the angles between the 300th key point and the 295th key point in the face and the 334th key point are less than the angle threshold, the degree of eyebrow curvature of the face is tightly wrinkled.

[0044] When the inclination of the mouth of the human face is that the corners of the mouth on both sides move downward, and the curvature of the eyebrows of the human face is tightly wrinkled, it is determined that the current emotional state of the human face is a confused expression.

[0045] d) Excited expression: To judge the degree of inclination of the mouth, when the angle formed by the line connecting the 62nd key point, the 292nd key point and the 17th key point in the face and the horizontal line is a positive number, it indicates that the degree of inclination of the mouth of the face is that the corners of the mouth on both sides are upturned relative to the horizontal line.

[0046] When the inclination of the mouth of the face is that the corners of the mouth are upturned, the eyes are naturally open, the eyebrows are naturally stretched, and the Euclidean distance between the 12th key point and the 14th key point of the mouth of the face is greater than the preset distance threshold, which is set to 2 pixels in specific implementation, then it is determined that the current emotional state of the face is a listening expression.

[0047] Table 1 Expression parameter definition

[0048] Then, the lecture videos of 15 students were obtained through the camera and the initial video frames were extracted. Each of the 15 students watched the online class for 4 minutes on the camera and made four expressions of listening, confusion, rejection and excitement every minute. In this way, several initial video frames with one of the four expressions were obtained, and each initial video frame was constructed into a self-made expression dataset. The self-made expression dataset is preprocessed for face detection, and each initial video frame is passed through the face detection model RetinaFace for face recognition. After obtaining the class video stream, the face detection model RetinaFace processes the video image frame by frame, and finally returns the detected face and its key point information in the video, which are stored in the array b

[14] . Array b

[14] is an array (vector) of length 15, corresponding to the information of a detected face, as follows: b[0] represents the horizontal coordinate of the upper left corner of the face frame in the original image, b[1] represents the vertical coordinate of the upper left corner of the face frame in the original image, b[2] represents the horizontal coordinate of the lower right corner of the face frame in the original image, b[3] represents the vertical coordinate of the lower right corner of the face frame in the original image, and b[4] represents the confidence score of whether the predicted frame is a face. , the higher the value, the greater the probability of a face. b[5] represents the horizontal coordinate of the left eye in the face key point, b[6] represents the vertical coordinate of the left eye in the face key point, b[7] represents the horizontal coordinate of the right eye in the face key point, b[8] represents the vertical coordinate of the right eye in the face key point, b[9] represents the horizontal coordinate of the nose in the face key point, b

[10] represents the vertical coordinate of the nose in the face key point, b

[11] represents the horizontal coordinate of the left corner of the mouth in the face key point, b

[12] represents the vertical coordinate of the left corner of the mouth in the face key point, b

[13] represents the horizontal coordinate of the right corner of the mouth in the face key point, and b

[14] represents the vertical coordinate of the right corner of the mouth in the face key point. Finally, the facial features of the 15 students in the detection frame are added with expression labels according to the four expressions of the present invention, and finally a facial expression image dataset is constructed.

[0049] The face detection model RetinaFace uses the backbone network ResNet50 to detect students' faces in each frame of teaching video to ensure accurate recognition of students' facial features. After testing, the face detection model RetinaFace, which uses the backbone network ResNet50, has shown strong adaptability and efficient face detection and recognition capabilities for various challenges and complex situations in actual classrooms. In actual classrooms, there are many situations, such as frequent student activities, uneven lighting, changing camera angles, small-scale recognition targets, low pixels, and multiple occlusions. The face detection model RetinaFace can still quickly and accurately recognize the facial features of each student.

[0050] Then, the recalibrated facial expression recognition dataset FER2013 and the facial expression image dataset were mixed in a ratio of 7:3 to construct the initial dataset for the training of the expression recognition neural network. In the initial dataset, there are 8774 excited data, 5740 confused data, 6628 listening data, and 5509 repulsive data. The pixel size of the four data images is 48×48, and the number of channels is 3.

[0051] Then, we construct a student expression recognition neural network model SFaceNet which contains four expression recognition blocks, such as Figure 3 As shown in the figure, the student expression recognition neural network model SFaceNet includes an input layer, a feature extraction and fusion layer, and an output layer connected in sequence. The input layer includes the first convolutional layer, the first batch normalization layer Batch Norm, the first activation function layer ReLu and the maximum pooling layer Maxpooling connected in sequence. It is responsible for receiving the 3-channel face image data of size 224×224 input by the facial expression image dataset, and changes the image size to 64×112×112 through the first convolutional layer with a size of 7×7, a stride of 2, and a padding of 3 to reduce the image size and extract preliminary features. Finally, the image size is changed to 64×56×56 through the maximum pooling layer Maxpooling with a size of 3×3, a stride of 2, and a padding of 1.

[0052] The feature extraction and fusion layer includes three expression discrimination blocks EmotionBlock V1, four expression discrimination blocks EmotionBlock V2, six expression discrimination blocks EmotionBlock V3, three expression discrimination blocks EmotionBlock V4 and an attention mechanism module CBAM connected in sequence, and also includes a second convolutional layer, a third convolutional layer and a fourth convolutional layer. The output of the maximum pooling layer Maxpooling is then input into the first expression discrimination block EmotionBlock V1, and the output of the maximum pooling layer Maxpooling and the first expression discrimination block EmotionBlock V1 are added to obtain a first addition result, and the first addition result is input into the second expression discrimination block EmotionBlock V1, and the first addition result and the output of the second expression discrimination block EmotionBlock V1 are added to obtain a second addition result, and the second addition result is input into the third expression discrimination block EmotionBlock V1, and the second addition result and the third expression discrimination block EmotionBlock V1 are added to obtain a second addition result. V1, a third addition result is obtained by adding the outputs of the first expression discrimination block EmotionBlock V2. The third addition result is processed by the second convolutional layer and added to the output of the first expression discrimination block EmotionBlock V2 to obtain a fourth addition result. The fourth addition result is input into the second expression discrimination block EmotionBlock V2. The fourth addition result and the output of the second expression discrimination block EmotionBlock V2 are added to obtain a fifth addition result. The fifth addition result is input into the third expression discrimination block EmotionBlock V2. The fifth addition result and the output of the third expression discrimination block EmotionBlock V2 are added to obtain a sixth addition result. The sixth addition result is input into the fourth expression discrimination block EmotionBlock V2. The sixth addition result and the output of the fourth expression discrimination block EmotionBlock V2 are added to obtain a seventh addition result. The seventh addition result is input into the first expression discrimination block EmotionBlock V3. V3, the seventh addition result is processed by the third convolutional layer and added to the output of the first expression discrimination block three EmotionBlock V3 to obtain an eighth addition result, the eighth addition result is input into the second expression discrimination block three EmotionBlock V3, the eighth addition result and the output of the second expression discrimination block three EmotionBlock V3 are added to obtain a ninth addition result, the ninth addition result is input into the third expression discrimination block three EmotionBlock V3, the ninth addition result and the output of the third expression discrimination block three EmotionBlock V3 are added to obtain a tenth addition result,The tenth addition result is input into the fourth expression discrimination block three EmotionBlock V3, the tenth addition result is added to the output of the fourth expression discrimination block three EmotionBlock V3 to obtain an eleventh addition result, the eleventh addition result is input into the fifth expression discrimination block three EmotionBlock V3, the eleventh addition result is added to the output of the fifth expression discrimination block three EmotionBlock V3 to obtain a twelfth addition result, the twelfth addition result is input into the sixth expression discrimination block three EmotionBlock V3, the twelfth addition result is added to the output of the sixth expression discrimination block three EmotionBlock V3 to obtain a thirteenth addition result, the thirteenth addition result is processed by the fourth convolutional layer and added to the output of the first expression discrimination block four EmotionBlock V4 to obtain a fourteenth addition result, the fourteenth addition result is input into the second expression discrimination block four EmotionBlock V4, the fourteenth addition result is added to the second expression discrimination block four EmotionBlock V4. The output of V4 is added to obtain the fifteenth addition result, which is input into the third expression discrimination block four EmotionBlock V4. The fifteenth addition result and the output of the third expression discrimination block four EmotionBlock V4 are added and then input into the attention mechanism module CBAM; the output layer includes the average pooling layer Avgpooling and the fully connected layer FC connected in sequence, which are responsible for converting the above feature maps into the final four expression prediction results. These layers are responsible for extracting expression features and information from the input face image. The attention mechanism module CBAM is placed at the end to strengthen the neural network's attention to important feature channels and key areas of expression; and the input features are added to the output of the convolution layer through the introduction of residual structure between modules for feature fusion. ,

[0053] The expression recognition block EmotionBlock is responsible for extracting expression features and information from the input face image. The attention mechanism module CBAM is placed at the end to strengthen the neural network's attention to important feature channels and key areas of expression, while suppressing unimportant features. The input features are added to the output of the convolutional layer through the introduction of residual structures between modules for feature fusion. The output layer is responsible for converting the above feature maps into feature maps through the average pooling layer Avgpooling, compressing the spatial information into a feature vector, which is then sent to the fully connected layer. The fully connected layer outputs a vector with a dimension of the number of expression categories, thereby obtaining the final prediction results of the four expressions of the students. The student expression recognition neural network model SFaceNet has been optimized and improved for its specific scenario requirements to identify the four emotions of students. The student expression recognition neural network model SFaceNet can accurately classify the expressions of faces in the class that are recognized by the face detection model RetinaFace neural network.

[0054] like Figure 4 As shown, expression discrimination block 1 EmotionBlock V1, expression discrimination block 2 EmotionBlock V2, expression discrimination block 3 EmotionBlock V3 and expression discrimination block 4 EmotionBlock V4 include a second batch normalization layer Batch Norm, a second activation function layer ReLu, a fifth convolutional layer, a third batch normalization layer Batch Norm, a third activation function layer ReLu, a sixth convolutional layer, a contextual transformer module CoT (Contextual Transformer), a fourth batch normalization layer Batch Norm, a fourth activation function layer ReLu and a seventh convolutional layer connected in sequence.

[0055] The expression discrimination block Emotionblock refines the features and strengthens the information related to the expression. The size of the fifth convolutional layer of the expression discrimination block EmotionBlock V1 is 1×1, the step size is 1, and the padding value is 1. The image size after processing becomes 64×56×56; the size of the sixth convolutional layer is 3×3, the step size is 1, and the padding value is 1. The image size after processing is 64×56×56; the size of the seventh convolutional layer is 3×3, the step size is 1, and the padding value is 1. The image size after processing is still 64×56×56.

[0056] The output of the maximum pooling layer Maxpooling will pass through three identical expression discrimination blocks EmotionBlock V1, and the final output will be given to four expression discrimination blocks EmotionBlock V2 in turn. The size of the fifth convolution layer of the first expression discrimination block EmotionBlock V2 is 1×1, the step size is 1, and the padding value is 1. The image size after processing is still 64×56×56. The size of the sixth convolution layer is 3×3, the step size is 2, and the padding value is 1. The image size after processing becomes 128×28×28. The size of the seventh convolution layer is 3×3, the step size is 1, and the padding value is 1. The image size after processing is still 128×28×28. The size of the fifth convolution layer of the remaining second, third and fourth expression discrimination blocks EmotionBlock V2 is 1×1, the stride is 1, and the padding value is 1. The size of the sixth convolution layer is 3×3, the stride is 1, and the padding value is 1. The size of the seventh convolution layer is 3×3, the stride is 1, and the padding value is 1. The processed image size is still 128×28×28; then the results are input to the six expression discrimination blocks three EmotionBlock V3, among which the size of the fifth convolution layer of the first expression discrimination block three EmotionBlock V3 is 1×1, the stride is 1, and the padding value is 1. The processed image size is still 128×28×28, the size of the sixth convolution layer is 3×3, the stride is 2, and the padding value is 1. The processed image size becomes 256×14×14, the size of the seventh convolution layer is 3×3, the stride is 1, and the padding value is 1. The processed image size is still 256×14×14. For the remaining second, third, fourth, fifth and sixth discriminant blocks EmotionBlock V3, the size of the fifth convolutional layer is 1×1, the step size is 1, and the padding value is 1. The size of the sixth convolutional layer is 3×3, the step size is 1, and the padding value is 1. The size of the seventh convolutional layer is 3×3, the step size is 1, and the padding value is 1. The processed image size is still 256×14×14; then the results are input to the last four expression discriminant blocks four EmotionBlock V4, among which the size of the fifth convolutional layer of the first expression discriminant block four EmotionBlock V4 is 1×1, the step size is 1, and the padding value is 1. The processed image size is still 256×14×14, the size of the sixth convolutional layer is 3×3, the step size is 2, and the padding value is 1. The processed image size becomes 512×7×7, the size of the seventh convolutional layer is 3×3, the step size is 1, and the padding value is 1. The processed image size is still 512×7×7. The size of the fifth convolution layer of the second and third expression discrimination blocks EmotionBlock V4 is 1×1, the stride is 1, and the padding value is 1. The size of the sixth convolution layer is 3×3, the stride is 1, and the padding value is 1. The size of the seventh convolution layer is 3×3, the stride is 1, and the padding value is 1. The image size after processing is still 512×7×7.Finally, the 512×7×7 image output is passed through an attention mechanism module CBAM and an average pooling layer Avgpooling in sequence to be adjusted to 512×1×1, and finally passed through a fully connected layer FC to obtain a 4-dimensional output, which represents the predicted probabilities of the four expressions respectively.

[0057] Then, the initial dataset is enhanced. First, the image is randomly cropped using the function RandomResizedCrop(224) and adjusted to a uniform size of 224×224 to increase the diversity of the data so that the model can better generalize to different image sizes and scenes. Then, the image is randomly flipped horizontally using the function RandomHorizontalFlip() to increase the symmetry of the data and help the model learn expression features that remain unchanged after left-right flipping. At the same time, the image is standardized using the function Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) to accelerate the convergence of the model and improve the stability of the training. Finally, the initial dataset after data enhancement is obtained, and the student expression recognition neural network model SFaceNet is trained.

[0058] When training the student expression recognition neural network model SFaceNet, the loss function used was the CrossEntropyLoss cross entropy loss function, which minimized the difference between the category probability output by the model and the true label, thereby optimizing the performance of the model; the optimization method used the Adam optimizer, and the learning rate was set to 0.0001 to cope with sparse gradients or different update speeds of different parameters. The number of iterations Epoch was 300, and the batch size Batch_size was 16. The final SFaceNet model showed good convergence during the training process and achieved ideal classification accuracy.

[0059] Then, a 40-minute classroom video of 80 students to be identified is obtained through a camera and the video frames to be identified are extracted. Then, face recognition is performed, and then the frequency of occurrence of different emotional states of each student is identified through the trained student expression recognition neural network model SFaceNet. For each video frame to be identified, each video frame to be identified is subjected to face recognition by the face detection model RetinaFace to obtain an image of a detection frame with facial features of each student to be identified. Then, the image of the detection frame with facial features of each student to be identified is input into the trained student expression recognition neural network model SFaceNet for processing to obtain the emotional state of each student in the current video frame. Finally, the frequency of occurrence of different emotional states of each student in the shooting time period of the classroom video of the students to be identified is identified according to the images of the detection frames with facial features of each student to be identified. After testing, 2 minutes before class, 58% of the 80 students had expressions of "listening", 36% of the students had expressions of "excitement", and only 4% of the students had expressions of "confusion" and 2% of the students had expressions of "rejection".

[0060] At the same time, the number of times each student is tired is identified by locating facial key points. For each image to be identified with a detection frame of each student's facial features, it is first processed by the 468 facial key point positioning method based on machine learning Mediapipe to obtain 468 key point information of each student's face in the image. For each student, 6 key points are taken from each of the three parts of the student's face, namely the left eye, right eye and mouth, for a total of 18 key points. The eye aspect ratio EAR (Eye Aspect Ratio) is obtained based on the 6 key points of the left eye and right eye of the student's face, and then the student's eye closing state is judged; the mouth opening degree MAR (Mouth Aspect Ratio) is obtained based on the 6 key points of the student's mouth, and then the student's yawning state is judged. Finally, the number of times each student closes his eyes and yawns during the shooting period of the classroom listening video of the student to be identified is identified according to each image to be identified with a detection frame of each student's facial features as the number of times each student closes his eyes and yawns as the number of times each student is tired.

[0061] Thresholds are determined for the two parameters, EAR and MAR, to determine the status of the students' eyes and mouths, as follows: Eyes closed: 0≤EAR≤0.15 Normal open eyes: 0.15<EAR Normal mouth state: 0≤MAR<0.25 Mouth slightly open: 0.25≤MAR<0.35 Yawning: 0.35 ≤ MAR like Figure 5 As shown, the present invention uses two indicators for characterization: the number of times a student closes his eyes and yawns in a class. The face detected by the face detection model RetinaFace algorithm is combined with the face key point information obtained by the 468 face key point positioning method based on machine learning Mediapipe, and then the eye aspect ratio EAR value reflecting the eye state and the mouth opening degree MAR value reflecting the mouth state are calculated. The calculation of the eye aspect ratio EAR value is based on several key points of the eye, and the opening and closing state of the eyes is evaluated by measuring the ratio of the width and height of the eyes. When the eye aspect ratio EAR value is lower than the set threshold, it can be judged that the student has closed his eyes. Similarly, the calculation of the mouth opening degree MAR value is based on several key points of the mouth, and the opening and closing state of the mouth is evaluated by measuring the ratio of the width and height of the mouth. When the mouth opening degree MAR value exceeds the set threshold, it can be judged that the student is yawning.

[0062] In order to accurately evaluate the fatigue state of students, the present invention determines the thresholds of the eye aspect ratio EAR and the mouth opening degree MAR by measuring the different states of students in class to distinguish different eye and mouth states. As shown in Table 2 below, the threshold of the eye aspect ratio EAR value is set to distinguish between normal open eyes and closed eyes.

[0063] Table 2 Relationship between eye status and eye aspect ratio EAR value

[0064] Similarly, the threshold value of the mouth opening degree MAR value in the present invention is used to distinguish the "normal mouth state", "slightly open mouth" and "yawning" states, as shown in Table 3 below.

[0065] Table 3 Relationship between mouth state and mouth opening degree MAR value

[0066] In summary, the present invention can understand students' fatigue status in real time by monitoring students' closing their eyes and yawning in class and classifying the status based on the set threshold. Such an assessment can not only help teachers detect students' fatigue in time, but also provide a scientific basis for adjusting teaching strategies, thereby better promoting students' learning effects and physical and mental health. The fatigue detection effect is that 2 minutes before class, only 0.5% of the 80 students "yawned" and "closed their eyes", and the rest of the students stayed focused on the class.

[0067] At the same time, the number of abnormal head postures of each student is identified through facial key point positioning. For each video frame to be identified, for each image to be identified with a detection frame of each student's facial features, it is first processed through the 468 facial key point positioning method based on machine learning Mediapipe to obtain the two-dimensional coordinates of the 468 key points of the face of each student in the image, and then the two-dimensional coordinates are normalized to avoid errors caused by size differences of different input images, making subsequent calculations more stable and universal. Then, the normalized two-dimensional coordinates of each key point are mapped to the three-dimensional world coordinate system. For each student, the position The pose estimation algorithm PnP matches the three-dimensional coordinates and two-dimensional coordinates of each key point of the student's face in the three-dimensional world coordinate system, thereby obtaining the rotation and displacement of the camera, and finally obtaining the three-degree-of-freedom information of the student's head, including the rotation Yaw around the vertical axis (z-axis), the rotation Pitch around the horizontal axis (y-axis), and the rotation Roll around the longitudinal axis (x-axis), wherein Pitch determines raising and lowering the head, Pitch>0 when raising the head, and Pitch<0 when lowering the head; Yaw determines left and right rotation, Yaw>0 when turning the head right, and Yaw<0 when turning the head left; Roll determines left and right tilting of the head, roll>0 when tilting the head right, and roll<0 when tilting the head left. Through the combination of these three Euler angles, the present invention can describe any posture of the head in three-dimensional space, and then provide accurate human posture information for teaching interaction, virtual reality and other applications, which is used for the establishment of subsequent student attention evaluation systems. The students' head-lowering status is judged based on the three-degree-of-freedom information of their heads, and finally the number of times each student lowers his head during the shooting time period of the classroom lecture video of the students to be identified is identified based on the images of the detection frames with the facial features of each student to be identified as the number of abnormal head postures of each student.

[0068] Determine whether the three values ​​of the rotation Yaw around the vertical axis (z-axis), the rotation Pitch around the horizontal axis (y-axis), and the rotation Roll around the vertical axis (x-axis) in the three-degree-of-freedom information of the student's head are within the set range, and then determine whether the student is lowering his head, as follows: Head down state: Pitch<-5, -15<Roll<15 After testing, it was found from the data that 2 minutes before class, only 2% of the 80 students lowered their heads, and the number of times a single student in the class lowered his head did not exceed 2 times.

[0069] like Figure 6 As shown in Figure 2, for each student, the frequency of occurrence of different emotional states, fatigue, and the number of abnormal head postures are input into the pre-trained extreme learning machine model ELM, and the obtained seven-dimensional eigenvalues ​​are used as the 1×7 input vector ( x 1: Number of times you bow your head; x2: Number of yawns; x 3: Number of eye closures; x 4: Listen to the number of times the expression appears; x 5: The number of times the confused expression appears; x 6: The number of times the resistance expression appears; x 7: The number of times excited expressions appear) is input into the pre-trained extreme learning machine model ELM model to evaluate the students' attention, and finally obtain an attention evaluation parameter between 0 and 1. This value reflects different intervals, representing the different attention states and classroom effects of students in the process of listening to the class. Then, the average value of the attention evaluation parameters of each student at each moment is used as the average attention evaluation parameter of the class at the current moment. The time dimension is introduced for time series analysis. The preset time interval is 1min, and the video is recorded every 15s to obtain the average attention evaluation parameter of the class at a moment. The average value of the average attention evaluation parameters of the class at several moments in each preset time interval is used as the attention evaluation parameter of the class unit period, so as to judge the attention state of the students in the class. The specific mapping relationship is shown in Table 4.

[0070] Table 4 Relationship between attention assessment parameter Focus and attention state and classroom effect

[0071] The present invention also introduces the month-on-month increment ▽t The concept of "student attention" can be used to analyze the changes in students' attention in class in a more detailed manner, helping educators identify the fluctuation trend of students' attention from the time dimension, and thus provide more scientific data support for teaching intervention. By paying attention to the changes in the class in adjacent time periods, the interference of single-time data can be avoided, so as to more accurately evaluate and improve the teaching effect in the classroom, as follows: ▽t =(Focus t / Focus t-1 -1)×100%, t =2, 3, …, N Focus0=0 Among them, Focus t and Focus t-1 Respectively t Minutes and t -1 minute is the classroom unit cycle attention evaluation parameter, and N represents the total duration of the shooting time period of the classroom listening video of the student to be identified.

[0072] Eventually, the changes in students' attention in the classroom under the time sequence state can be calculated. Taking a 40-minute university course with 80 students as an example, when evaluating students' attention, in order to ensure the simplicity and intuitiveness of the interface, the visualization window on the computer side only displays the face frame and five key facial feature points of each identified student. Other attention-related behaviors such as the number of times you lower your head, yawn, close your eyes, etc. are all recorded and processed in real time in the system background through automated processes. Through these data, the system can provide real-time feedback on the changes in attention of each student in the classroom and provide decision support for teachers. Figure 7 As shown, the changing trend of attention of 80 students in 40 minutes and the corresponding month-on-month increment are shown, which is used to analyze and measure the difference in attention change of students in the class between two consecutive time periods; at the same time, it can be obtained that the student attention evaluation parameters obtained by the present invention roughly conform to the 10-minute rule, that is, the students' attention state is higher when they just start the class, but it can usually only last for about 10-15 minutes, and then it will begin to gradually decline. This phenomenon shows that students' attention is most concentrated at the beginning of the class, which may be due to curiosity and thirst for knowledge about the new course content. However, as time goes by, students' attention begins to gradually disperse, which may be due to fatigue caused by long-term concentration, or loss of interest in the class content. When the class is about to end, the student's attention will return to the classroom, and the facial expression will also be more positive, so the attention evaluation parameter will increase again.

[0073] In order to verify the accuracy of the established multi-feature student attention assessment model, the present invention uses manual supervision evaluation parameters to score the classroom effect, that is, three students are invited to watch the same classroom video at different time periods, and they are scored between 0 and 1 according to the standards in Table 4 and then compared with the evaluation parameters of the model in this paper. The comparison results show that the accuracy of the model is high and basically consistent with the manual evaluation, which shows that the method of the present invention has high accuracy and reliability.

[0074] The classroom student attention assessment of the present invention comprises a face detection unit, an expression recognition unit, a fatigue detection unit, a head posture estimation unit and an attention assessment unit. The face detection unit uses a face detection model RetinaFace to perform face recognition on a video frame of a classroom lecture video of a student to be identified obtained through a camera, and then outputs an image of a detection frame with facial features of each student as a face detection result, which can realize accurate detection of the student face area and ensure that small-scale, low-pixel and multi-occluded student faces can be effectively identified in complex classroom scenes; the expression recognition unit constructs a student expression recognition neural network model SFaceNet containing four expression discrimination blocks and performs training, so as to use the trained student expression recognition neural network model SFaceNet to identify the student face. The network model SFaceNet recognizes the different emotional states of each student as the expression recognition result, and then obtains the occurrence frequency of each student's different emotional states; the fatigue detection unit identifies the number of times each student closes his eyes and yawns as the number of fatigue points through facial key point positioning; the head posture estimation unit identifies the number of times each student lowers his head as the number of abnormal head postures through facial key point positioning; the attention evaluation unit inputs the frequency of occurrence of students' different emotional states, fatigue and the number of abnormal head postures into the pre-trained extreme learning machine model ELM, and outputs the students' attention evaluation parameters after processing. The attention status of students in the class is evaluated according to the attention evaluation parameters of each student and displayed on the monitor.

[0075] It should be noted that the parts not elaborated in the embodiments of the present invention belong to those common sense or disclosed prior art for those skilled in the art, and these contents are regarded as common knowledge in the industry.

[0076] In addition, for those skilled in the art, the steps in the embodiments described in the present invention, whether all or part, can be implemented by programming instructions corresponding to the hardware device. The corresponding control program can be stored in various computer-readable media, such as but not limited to read-only memory, hard disk or optical disk.

[0077] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for evaluating students' attention in classroom based on multi-source feature fusion, characterized in that: include: Step S1: Obtain several videos of students listening to lectures in class through a camera and extract initial video frames, perform face detection preprocessing on each initial video frame, and then identify different emotional states of the face by locating facial key points and adding expression labels, thereby obtaining several facial expression images and constructing a facial expression image dataset; Step S2: construct a student expression recognition neural network model SFaceNet containing four expression discrimination blocks, and train the student expression recognition neural network model SFaceNet using a facial expression image dataset; Step S3: Obtain the classroom video of the student to be identified through the camera and extract the video frame to be identified, then perform face recognition, and then identify the frequency of occurrence of different emotional states of each student through the trained student expression recognition neural network model SFaceNet, and identify the fatigue level and number of abnormal head postures of each student through facial key point positioning; Step S4: For each student, the frequency of occurrence of different emotional states, fatigue level and number of abnormal head postures of the student are input into the pre-trained extreme learning machine model ELM. After processing, the student's attention evaluation parameters are output. The attention status of the students in the class is evaluated and displayed according to the attention evaluation parameters of each student.

2. The method for evaluating classroom student attention based on multi-source feature fusion according to claim 1, characterized in that: In the step S1, each initial video frame is subjected to face recognition through the face detection model RetinaFace to obtain an image of a detection frame with facial features of each student, and then data enhancement processing is performed, including random cropping, adjustment to a uniform size, random horizontal flipping and image normalization, so as to obtain a preprocessed video frame.

3. The method for evaluating classroom student attention based on multi-source feature fusion according to claim 2 is characterized in that: In the step S1, each pre-processed video frame is identified by the 468 facial key point positioning method based on machine learning Mediapipe and the Russell emotion model to identify four emotional states of the face including rejection expression, listening expression, confusion expression and excitement expression. For the face of the student in each detection frame of the pre-processed video frame, the four emotional states of the face are as follows: a) Rejection expression: Determine the degree of mouth tilt. When the angle between the line connecting the 62nd key point, the 292nd key point and the 17th key point and the horizontal line is negative, the degree of mouth tilt is that the corners of the mouth on both sides move downward, and the current emotional state of the face is determined to be a rejection expression. b) Listen to the expression: To judge the degree of mouth tilt, when the line connecting the 62nd key point, the 292nd key point and the 17th key point in the face is parallel to the horizontal line, the degree of mouth tilt is not obvious; Determine the degree of eye openness, obtain the eye height of the left eye in the eye feature by subtracting the ordinates of the 470th key point and the 472nd key point in the face, obtain the eye width of the left eye in the eye feature by subtracting the abscissas of the 33rd key point and the 133rd key point in the face, obtain the eye height of the right eye in the eye feature by subtracting the ordinates of the 475th key point and the 477th key point in the face, obtain the eye width of the right eye in the eye feature by subtracting the abscissas of the 362nd key point and the 263rd key point in the face, and if the ratio of the eye height to the eye width of the left eye and the ratio of the eye height to the eye width of the right eye are both greater than the ratio threshold, the degree of eye openness of the face is open; To judge the degree of eyebrow curvature, if the angles between the 70th key point and the 65th key point and the 105th key point in the face are greater than the angle threshold, and the angles between the 300th key point and the 295th key point and the 334th key point in the face are greater than the angle threshold, the degree of eyebrow curvature of the face is stretched; When the inclination of the mouth of the human face is not obvious, the opening degree of the eyes of the human face is open, and the curvature degree of the eyebrows of the human face is stretched, it is determined that the current emotional state of the human face is a listening expression; c) Confused expression: To judge the degree of eyebrow curvature, if the angles between the 70th key point and the 65th key point and the 105th key point in the face are less than the angle threshold, and the angles between the 300th key point and the 295th key point and the 334th key point in the face are less than the angle threshold, the degree of eyebrow curvature of the face is frowning; When the inclination of the mouth of the face is that the corners of the mouth move downward, and the curvature of the eyebrows of the face is tightly wrinkled, it is determined that the current emotional state of the face is a confused expression; d) Excited expression: Determine the degree of mouth tilt. When the angle between the line connecting the 62nd key point, the 292nd key point and the 17th key point and the horizontal line is a positive number, the degree of mouth tilt is that the corners of the mouth on both sides are upturned. When the inclination of the mouth of the face is that the corners of the mouth on both sides are upturned, the eye opening degree of the face is open, the curvature degree of the eyebrows of the face is stretched, and the Euclidean distance between the 12th key point and the 14th key point of the mouth of the face is greater than the preset distance threshold, it is determined that the current emotional state of the face is a listening expression; An expression label of an emotional state is added to the face of each student in the detection frame.

4. The method for evaluating classroom student attention based on multi-source feature fusion according to claim 1, characterized in that: In the step S2, the student expression recognition neural network model SFaceNet includes an input layer, a feature extraction and fusion layer, and an output layer connected in sequence, the input layer includes a first convolutional layer, a first batch normalization layer Batch Norm, a first activation function layer ReLu, and a maximum pooling layer Maxpooling connected in sequence, the feature extraction and fusion layer includes three expression discrimination blocks EmotionBlock V1, four expression discrimination blocks EmotionBlock V2, six expression discrimination blocks EmotionBlock V3, three expression discrimination blocks EmotionBlock V4, and an attention mechanism module CBAM connected in sequence, and also includes a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, the output of the maximum pooling layer Maxpooling is then input into the first expression discrimination block EmotionBlock V1, the maximum pooling layer Maxpooling and the output of the first expression discrimination block EmotionBlock V1 are added to obtain a first addition result, the first addition result is input into the second expression discrimination block EmotionBlock V1, the first addition result and the second expression discrimination block EmotionBlock V1 to obtain a second addition result, and the second addition result is input into the third expression discrimination block one EmotionBlock V1. The second addition result and the output of the third expression discrimination block one EmotionBlock V1 are added to obtain a third addition result, and the third addition result is input into the first expression discrimination block two EmotionBlock V2. The third addition result is processed by the second convolutional layer and added to the output of the first expression discrimination block two EmotionBlock V2 to obtain a fourth addition result. The fourth addition result is input into the second expression discrimination block two EmotionBlock V2. The fourth addition result and the output of the second expression discrimination block two EmotionBlock V2 are added to obtain a fifth addition result. The fifth addition result is input into the third expression discrimination block two EmotionBlock V2. The fifth addition result and the output of the third expression discrimination block two EmotionBlock V2 are added to obtain a sixth addition result. The sixth addition result is input into the fourth expression discrimination block two EmotionBlock V2. The sixth addition result and the fourth expression discrimination block two EmotionBlock V2 are added to obtain a sixth addition result. The outputs of V2 are added to obtain a seventh addition result, which is input into the first expression discrimination block three EmotionBlock V3. The seventh addition result is processed by the third convolutional layer and added to the output of the first expression discrimination block three EmotionBlock V3 to obtain an eighth addition result, which is input into the second expression discrimination block three EmotionBlock V3.The eighth addition result is added to the output of the second expression discrimination block three EmotionBlock V3 to obtain a ninth addition result, and the ninth addition result is input into the third expression discrimination block three EmotionBlock V3. The ninth addition result is added to the output of the third expression discrimination block three EmotionBlock V3 to obtain a tenth addition result, and the tenth addition result is input into the fourth expression discrimination block three EmotionBlock V3. The tenth addition result is added to the output of the fourth expression discrimination block three EmotionBlock V3 to obtain an eleventh addition result, and the eleventh addition result is input into the fifth expression discrimination block three EmotionBlock V3. The eleventh addition result is added to the output of the fifth expression discrimination block three EmotionBlock V3 to obtain a twelfth addition result, and the twelfth addition result is input into the sixth expression discrimination block three EmotionBlock V3. The twelfth addition result and the sixth expression discrimination block three EmotionBlock V3 are added together. The output of V3 is added to obtain the thirteenth addition result, the thirteenth addition result is processed by the fourth convolutional layer and added to the output of the first expression discrimination block four EmotionBlock V4 to obtain the fourteenth addition result, the fourteenth addition result is input to the second expression discrimination block four EmotionBlock V4, the fourteenth addition result and the output of the second expression discrimination block four EmotionBlock V4 are added to obtain the fifteenth addition result, the fifteenth addition result is input to the third expression discrimination block four EmotionBlock V4, the fifteenth addition result and the output of the third expression discrimination block four EmotionBlock V4 are added and then input to the attention mechanism module CBAM; the output layer includes the average pooling layer Avgpooling and the fully connected layer FC connected in sequence. , 5. The method for evaluating classroom student attention based on multi-source feature fusion according to claim 4 is characterized in that: The expression discrimination block EmotionBlock V1, the expression discrimination block EmotionBlock V2, the expression discrimination block EmotionBlock V3 and the expression discrimination block EmotionBlock V4 include a second batch normalization layer Batch Norm, a second activation function layer ReLu, a fifth convolutional layer, a third batch normalization layer Batch Norm, a third activation function layer ReLu, a sixth convolutional layer, a context converter module CoT, a fourth batch normalization layer Batch Norm, a fourth activation function layer ReLu and a seventh convolutional layer connected in sequence.

6. The method for evaluating classroom student attention based on multi-source feature fusion according to claim 1, characterized in that: In the step S3, for each video frame to be identified, each video frame to be identified is subjected to face recognition through the face detection model RetinaFace to obtain an image of the detection frame with the facial features of each student to be identified, and then the image of the detection frame with the facial features of each student to be identified is input into the trained student expression recognition neural network model SFaceNet for processing to obtain the emotional state of each student in the current video frame, and finally the frequency of occurrence of different emotional states of each student during the shooting time period of the classroom lecture video of the student to be identified is identified according to the images of the detection frame with the facial features of each student to be identified.

7. The method for evaluating classroom student attention based on multi-source feature fusion according to claim 6 is characterized in that: In the step S3, for each image with a detection frame of each student's facial features to be identified, it is first processed by a 468 facial key point positioning method based on machine learning Mediapipe to obtain 468 key point information of each student's face in the graphic. For each student, 6 key points are taken from each of the three parts of the student's face, namely the left eye, right eye and mouth, for a total of 18 key points. The eye aspect ratio EAR is obtained based on the 6 key points of the left eye and the right eye of the student's face, and then the student's eye closing state is judged; the mouth opening degree MAR is obtained based on the 6 key points of the student's mouth, and then the student's yawning state is judged. Finally, the number of times each student closes his eyes and yawns during the shooting period of the classroom lecture video of the student to be identified is identified according to each image with a detection frame of each student's facial features to be identified as the number of times each student is tired.

8. The method for evaluating classroom student attention based on multi-source feature fusion according to claim 6, characterized in that: In the step S3, for each video frame to be identified and for each image of a detection frame with facial features of each student to be identified, firstly, the image is processed by the 468 facial key point positioning method based on machine learning Mediapipe to obtain the two-dimensional coordinates of 468 key points of the face of each student in the figure, and then the two-dimensional coordinates are normalized, and then the two-dimensional coordinates of each normalized key point are mapped to the three-dimensional world coordinate system. For each student, the three-dimensional coordinates and the two-dimensional coordinates of each key point of the student's face in the three-dimensional world coordinate system are matched by the pose estimation algorithm PnP, so as to obtain the rotation and displacement of the camera, and finally obtain the three-degree-of-freedom information of the student's head, judge the student's head-lowering state according to the three-degree-of-freedom information of the student's head, and finally identify the number of times each student lowers his head during the shooting time period of the classroom listening video of the student to be identified according to each image of the detection frame with facial features of each student to be identified as the number of abnormal head postures of each student.

9. The method for evaluating classroom student attention based on multi-source feature fusion according to claim 3, characterized in that: In step S4, the average value of the attention evaluation parameters of each student at each moment is used as the average attention evaluation parameter of the class at the current moment, and the average value of the average attention evaluation parameters of the class at several moments within each preset time interval is used as the class unit period attention evaluation parameter, so as to judge the attention state of the students in the class, as follows: When 0≤Focus t When it is less than 0.4, the students in the class have very poor attention. t It is the attention assessment parameter of the class unit cycle at the current preset time interval; When 0.4≤Focus t When it is less than 0.6, the students in the class have poor attention status; When 0.6≤Focus t When it is less than 0.8, the students’ attention in the class is average; When 0.8≤Focus t When it is less than 1.0, the students in the class have excellent attention.

10. A classroom student attention assessment system applicable to the method according to any one of claims 1 to 9, characterized in that: include: The face detection unit uses the face detection model RetinaFace to perform face recognition on the video frames of the classroom lecture video of the students to be identified obtained through the camera, and then outputs an image of the detection frame with the facial features of each student as the face detection result; The expression recognition unit constructs and trains the student expression recognition neural network model SFaceNet, which contains four expression discrimination blocks. The trained student expression recognition neural network model SFaceNet is used to identify the different emotional states of each student as the expression recognition result, and then the occurrence frequency of each student's different emotional states is obtained. The fatigue detection unit locates the key points of the face and identifies the number of times each student closes his eyes and yawns as the number of times of fatigue; The head posture estimation unit identifies the number of times each student lowers his head as the number of abnormal head postures by locating facial key points; The attention assessment unit inputs the frequency of occurrence of students' different emotional states, fatigue levels, and the number of abnormal head postures into the pre-trained extreme learning machine model ELM. After processing, it outputs the students' attention assessment parameters. Based on the attention assessment parameters of each student, the attention status of the students in the class is evaluated and displayed on the monitor.

Images