A method for recognizing academic emotions based on human feature fusion

Abstract

The present invention discloses a method for recognizing academic emotions based on human feature fusion, belonging to the field of emotion recognition. Based on classroom monitoring image data, the present invention extracts features of the facial expressions and body postures of learners through a target detection and feature extraction module, and obtains vector representations of the expression and posture features of the learners. On this basis, the obtained feature vectors are spliced and fused, and then a two-branch academic emotion recognition network is proposed to recognize the academic emotions of learners in the classroom. The present invention solves the problems of low accuracy and poor robustness of traditional academic emotion recognition methods, and helps teachers to more accurately understand the learning status and knowledge mastery of students to improve teaching quality.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition, and in particular to an academic emotion recognition method based on human body feature fusion. Background Art

[0002] Academic emotion recognition refers to the identification and analysis of learners' emotional states during the learning process. Leveraging AI technology to empower offline classroom teaching activities and analyze students' academic emotions can help identify problems in the teaching process, allowing for targeted adjustments to teaching methods and improving teaching quality.

[0003] Currently, mainstream methods for identifying learners' academic emotions in classroom settings rely solely on facial expression recognition, analyzing data collected from classroom monitors to infer students' emotions. Lasri et al. constructed a student emotion recognition system. The system first uses a cascade classifier with Haar features for face detection. Then, a convolutional neural network is trained on the FER2013 dataset, enabling the convolutional neural network to recognize seven basic expressions and thus infer learners' emotions. Bian et al. constructed a publicly available online facial expression dataset of learners, containing facial expressions representing five common academic emotions (confusion, enjoyment, fatigue, distraction, and neutrality). The database was collected from 82 student volunteers. This study also introduced an adaptive data augmentation algorithm based on a spatial transformer network to remove clutter from the original images, ultimately enabling emotion recognition through a convolutional neural network. Han Li et al. proposed an efficient classroom evaluation method based on facial expression analysis. The method first combines the AAM model with the local constraint model to detect facial feature points on images obtained through intelligent monitoring equipment, and then judges five expressions (listening, doubt, understanding, resistance, and disdain) through the changes in the characteristics of the corners of the mouth, the angle between the eyes and eyebrows, and the facial offset angle. Finally, the panoramic teaching effect of the classroom is measured by participation, attention, active time, and degree of doubt; Sun Bo et al. proposed a detailed correspondence between seven academic emotions and facial activity features based on the facial activity unit coding system, and through feature decomposition, individual features and expression features can be decomposed into different subspaces, eliminating the interference of individual features on expression recognition.

[0004] In summary, the existing methods for academic emotion recognition mainly rely on facial expression recognition only, rarely considering other human body features beyond facial expressions. However, in the offline classroom scenario, due to reasons such as large differences in the resolution of students' facial expressions, low expression activation levels, and facial occlusion, the expression features are less distinguishable or unavailable, resulting in low recognition accuracy and poor robustness of the emotion recognition method based solely on expressions, making it difficult to achieve practical applications. Learners' emotion expression media are not limited to facial expressions, and the expression of emotions through other parts of the body is also obvious. Therefore, combining more human body features for academic emotion recognition is an important way to improve the recognition accuracy. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings of the existing technologies and provide an academic emotion recognition method based on the fusion of human body features.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions:

[0007] An academic emotion recognition method based on the fusion of human body features includes the following steps:

[0008] 1) Input the original data P of the monitoring video into the human target detection module Faster R-CNN, process it based on the target detection algorithm to output the student individual box to be recognized, and then extract the local area image information O of each student i and the corresponding human body bounding box B i ;

[0009] 2) Align the local area image information O of the student to be recognized i through the face alignment module MTCNN, then obtain the preprocessed face image through data augmentation, input the preprocessed face image into the expression feature extraction network for expression feature learning, and finally output the expression feature X1;

[0010] Input the student individual box to be recognized and the global image into the joint candidate single-person pose estimation module in the CrowdPose algorithm to estimate the human body joint points and output a set of heatmaps indicating the human body joint position information. Then, through the global average pooling layer and a multi-layer perceptron, convert the heatmaps into 512-dimensional vectors as the pose feature X2 of the student to be recognized;

[0011] 3) Concatenate the expression feature X1 and the pose feature X2 to obtain X b , and input X b into the academic emotion classifier for learner academic emotion recognition.

[0012] Further, in step 2), the feature extraction network for facial expression feature learning is constructed by making the following adjustments based on the pre-trained VGGFace face recognition model:

[0013] (1) Insert a global average pooling layer after the last convolutional layer in the original VGGFace network;

[0014] (2) Change the last three fully-connected layers in the original VGGFace to two fully-connected layers;

[0015] (3) Set the output channels of the last fully-connected layer to 512;

[0016] (4) Add a Dropout layer after the last fully-connected layer, and set the random inactivation probability to 0.3;

[0017] (5) Replace the original classification loss softmax loss function with an island loss function during training. The loss function is expressed as:

[0018] L cls =L c +αL p

[0019] where L c represents the distance between the sample and the feature clustering center of this class of samples; L cls represents the loss value of the classification module, α represents the distance weight between classes, and L p represents the distance between the feature clustering centers of samples of different classes.

[0020] Further, in step 2), the preprocessing method for the face image is as follows:

[0021] For the face image aligned by MTCNN, three methods of randomly erasing image regions, histogram equalization, and horizontal flipping of the image are used to perform data augmentation operations on the face image.

[0022] Further, in step 2), the pose feature estimation adopts a top-down multi-person pose estimation strategy based on the CrowdPose algorithm.

[0023] Further, the 17 key points are the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0024] Further, in step 3), both the facial expression feature X1 and the pose feature X2 are 512-dimensional vectors. Therefore, the concatenated X b is a 1024-dimensional vector.

[0025] Further, in step 3), the emotion classifier consists of a fully connected layer, a batch normalization layer, a Dropout layer, and an output layer:

[0026] Fully connected layer: Maps the input X b to a 256-dimensional feature vector;

[0027] Batch normalization layer: Normalizes the input 256-dimensional feature vector, that is, normalizes the data on each feature dimension;

[0028] Dropout layer: Randomly sets the outputs of some neurons to zero; there is a hyperparameter p in the Dropout layer, representing the probability of randomly setting the neuron outputs to zero;

[0029] Output layer: A fully connected layer that maps the output vector of the Dropout layer to a 7-dimensional vector, representing 7 different emotion classification results.

[0030] Further, the hyperparameter p is set to 0.6, that is, 60% of the neuron outputs are randomly set to zero.

[0031] Further, the 7 different emotion classification results are boredom, relaxation, concentration, confusion, frustration, happiness, and fatigue.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] When the academic emotion recognition algorithm based on human feature fusion of the present invention is operating, by combining facial expression recognition and human pose estimation technologies, it comprehensively considers the learner's expression and pose features to perform academic emotion recognition. In the neural network structure design, based on VGGFace, the network structure is adjusted, and the idea of transfer learning is used to improve the generalization ability of the model in the case of less data volume. At the same time, the island loss function is introduced to guide the neural network to perform "intra-class cohesion and inter-class separation" expression feature learning, improving the accuracy of expression recognition. This method fuses pose features and expression features for academic emotion recognition, enhancing the robustness to expression features in the academic emotion recognition task. Experiments show that: in the learner academic emotion recognition task in the real classroom scenario, the recognition effect of the present invention is better than the classic expression-based academic emotion recognition method. Description of the Drawings

[0034] Figure 1 is the framework diagram of the present invention;

[0035] Figure 2 is the annotation example diagram of the dataset used in the present invention;

[0036] Figure 3 is the emotion classification example diagram of the dataset used in the present invention. Specific Embodiment

[0037] In order to enable those skilled in the art of this technology to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data used in appropriate cases can be interchanged so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings:

[0040] See Figure 1 , the academic emotion recognition method based on human feature fusion of the present invention includes the following steps:

[0041] 1) Data preprocessing of the expression feature extraction branch

[0042] The original data P is first processed by the object detection algorithm Faster R-CNN trained on the MS COCO dataset to obtain the local image O of the learner to be recognized i and the learner individual box B i . On this basis, face alignment is performed through the MTCNN algorithm to normalize the face parts in different individual images to the same position, size and direction, so as to better extract face features and thus improve the accuracy of expression recognition. If the student to be recognized is in a bowed head state or the face is blocked by people or objects around, resulting in the MTCNN being unable to detect the face, an image of a fixed size will be generated, and this image is filled with 1 and used instead of the face image to be input into the expression feature extraction network. Next, data augmentation operations are performed on the obtained face images, aiming to reduce the overfitting risk of the deep neural network and improve the generalization ability of the model. The data augmentation methods used include the following three:[[]]

[0043] (1) Random Erasing of Image Region: The specific operation is to randomly select a rectangular region in the image and replace its pixel values with 0 values. Random erasing operations are performed on some regions in the input image data to simulate various occlusion situations in a real classroom scenario.

[0044] (2) Histogram Equalization: Histogram equalization redistributes the pixel values of the image, making the pixel values more evenly distributed across the entire brightness range, thereby increasing the contrast of the image. By histogram equalization, the originally indistinguishable textures become more distinguishable, thereby alleviating the negative impact on model learning caused by relatively bright or dark local light in the classroom.

[0045] (3) Horizontal Image Flipping: Flip an image 180 degrees horizontally to obtain a new mirror-symmetric image. By horizontally flipping the image, it can better alleviate the differences in the appearance of students caused by different perspectives of surveillance images in the classroom. These differences may lead to biased features learned during subsequent neural network learning.

[0046] 2) Construction of the Network Model for Facial Expression Feature Extraction Branch

[0047] In recent years, the field of face recognition has developed rapidly and has achieved very high accuracy in some real scenarios. Therefore, based on the idea of transfer learning, this method uses the pre-trained VGGFace face recognition model as the source model for transfer learning and adjusts it to obtain the model for the facial expression recognition branch. The model parameters of VGGFace are trained on a dataset of 2.6 million face images, with an accuracy of 98.95% on the LFW face recognition dataset and 97.3% on the YTF face recognition dataset. The original network structure of VGGFace is shown in Table 1. It has a total of 5 convolutional blocks, each convolutional block includes 2 or 3 convolutional layers, and each convolutional layer is followed by a ReLu activation function unit and a pooling layer. The size of all convolutional kernels in VGGFace is 3*3, and the stride of all pooling layers is 2. The last layer of the network is 3 fully connected layers and a softmax layer.

[0048] Table 1 VGGFace Network Structure

[0049]

[0050]

[0051] The facial expression feature extraction branch network of the present invention makes the following adjustments based on the pre-trained VGGFace face recognition model:

[0052] (1) Insert a global average pooling layer (Avg Poll) after the last convolutional layer in the original VGGFace network because the global average pooling layer can effectively reduce the network model parameters, reducing the risk of overfitting during the training process.

[0053] (2) Change the last three fully connected layers in the original VGGFace to two fully connected layers. This is because it is found during the actual training process that appropriately reducing the number of fully connected layers based on the original network model will not affect the result of facial expression recognition.

[0054] (3) Set the number of output channels of the last fully connected layer to 512.

[0055] (4) Add a Dropout layer after the last fully connected layer, with the random inactivation probability set to 0.3 to reduce the risk of model overfitting.

[0056] (5) Replace the original classification loss softmax loss function with the island loss function during training to help the neural network learn more discriminative facial expression features. The loss function is expressed as:

[0057] L cls =L c +αL p

[0058] Among them, L c represents the distance between a certain sample and the feature clustering center of samples in this class to punish the intra-class distance; L cls represents the loss value of the classification module, α represents the inter-class distance weight, set to 0.2, and L p represents the distance between the feature clustering centers of samples in different classes to punish the inter-class distance. The adjusted network structure is shown in Table 2, where CNNlayers represents the set of several convolutional layers in Table 1. Dropout(0.3) represents that the random inactivation probability of Dropout is 0.3.

[0059] Table 2 Adjusted network structure of the facial expression feature extraction branch

[0060]

[0061] 3) Human pose feature extraction

[0062] The pose feature extraction branch adopts a top-down multi-person pose estimation strategy based on the CrowdPose algorithm. This algorithm uses a novel and effective way to solve the pose estimation problem in a crowd. The whole model consists of two key components: Joint-candidate Single Person Pose Estimation (Joint-candidate SPPE) and Global Association. This method first detects individuals and then performs single-person pose estimation, which is robust to the inevitable interference in crowded scenes and has high inference efficiency.

[0063] In 1), the local image O of the learner to be recognized is obtained through the human object detection module i and the learner individual box B i On this basis, first, the human box B i and the global image P are jointly input into the pose estimation network. First, the Faster R-CNN detector is used for human detection to obtain the individual box of the student to be recognized. Then, the individual box of the student to be recognized and the global image are jointly input into the Joint-candidate Single Person Pose Estimation module in the CrowdPose algorithm to estimate the 17 joint points of the human body (nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle), and then a set of heatmaps indicating the position information of the human joints is obtained. Finally, a global average pooling layer and a multi-layer perceptron are concatenated after the Joint-candidate Single Person Pose Estimator Joint-candidate SPPE to convert the output heatmap into a 512-dimensional vector, which is used as the pose feature X2 of this person, expressed as:

[0064] X2 = MLP(AvgPool(X iheatmap ))

[0065] Among them, MLP represents the multi-layer perceptron, which is a fully connected layer in the present invention. AvgPool represents the global average pooling layer, and X iheatmap represents the joint point heatmap of the i-th person. Joint-Candidates SPPE adopts the parameters pre-trained on the CrowdPoseDataset dataset.

[0066] 4) Human feature fusion

[0067] In the final stage of this method, the obtained expression feature vector X1 and pose feature vector X2 are concatenated to obtain X b to comprehensively consider the facial expression and body pose features. Then, X b is input into the academic emotion classifier for learner academic emotion recognition. Since both the expression feature X1 and the pose feature X2 are 512-dimensional vectors, the concatenated Xb It is a 1024-dimensional vector. The academic emotion classifier consists of the following parts:

[0068] (1) Fully connected layer fc_in: Maps the input joint feature vector X b into a 256-dimensional feature vector.

[0069] (2) Batch normalization layer bn: Normalizes the input 256-dimensional feature vector, that is, normalizes the data on each feature dimension to accelerate convergence and enhance generalization performance.

[0070] (3) Dropout layer dp: Randomly sets the outputs of a part of the neurons to zero to prevent overfitting. This layer has a hyperparameter p, representing the probability of randomly setting the neuron outputs to zero, which is set to 0.6 here, that is, randomly setting 60% of the neuron outputs to zero.

[0071] (4) Output layer fc_out: This layer consists of a fully connected layer, which maps the output vector of the previous layer into an n-dimensional vector. Combining previous related research and the investigation and analysis of classroom monitor images, this method classifies classroom academic emotions into seven categories (bored, relaxed, focused, confused, frustrated, happy, and fatigued), so n is set to 7 to represent the classification results of the model for 7 academic emotions.

[0072] The total loss function of the academic emotion recognition algorithm based on human feature fusion in the present invention during training is expressed as:

[0073] L BF = α * L1 + β * L2

[0074] Among them, L1 is the loss function of the expression feature extraction branch, and L2 is the loss function of the pose feature extraction branch. The hyperparameters α and β respectively control the weights of the two loss functions, and are uniformly set to 0.5. During the training process, the parameters of the expression feature extraction network and the CrowdPose heatmap learning network are both fixed, and only the parameters of the fully connected layer are learned.

[0075] Embodiment

[0076] (1) Dataset description

[0077] The present invention uses the classroom scene monitoring camera videos collected in real time from the "Real-time Detection Big Data Platform for Education and Teaching Quality" of universities as the dataset for recognizing the academic emotions of students in offline classroom scenes. Referring to the annotation formats for discrete emotions in the public datasets EMOTIC dataset and CSRE-S dataset, at the same time, the human body frames and head frames of students are annotated, and the annotated academic emotions are divided into seven categories: bored, relaxed, focused, confused, frustrated, happy, and fatigued.

[0078] (2) Comparative model description

[0079] The Baseline comparison model in this paper is VGG-RDA-ADA proposed by Bian et al. The dataset used by this comparison model is the same as the dataset used by the method proposed in the present invention.

[0080] (3) Student academic emotion recognition

[0081] Based on the constructed dataset above, each student individual image region is cut out from the whole image to become a separate image. A total of 24,000 single-person image regions containing a frontal face or a face partially or completely occluded are screened out, and they are divided into a training set and a test set according to a ratio of 4:1. The deep learning framework used is Pytorch version 1.4, and the training is carried out on a single TITANX GPU. The initial learning rate is set to 0.001, and the cosine annealing learning rate execution strategy is adopted during the training process. The training batch size is set to 16. In the experiment, the average accuracy of various academic emotion recognitions is used to measure the performance of the model. The comparison between the method proposed in the present invention and the Baseline effect is shown in Table 1.

[0082] Table 1 Experimental results of academic emotion recognition

[0083]

[0084] As can be seen from Table 1, the academic emotion recognition method proposed in the present invention that fuses expression features and pose features is superior to the Baseline method in terms of the average accuracy of academic emotion classification, and the average accuracy is increased by 7.04% compared with the Baseline method, which verifies the effectiveness of the method proposed in the present invention.

[0085] The above content is only to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any modification made on the basis of the technical solution according to the technical idea proposed in the present invention falls within the protection scope of the claims of the present invention.

Claims

1. A method for recognizing academic emotions based on human feature fusion, characterized in that, It includes the following steps: 1) The original data of the surveillance video is input into the human target detection module Faster R-CNN, and the individual student boxes to be recognized are processed and output based on the target detection algorithm, and then the local region image information of each student and the corresponding human body bounding box are extracted are obtained; ; 2) The local region image information of the student to be recognized is aligned by the face alignment module MTCNN, and then the preprocessed face image is obtained through data augmentation. The preprocessed face image is input into the expression feature extraction network for expression feature learning, and finally the expression features are output ; The student individual box to be recognized and the global image are jointly input into the joint candidate single-person pose estimation module in the CrowdPose algorithm to estimate human joint points and output a set of heatmaps indicating the position information of human joints. Then, through the global average pooling layer and a multi-layer perceptron, the heatmaps are transformed into 512-dimensional vectors as the pose features of the student to be recognized. ; 3) Concatenate the facial expression features and the pose features to obtain , and input into the academic emotion classifier for learner academic emotion recognition; In step 2), the feature extraction network for facial expression feature learning is constructed by making the following adjustments based on the pre-trained VGGFace face recognition model: (1) Insert a global average pooling layer after the last convolutional layer in the original VGGFace network; (2) Change the last three fully connected layers in the original VGGFace to two fully connected layers; (3) Set the output channels of the last fully connected layer to 512; (4) Add a Dropout layer after the last fully connected layer, and set the random inactivation probability to 0.3; (5) Replace the original classification loss softmax loss function with an island loss function during training. The loss function is expressed as: Among them, represents the distance between the sample and the clustering center of the sample features of this class; represents the loss value of the classification module, represents the distance weight between classes, represents the distance between the clustering centers of the sample features of different classes; In step 3), the emotion classifier consists of a fully connected layer, a batch normalization layer, a Dropout layer, and an output layer as follows: Fully connected layer: Map the input into a 256-dimensional feature vector; Batch normalization layer: Normalize the input 256-dimensional feature vector, that is, normalize the data on each feature dimension; Dropout layer: Randomly set the output of a part of the neurons to zero; there is a hyperparameter p in the Dropout layer, which represents the probability of randomly setting the neuron output to zero; Output layer: A fully connected layer that maps the output vector of the Dropout layer to a 7-dimensional vector, representing 7 different emotion classification results.

2. The academic emotion recognition method based on human feature fusion according to claim 1, wherein In step 2), the preprocessing method for the face image is: For the face image aligned by MTCNN, use three methods of random erasing in the image area, histogram equalization, and horizontal flipping of the image to perform data augmentation operations on the face image.

3. The academic emotion recognition method based on human feature fusion according to claim 1, characterized in that, In step 2), the pose feature estimation adopts a top-down multi-person pose estimation strategy based on the CrowdPose algorithm.

4. The academic emotion recognition method based on human feature fusion according to claim 1, wherein The 17 key points are the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

5. The academic emotion recognition method based on human feature fusion according to claim 1, characterized in that In step 3), the facial expression feature and the pose feature are both 512-dimensional vectors. Therefore, the obtained after splicing is a 1024-dimensional vector.

6. The academic emotion recognition method based on human feature fusion according to claim 1, characterized in that The hyperparameter p is set to 0.6, that is, 60% of the neuron outputs are randomly set to zero.

7. The academic emotion recognition method based on human feature fusion according to claim 1, characterized in that The 7 different emotion classification results are boredom, relaxation, concentration, confusion, frustration, happiness, and fatigue.

Images