Personalized feature enhanced emotion recognition method

Abstract

The invention relates to a personalized feature enhanced emotion recognition method, and belongs to the field of online education emotion calculation. According to the method, on the basis of extracting student facial expression image spatial features, emotion association of different image frames in a time dimension is captured by using a bidirectional long-short-term memory network, and effective student facial spatial-temporal features are obtained; and on the basis of a psychological'big five personality 'theory, personalized personality features hidden in student facial expressions are mined by using a deep residual network and are fused with facial features, so that student emotion classification is realized for references of teachers. According to the method, the accuracy of student learning emotion recognition is improved, the problem of unsmooth teacher-student emotion exchange in online teaching can be effectively relieved, the teacher is helped to master the student classroom state in online teaching, and the teaching implementation quality is improved.

Description

Technical Field

[0001] The present invention belongs to the field of online education emotion computing and relates to an emotion recognition method with enhanced personalized features. Background Art

[0002] Classrooms are important places for disseminating educational knowledge. Students' emotions in the classroom represent their state of class. Teachers' grasp of the state of the students in the class can help teachers adjust the classroom atmosphere, knowledge difficulty and teaching methods, and pay special attention to individual students with learning difficulties, thereby improving the overall quality of classroom teaching. With the development of the Internet, online education and teaching models have become more mature and have become an important way of modern education. Online teaching platforms such as Tencent Conference and DingTalk are more widely used by students and teachers. Online education brings many conveniences to teachers and students due to its unique time and space advantages, but due to the spatial distance, teachers' grasp of students' classroom status is not as good as traditional offline teaching models. How to use modern technology to overcome the problem of poor emotional communication between teachers and students in online teaching is one of the important challenges to promote the development of online education and improve the quality of online education and teaching.

[0003] In the field of online education emotional computing, current research mainly obtains student emotional data through psychological measurement, behavioral measurement and physiological measurement, such as behavioral data such as student facial expressions, voices, body postures, and physiological data such as student electroencephalograms, electrocardiograms, and eye tracking. The acquired data is analyzed using emotional computing technology to achieve student emotional analysis. For example, Apicella et al. converted the EEG signals obtained from physiological measurements into digital signals through a digital-to-analog converter in "EEG-based measurement system for monitoring student engagement in learning 4.0" [in Scientific Reports, 2022, 12(1): 1-13]. The digital signals after filtering were extracted using principal component analysis, and the support vector machine was used to identify student classroom participation. However, psychological measurement is highly subjective and requires students to accurately and truthfully reflect their personal emotions; physiological measurement requires students to wear sensors, which has the defects of distracting students and reducing comfort, and can easily lead to untrue and inaccurate emotion recognition results. Therefore, many current studies analyze students' behavioral data to reduce the intrusiveness of data collection. For example, Savchenko AV et al. used student facial images obtained from behavioral measurements as research objects in "Classifying emotions and engagement in online learning based on a single facial expression recognition neural network" [in IEEE Transactions on Affective Computing, 2022, 13(4): 2132-2143]. They used a lightweight CNN to train emotion classification for static facial images and used the trained model to detect the engagement status of online students. However, using only students' facial expressions as clues results in a low accuracy rate in emotion recognition results.

[0004] Although relevant research on student emotion recognition has achieved certain results in the past, student emotion recognition in online scenarios still faces many challenges. On the one hand, the non-invasiveness and reliability of data collection can achieve high-accuracy emotion recognition using available data sets. On the other hand, psychological research shows that people with different personalities have different expressions of the same emotion. In the past, student emotion recognition lacked consideration of the impact of personality on the expression of emotion, and did not model the personality characteristics of students. It was difficult to effectively analyze students with introverted personalities and whose emotions were not obvious in their behavior, which restricted the accurate analysis of real emotions. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an emotion recognition method with enhanced personalized features, which collects non-invasive student facial expression data, uses the student facial expression data to mine potential personalized feature information, and assists the student facial features to achieve more accurate emotion recognition results, so as to solve the problems of poor emotional communication between teachers and students and limited accuracy of emotion recognition in online education.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A personalized feature-enhanced emotion recognition method specifically comprises the following steps:

[0008] S1: Collect students’ facial video data during class and perform data preprocessing;

[0009] S2: Construct a student facial spatiotemporal feature extraction model, and use deep residual network, attention mechanism and bidirectional long short-term memory network to extract effective student facial spatiotemporal features;

[0010] S3: Build a personalized feature extraction model and use the improved deep residual neural network to mine the personalized characteristics of the personality contained in the facial expression videos of students during class;

[0011] S4: Construct a feature fusion and classification model to integrate students’ facial spatiotemporal features and personalized features. The model outputs the emotion category of students’ facial video samples during class based on the fused features.

[0012] S5: Train the classification model until the model converges, so that the model can achieve emotion recognition results with high accuracy.

[0013] Further, step S1 specifically includes the following steps:

[0014] S11: using a high-definition camera to collect video data of students’ facial expressions during class, and extracting valid image frames for each segment of student facial video data;

[0015] S12: Perform random cropping and normalization preprocessing operations on each image frame, resize each facial image frame of the student to a size of 224×224 and convert it into a tensor.

[0016] Further, step S2 specifically includes the following steps:

[0017] S21: Input the preprocessed student facial image into the deep residual network, use the adaptive average pooling layer to control the output dimension to 1, and obtain the visual features of the student facial image with a dimension of 1;

[0018] S22: Input the visual features of each image into the self-attention module composed of a fully connected layer and a sigmoid activation function, calculate the attention distribution of the student's single-frame image features, and obtain the feature vector of the attention weight of the single frame

[0019]

[0020] Where, T i represents the spatial features of each image frame, w i Represents the spatial features of the image frame T i The attention weight vector obtained by the fully connected layer, σ represents the sigmoid activation function, and the image space features obtained by the self-attention module are connected to obtain the following expression:

[0021]

[0022] Where, T self_att represents the spatial features of the student video obtained by the self-attention module, K represents the number of image frames corresponding to each video, Represents the spatial features of the i-th image frame;

[0023] S23: Use the cross attention module to extract the feature weights between the frames of the student video sequence, assign higher weights to the facial image frames with obvious emotional features, and improve the accuracy of student emotion recognition; the cross attention module includes a 1×1 average pooling layer, a two-dimensional convolutional layer, and a sigmoid activation function. The calculation process is shown in the following formula:

[0024]

[0025] Where w is T self_att The weight vector obtained by the pooling layer and the convolution layer, is the feature obtained by the cross attention module for the i-th image frame, K represents the number of image frames contained in each video data, σ is the sigmoid activation function, is the obtained space vector;

[0026] S24: The spatial features extracted from each image frame of each student’s facial video are input into a bidirectional long short-term memory (BiLSTM) network to explore the influence of two directions between the sequence frames of the video and simulate the temporal changes of the students’ emotions; then the results of each image frame of the student are connected and linearly operated to obtain the final student’s facial spatiotemporal features.

[0027] Further, step S3 specifically includes the following steps:

[0028] S31: Input the student’s facial video image frame into the initial two-dimensional convolutional layer of the deep residual neural network to extract preliminary features:

[0029]

[0030] Where Y i,j is the (i,j)th pixel position of the output feature map Y, X i+m,j+n is the (i+m,j+n)th pixel position of the input feature map X, where M and N are the height and width of the feature map, respectively, and W m,n is the (m,n)th element of the weight matrix, b is the bias term, · represents the multiplication of the corresponding positions of the elements, G(·) represents the batch normalization process, and H(·) represents the nonlinear function ReLU;

[0031] S32: Perform a two-dimensional maximum pooling operation on the output of the initial convolutional layer, with a pooling size of 3×3 and a stride of (2,2) to further reduce the size of the feature map and retain important spatial information, effectively reducing the number of parameters while preventing overfitting and improving the computational efficiency of the model;

[0032] S33: The preliminary features are input into the residual network composed of 8 residual blocks to obtain personalized features that contain rich student characteristics:

[0033] Y p =F(x)+x

[0034] In the formula, F(x) represents the feature map after passing through two convolutional layers, and x represents the input of the residual block; the residual block contains two 3×3 convolutional layers, each of which is connected by batch normalization and linear rectification function ReLU;

[0035] S34: Global average pooling is used after the last residual block, and the average value of each feature map is used as the output to form a personalized feature vector of fixed length. Then, the personalized feature vector outputs of each image frame of the student’s facial performance video are connected to obtain the personalized features corresponding to the entire video of the student:

[0036]

[0037] Where M and N are the height and width of the feature map, respectively, K represents the number of image frames corresponding to each video, and y i,j is the element in the feature map, and p represents the obtained personalized feature;

[0038] S35: Use the fully connected layer to map personalized features to the target output space to obtain students’ deep personality features and explore more clues for students’ classroom emotion recognition;

[0039] P=W f p+bf

[0040] Where W f and b f are the weight and bias of the fully connected layer, p represents the input feature vector, and P represents the final personalized feature.

[0041] Further, step S4 specifically includes the following steps:

[0042] S41: concatenate the vectors of students’ facial spatiotemporal features and personalized features;

[0043] S42: The concatenated feature vectors are linearly transformed to form three matrices Q, K, and V. The transpose of Q and K is dot-producted to obtain the similarity matrix of Q and K. The dimension values ​​of Q and K are used to scale the similarity matrix values. The activation function softmax is used to normalize the similarity matrix values. After that, the similarity matrix is ​​weightedly summed with V to obtain the output, completing the preliminary interactive fusion of the two features of the students.

[0044] S43: Perform residual connection on the features after preliminary fusion and the features before fusion, and perform normalization operation on the connection result to complete subsequent feature interaction and transformation, so as to realize the fusion of the two features of the student;

[0045] S44: The fused feature vector is directly input into the fully connected layer, and the prediction results corresponding to different emotion categories are output by linear combination and nonlinear transformation of the features;

[0046] S45: Obtain the probability distribution corresponding to each student learning emotion category through the activation function, and set the threshold to determine the final emotion category prediction result.

[0047] Further, in step S5, a focal loss function based on binary cross entropy is used for model training.

[0048] The beneficial effects of the present invention are as follows: the present invention designs an emotion recognition method with personalized feature enhancement, which realizes emotion classification by mining students' personality features to assist facial features, thereby improving the accuracy of students' learning emotion recognition, and can effectively alleviate the problem of poor emotional communication between teachers and students in online teaching, help teachers grasp the classroom status of students in online teaching, and improve the quality of teaching implementation.

[0049] Specifically, due to the weak time-varying nature of students' facial expressions during class, the performance of facial expression feature extraction of traditional methods is limited. Facial recognition of student emotions requires capturing more abundant and effective student facial features. The present invention uses an attention mechanism to weightedly fuse features of different importance, and mines more important features in student facial image frames in space and time. Students' learning emotions have extremely strong differentiation characteristics. The present invention introduces psychological ideas and uses an improved deep residual network to mine personalized features contained in students' facial behavior data, and uses facial expression changes in videos to estimate students' personality characteristics, and further integrates them with facial features to more accurately identify students' true emotions.

[0050] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0052] Figure 1 A diagram of the architecture of an emotion recognition model with enhanced personalized features provided by the present invention;

[0053] Figure 2 This is a flow chart of the personalized feature enhanced emotion recognition method proposed in the present invention. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0055] See also Figure 1-2 , Figure 1 The figure shows the architecture of the personalized feature-enhanced emotion recognition model proposed by the present invention. The model consists of a student facial spatiotemporal feature extraction model, a student personalized personality feature extraction model, and a feature fusion and classification model to complete the student classroom emotion recognition. The relationship between the modules is shown in the figure. Figure 2shown.

[0056] The student facial spatiotemporal feature extraction model uses a deep residual network, attention mechanism and bidirectional long short-term memory network to extract effective student facial spatiotemporal features.

[0057] The personalized feature extraction model uses an improved deep residual neural network to mine the personalized characteristics of the personality contained in the facial expression videos of students during class.

[0058] The feature fusion and classification model integrates the unique emotional features of students' faces and the potential personalized emotional clues contained in their facial expressions. The model outputs the emotional category of students' facial video samples during class based on the fused features.

[0059] The embodiment of the present invention proposes an emotion recognition method enhanced by personalized features. On the basis of extracting the spatial features of students' facial expression images, a bidirectional long short-term memory network is used to capture the emotional associations of different image frames in the time dimension, and effective student facial spatiotemporal features are obtained. Based on the "Big Five Personality" theory of psychology, a deep residual network is used to mine the personalized personality features hidden in students' facial expressions, and merge them with facial features, so as to achieve student emotion classification for reference by teachers. The method mainly includes the following steps:

[0060] S1: Collect students’ facial video data during class and perform data preprocessing.

[0061] In step S1, the signal used for emotion recognition in this embodiment is the student's facial video data, which avoids the invasiveness of collecting physiological signals such as brain waves. Data preprocessing includes: for each small segment of student facial video data, extracting effective image frames for subsequent emotion analysis research, for each image frame, performing random cropping and normalization preprocessing operations to reduce computational complexity, and finally adjusting each student's facial image frame to a size of 224×224 and converting it into a tensor for subsequent feature extraction.

[0062] S2: Construct a student facial spatiotemporal feature extraction model, and use deep residual network, attention mechanism and bidirectional long short-term memory network to extract effective student facial spatiotemporal features.

[0063] In step S2, extracting effective student facial spatiotemporal features specifically includes the following steps:

[0064] First, the preprocessed 224×224 pixel student facial image is used as the input of the ResNet-152 network. The first convolution extracts features from the 3-channel student image in RGB format. The middle 150 convolution layers extract richer features. The last layer is an adaptive average pooling layer to obtain the visual features of the student facial image with a dimension of 1.

[0065] For each image frame of the student's facial video, a self-attention mechanism is used to assign a higher weight to the area of ​​each image that reflects the student's emotional information, so as to reduce the influence of irrelevant background information and other irrelevant information, and improve the accuracy under natural learning environment conditions. The spatial features of each image are input into the self-attention module composed of a fully connected layer and a sigmoid activation function, and the attention distribution of the student's single-frame image features is calculated to obtain the feature vector of the attention weight of the single frame.

[0066]

[0067] Where, T i represents the visual features of each image frame, w i Represents the spatial features T of the image frame i The attention weight vector obtained by the fully connected layer, σ represents the sigmoid activation function, and the image space features obtained by this module are connected to obtain the following expression:

[0068]

[0069] Where T self_att represents the spatial features of the student video obtained by the self-attention module, K represents the number of image frames corresponding to each video, Represents the spatial features of each image frame. self_att The input is sent to the cross-attention module to extract the feature weights between the frames of the student video sequence. Considering that different frames of the video data have different effects on the student emotion recognition of the video, some frames have more obvious emotional features. After the cross-attention mechanism, the features of the student facial expression image frame are given higher weights to improve the accuracy of student emotion recognition. The cross-attention module consists of a 1×1 average pooling layer, a two-dimensional convolutional layer, and a sigmoid activation function. The calculation process is shown in the following formula:

[0070]

[0071] Where w is T self_att The weight vector obtained by the pooling layer and the convolution layer, is the feature obtained by the cross attention module for the i-th image frame, K represents the number of image frames contained in each video data, σ is the sigmoid activation function, T wc is the obtained space vector.

[0072] The spatial features extracted from each image frame of each student's facial video are input into the BiLSTM network to explore the influence of two directions between the sequence frames of the video and simulate the changes in students' emotions over time. After that, the results of each image frame of the student are connected and linearly operated to obtain the final spatiotemporal features of the student video.

[0073] S3: Build a personalized feature extraction model and use the improved deep residual neural network to mine the personalized characteristics of the personality contained in the facial expression videos of students during class.

[0074] In step S3, the mining of students' deep personalized personality characteristics includes:

[0075] The student video image frames are input into the initial 2D convolutional layer with a stride of (2, 2) and a 7×7 layer for preliminary feature extraction:

[0076]

[0077] Where Y i,j is the (i,j)th pixel position of the output feature map Y, X i+m,j+n is the (i+m,j+n)th pixel position of the input feature map x, where M and N are the height and width of the feature map, respectively, and W m,n is the (m,n)th element of the weight matrix, b is the bias term, · represents the multiplication of the corresponding positions of the elements, G(·) represents the batch normalization process, and H(·) represents the nonlinear function ReLU;

[0078] A two-dimensional maximum pooling operation is performed on the output of the initial convolutional layer with a pooling size of 3×3 and a stride of (2, 2) to further reduce the size of the feature map and retain important spatial information, effectively reducing the number of parameters while preventing overfitting and improving the computational efficiency of the model.

[0079] Next, the processed feature map is input into the core module of the network, a residual network composed of 8 residual blocks, to obtain personalized features that contain rich student personality. The residual block contains two 3×3 convolutional layers, each of which is connected through batch normalization and linear rectification function ReLU. The connection method of the residual block alleviates the gradient vanishing problem in the deep network, allowing the deep neural network to capture more subtle changes in students' facial expressions to reflect the students' personality characteristics and emotional cognitive style. The batch normalization layer can normalize the student's image frame input to improve the speed and effect of model training. The linear rectification function, as a nonlinear activation function, introduces nonlinear characteristics to the network, allowing it to learn more complex nonlinear relationships in the data set, which is conducive to the model extracting richer and more effective personalized features of students.

[0080] After the last residual block, global average pooling is used to take the average value of each feature map as output to form a personalized feature vector of fixed length. Then, the personalized feature vector outputs of each image frame of the student’s facial performance video are connected to obtain the personalized feature p corresponding to the entire video of the student. The calculation process is expressed as:

[0081]

[0082] Where M and N are the height and width of the feature map, respectively, K represents the number of image frames corresponding to each video, and y i,j Personalize the elements in the feature map for your students.

[0083] Finally, the feature vector is input into the fully connected layer to map the personalized features to the target output space to obtain the students’ deep personality characteristics and explore more clues for students’ classroom emotion recognition.

[0084] S4: Construct a feature fusion and classification model to integrate students’ facial spatiotemporal features and personalized features. The model outputs the emotion category of students’ facial video samples during class based on the fused features.

[0085] In step S4, student feature fusion and emotion classification include:

[0086] The spliced ​​features of the student's facial spatiotemporal features and the student's personalized features are input into the Transformer encoder, and three matrices Q, K, and V are formed after linear transformation. In the multi-head attention mechanism, the transpose of Q and K is dot-producted to obtain the similarity matrix of Q and K, and the similarity matrix is ​​numerically normalized using the activation function softmax. After that, the similarity matrix is ​​weightedly summed with V to obtain the output of each attention head, completing the initial interactive fusion of the two features of the student:

[0087]

[0088] The residual connection plus normalization module and the feedforward network complete the subsequent feature interaction and transformation to achieve the fusion of the two student features.

[0089] The fused feature vector is directly input into the fully connected layer, and the prediction results corresponding to different emotion categories are output by linearly combining and nonlinearly transforming the features. After the calculation of the fully connected layer, in order to obtain the probability distribution corresponding to each student learning emotion category, the present invention adopts the softmax activation function. The softmax activation function converts the output of the fully connected layer into a probability value, so that each output value is between 0 and 1, and the sum of all output values ​​is 1.

[0090] S5: Train the classification model until the model converges, so that the model can achieve emotion recognition results with high accuracy.

[0091] In order to solve the problem of imbalanced distribution of category samples in data sets, the loss function Focal Loss is used to alleviate it. The intensity of the emotion to be identified for each sample belongs to a multi-classification task. The loss function of the training model on this task is:

[0092]

[0093] Where CE(p,y) represents the cross entropy loss function, p t And FL(p,y) is calculated as:

[0094]

[0095]

[0096] In the formula, y represents the true emotion category label of the student video data, p is the probability that the student emotion label predicted by the model is y = 1, and α t is a balancing factor used to reduce the impact of class imbalance. (1-p t ) γ is the adjustment factor of the cross entropy, and γ is an adjustable focus parameter that determines the speed at which the weight of easily classified samples decreases. It is used to reduce the relative loss of well-identified student video samples and increase the relative loss of poorly classified student video samples, helping the model to pay more attention to minority categories or samples that are more difficult to correctly classify, so that as many student video samples as possible can be accurately identified.

[0097] Each sample may have two or more emotions at the same time. The loss function of the multi-label classification model is:

[0098]

[0099] Where N represents the volume of all possible samples in the class feature space, n y is the number of samples of category y, the present invention sets β=0.9, γ=2.0, and BCELoss(p, y) represents the binary cross entropy loss function.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A personalized feature enhanced emotion recognition method, characterized in that: The method specifically comprises the following steps: S1: Collect students’ facial video data during class and perform data preprocessing; S2: Construct a student facial spatiotemporal feature extraction model, and use deep residual network, attention mechanism and bidirectional long short-term memory network to extract effective student facial spatiotemporal features; S3: Build a personalized feature extraction model and use the improved deep residual neural network to mine the personalized characteristics of the personality contained in the facial expression videos of students during class; S4: Construct a feature fusion and classification model to integrate students’ facial spatiotemporal features and personalized features. The model outputs the emotion category of students’ facial video samples during class based on the fused features. S5: Train the classification model until the model converges, so that the model can achieve high-accuracy emotion recognition results.

2. The emotion recognition method with personalized feature enhancement according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: using a high-definition camera to collect video data of students’ facial expressions during class, and extracting valid image frames for each segment of student facial video data; S12: Perform random cropping and normalization preprocessing operations on each image frame, resize each facial image frame of the student to a size of 224×224 and convert it into a tensor.

3. The emotion recognition method with personalized feature enhancement according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21: Input the preprocessed student facial image into the deep residual network, use the adaptive average pooling layer to control the output dimension to 1, and obtain the visual features of the student facial image with a dimension of 1; S22: Input the visual features of each image into the self-attention module composed of a fully connected layer and a sigmoid activation function, calculate the attention distribution of the student's single-frame image features, and obtain the feature vector of the single-frame attention weight Where, T i represents the spatial features of each image frame, w i Represents the spatial features T of the image frame i The attention weight vector obtained by the fully connected layer, σ represents the sigmoid activation function, and the image space features obtained by the self-attention module are connected to obtain the following expression: Where, T self_att represents the spatial features of the student video obtained by the self-attention module, K represents the number of image frames corresponding to each video, Represents the spatial features of the i-th image frame; S23: Use the cross-attention module to extract the feature weights between the frames of the student video sequence; the cross-attention module includes a 1×1 average pooling layer, a two-dimensional convolution layer, and a sigmoid activation function. The calculation process is shown in the following formula: Where w is T self_att The weight vector obtained by the pooling layer and the convolution layer, is the feature obtained by the cross attention module for the i-th image frame, σ is the sigmoid activation function, is the obtained space vector; S24: The spatial features extracted from each image frame of each student's facial video are input into a bidirectional long short-term memory network to explore the influence of two directions between the sequence frames of the video and simulate the temporal changes of students' emotions; then the results of each image frame of the student are connected and linearly operated to obtain the final student's facial spatiotemporal features.

4. The emotion recognition method with personalized feature enhancement according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31: Input the student’s facial video image frame into the initial two-dimensional convolutional layer of the deep residual neural network to extract preliminary features: Where Y i,j is the (i,j)th pixel position of the output feature map Y, X i+m,j+n is the (i+m,j+n)th pixel position of the input feature map X, where M and N are the height and width of the feature map, respectively, and W m,n is the (m,n)th element of the weight matrix, b is the bias term, · represents the multiplication of the corresponding positions of the elements, G(·) represents the batch normalization process, and H(·) represents the nonlinear function ReLU; S32: Perform a two-dimensional maximum pooling operation on the output of the initial convolutional layer, with a pooling size of 3×3 and a stride of (2,2); S33: The preliminary features are input into the residual network composed of 8 residual blocks to obtain personalized features that contain rich student characteristics: Y p =F(x)+x In the formula, F(x) represents the feature map after passing through two convolutional layers, and x represents the input of the residual block; the residual block contains two 3×3 convolutional layers, each of which is connected by batch normalization and linear rectification function ReLU; S34: Global average pooling is used after the last residual block, and the average value of each feature map is used as the output to form a personalized feature vector of fixed length. Then, the personalized feature vector outputs of each image frame of the student’s facial performance video are connected to obtain the personalized features corresponding to the entire video of the student: Where M and N are the height and width of the feature map, respectively, K represents the number of image frames corresponding to each video, and y i,j is the element in the feature map, and p represents the obtained personalized feature; S35: Use the fully connected layer to map personalized features to the target output space to obtain students’ deep personality features and explore more clues for students’ classroom emotion recognition; P=W f p+b f Where W f and b f are the weight and bias of the fully connected layer respectively, and P represents the final personalized features.

5. The emotion recognition method with personalized feature enhancement according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41: concatenate the vectors of students’ facial spatiotemporal features and personalized features; S42: The concatenated feature vectors are linearly transformed to form three matrices Q, K, and V. The transpose of Q and K is dot-producted to obtain the similarity matrix of Q and K. The dimension values ​​of Q and K are used to scale the similarity matrix values. The activation function softmax is used to normalize the similarity matrix values. After that, the similarity matrix is ​​weightedly summed with V to obtain the output, completing the preliminary interactive fusion of the two features of the students. S43: Perform residual connection on the features after preliminary fusion and the features before fusion, and perform normalization operation on the connection result to complete subsequent feature interaction and transformation, so as to realize the fusion of the two features of the student; S44: The fused feature vector is directly input into the fully connected layer, and the prediction results corresponding to different emotion categories are output by linear combination and nonlinear transformation of the features; S45: Obtain the probability distribution corresponding to each student learning emotion category through the activation function, and set the threshold to determine the final emotion category prediction result.

6. The emotion recognition method with personalized feature enhancement according to claim 1, characterized in that: In step S5, the focal loss function based on binary cross entropy is used for model training.

Images