Emo-ResNet-based facial expression recognition method, device, equipment and medium

Abstract

The application discloses a facial expression recognition method and device based on Emo-ResNet, an improved residual network, and a multi-head self-attention mechanism, and introduces a loss function into a deep attention center loss and applies it to daily facial expression recognition. The application first acquires a real-time dynamic video stream of a person to be recognized, performs frame extraction, and saves the stream as a static picture stream; then recognizes a face from the picture to be recognized, corrects and aligns the face, and cuts the face to obtain a corresponding face picture; then inputs the face picture into an Emo-ResNet network for classification training; and finally outputs the determined facial expression category and the time of the facial expression in the real-time dynamic video on an interface, and visualizes the recognition result. Experiments prove that the method has low calculation complexity, high recognition rate, and is more suitable for facial expression recognition.

Description

Technical Field

[0001] This invention relates to the field of facial expression recognition technology, specifically to a facial expression recognition method, apparatus, device, and medium based on Emo-ResNet. Background Technology

[0002] Most current facial expression recognition systems are based on deep learning. However, many facial expression recognition systems are based on datasets from laboratory environments, meaning that the subjects' expressions are deliberately made and are quite obvious, rather than natural. In contrast, most facial expressions in people are natural.

[0003] Convolutional neural networks (CNNs) have certain advantages in feature extraction, and most current deep learning-based facial expression recognition methods are based on CNNs. CNNs use convolution operations to extract features from facial images, learning low-level features such as color and edges in shallow layers, more complex features like texture in intermediate layers, and key, discriminative features in higher layers. However, ordinary CNNs can only learn local features and cannot learn features in the overall spatial domain of the image, lacking contextual information. Furthermore, excessively high layers in a CNN can lead to vanishing or exploding gradients, meaning ordinary CNNs cannot effectively suppress factors affecting facial expression recognition. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned deficiencies in the prior art and provide a facial expression recognition method, apparatus, device, and medium based on Emo-ResNet.

[0005] The first objective of this invention is to provide a facial expression recognition method based on Emo-ResNet, the facial expression recognition method comprising:

[0006] S1. Video capture step: Obtain the real-time dynamic video stream of the person to be identified;

[0007] S2, Data transmission step: The real-time dynamic video stream is subjected to frame interval extraction and saved as a static image stream to obtain an image including the face of the person to be identified.

[0008] S3, Face detection and extraction step: Identify faces from images containing the faces of the people to be identified, perform face correction and face alignment, and then crop to obtain face images of each person to be identified;

[0009] S4. Facial Expression Recognition Step: Input the face image of each person to be recognized into the trained Emo-ResNet network to obtain the probability of each person's facial expression category. Take the facial expression corresponding to the highest probability as the facial expression recognition result of the person to be recognized. The main framework of the Emo-ResNet network is an improved ResNet50. In the first bottleneck layer of the fourth convolutional block, the multi-head self-attention mechanism MHSA (Multi-Head Self-Attention) is used to replace the corresponding convolutional layer and normalized network layer. At the same time, the deep attention center loss is used as the loss function.

[0010] S5. Visualization of recognition results: The recognized facial expression category and the time of the facial expression in the real-time dynamic video are output on the display device, thus visualizing the recognition results.

[0011] Furthermore, the video acquisition step includes obtaining a dynamic video stream of the person to be identified, which can be a real-time online screen recording or a video transmitted in real-time from an offline camera, allowing for real-time monitoring of the other party's emotions; or it can be a pre-recorded video. Since this technical solution is applicable to daily life, the materials used need to be natural, everyday videos to make the invention more authentic and applicable.

[0012] Furthermore, in the data transmission step, a frame extraction operation is performed on the real-time dynamic video stream. The dynamic video stream is extracted at pre-specified short time intervals and saved as a static image stream of the subject to be identified. The frame extraction operation divides a continuous video stream into several images at the same time interval, typically between 0.1s and 0.5s. Too short an interval can lead to an excessive number of input images and system lag, while too long an interval can cause the omission of important facial details. This operation provides more suitable input for the subsequent face detection and extraction steps and the recognition network, thus improving the recognition efficiency of the invention.

[0013] Furthermore, in the face detection and extraction step, faces are detected in each frame of the static image stream. A multi-task convolutional neural network (MTCNN) is used to extract face regions from the static images, and multiple regions of interest (ROIs) for each face are extracted simultaneously. These ROIs include the eye region, eyebrow region, mouth region, and facial contour region. The center point of the face is determined based on the eye region, and face correction is performed on the entire face based on the center point. Then, the face image is aligned relative to a Cartesian coordinate system to achieve face alignment. Finally, the image is cropped to obtain a face image of each subject after face correction and alignment. The multi-task convolutional neural network in the above steps uses a cascaded approach, filtering out face regions that meet the standards step by step. It decomposes a large network into three smaller networks, adding two more confidence loss functions (confidence loss function, face position offset loss function, and facial feature offset loss function) compared to a single network, doubling the number of loss functions and reducing the overall number of parameters, resulting in faster computation. Furthermore, multi-task convolutional neural networks use smaller convolutional kernels instead of larger ones to maintain the same receptive field, resulting in fewer parameters and faster computation. This makes it easier to extract key points and regions of interest in facial regions and also speeds up the preprocessing of datasets.

[0014] Among them, the multitask convolutional neural network comes from the literature "Zhang K, Zhang Z, Li Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J].IEEE signal processing letters,2016,23(10):1499-1503.", which belongs to the existing technology.

[0015] The face detection and extraction process is as follows:

[0016] S31. Use OpenCV's built-in face detection model to detect faces in each frame of a static image stream; the OpenCV built-in face detection model is from "Khan M, Chakraborty S, Astya R, et al. Face detection and recognition using OpenCV[C] / / 2019 International Conference on Computing, Communication, and Intelligent Systems(ICCCIS).IEEE,2019:116-119."

[0017] S32. Use a multi-task convolutional neural network to extract face regions from the detected images, and extract multiple regions of interest for the corresponding faces at the same time.

[0018] S33. Based on the binocular regions, determine the location of the center point of the face, and perform face correction and alignment on the entire face based on the center point. The center position of the left eye in the binocular regions is set as (x... left_i ,y left_i The center position of the right eye is set to (x). right_i ,y right_i The tilt angle θ of the sample is calculated using equation (1), which is:

[0019]

[0020] S34, Move the center position of the left eye (x) left_i ,y left_i ) and the center position of the right eye (x right_i ,y right_i Using the center point of the image as the center, perform an affine transformation on the image to obtain the corresponding frontal face image. The formula for the affine transformation is shown in equation (2):

[0021]

[0022] in, The input sample is transformed into... Offset is A is the affine matrix related to the tilt angle θ, as follows:

[0023]

[0024] S35. Cropping yields a face image of each person to be identified, after face correction and alignment.

[0025] Furthermore, the Emo-ResNet network is specifically as follows:

[0026] The input to the Emo-ResNet network is a face image of each subject after face correction and alignment. The Emo-ResNet network, from input to output, consists of sequentially connected convolutional layers (Conv 1), the first module, the second module, the third module, the fourth module, a pooling layer (Avgpool), a fully connected layer (FC), and a softmax module. Convolutional layers (Conv 1) have a kernel size of 3, a stride of 1, and a padding width of 1. The basic units of the first, second, third, and fourth modules are bottleneck layers. Each bottleneck layer has two branches. The first branch consists of sequentially connected convolutional layers (Conv 01, BN01, RELU activation function), convolutional layers (Conv 02, BN02, RELU activation function), convolutional layers (Conv 03, BN03), and so on. The second branch consists of sequentially connected convolutional layers (Conv 01, BN01, RELU activation function), and so on. 04. The normalized network layer BN04 adds the outputs of the first and second branches and passes them through the ReLU activation function to obtain the bottleneck layer output. The number of bottleneck layers in the first, second, third, and fourth modules are 3, 4, 6, and 3, respectively. In the fourth module, the first bottleneck layer replaces the convolutional layer Conv04 and the normalized network layer BN04 with a multi-head self-attention mechanism (MHSA). In the first, second, and fourth modules, except for the first bottleneck layer, the second branches of the other bottleneck layers do not have convolutional layers Conv04 or normalized network layers BN04. The main framework of the Emo-ResNet network is an improvement based on ResNet50, specifically based on residual networks. Residual networks are a type of convolutional neural network, characterized by ease of optimization. This invention increases network depth and embeds an attention mechanism to improve accuracy to some extent. Furthermore, the fact that the second branches of the bottleneck layers do not have convolutional or normalized network layers alleviates the network degradation problems (such as vanishing gradients and gradient degeneration) caused by increasing network depth. Residual networks use the identity mapping of the bottleneck layer to pass the current output to the next layer, and the direct connections (shortcuts) do not introduce additional parameters or increase computational complexity. Meanwhile, the normalized network layers used in the network effectively prevent problems such as overfitting and vanishing gradients. The fully connected layers of the residual network act as a "classifier" for the learned "feature representations."

[0027] In the Multi-Head Self-Attention (MHSA) mechanism, each attention convergence is called a head, and the number of heads is set to 4. The width and height of the two-dimensional feature map are both set to 14. The calculation formula for MHSA is as follows:

[0028] MHSA(Q,K,V)=Concat(head1,head2,...,head n W 0 ,

[0029] The formula for calculating Attention is: Where n is the number of heads, Q refers to the query vector matrix, K refers to the "queried" vector matrix, Q and K are weight matrices used to calculate attention, V is the weight matrix used to sum the attention values ​​to obtain the final result, and dk is the dimension of matrix K. Let Through weights A linear transformation is performed, ultimately resulting in a learnable weight matrix W. 0 Multiple heads obtained by Attention n The concatenation results in the final multi-head self-attention result MHSA(Q,K,V), calculated using Q, K, and V, which is used to extract features from the image. Contact(head1,head2,...,head) n This indicates that head1, head2, ..., head... n To perform concatenation, use Concat(head1, head2, ..., head...). n W 0 This indicates that head1, head2, ..., head n After concatenation, it is then combined with a learnable weight matrix W. o A linear transformation is performed to obtain the final attention output. Using a multi-head self-attention mechanism not only improves computational efficiency but also allows the output of the attention layer to contain encoded representation information from different subspaces, thereby enhancing the model's expressive power.

[0030] Furthermore, the loss function of the Emo-ResNet network is a deep attention center loss, derived from "Farzaneh AH, Qi X. Facial expression recognition in the wildvia deep attentive center loss[C] / / Proceedings of the IEEE / CVF winterconference on applications of computer vision.2021:2402-2411." The deep attentive center loss adaptively selects a set of important feature elements for enhanced recognition. It integrates an attention mechanism, using the spatial feature map generated in the convolutional neural network as a background, to predict attention weights related to feature importance. The predicted weights adapt to the sparse formula of the center loss, selectively achieving intra-class compactness and inter-class separation for relevant information in the embedding space. The facial expression categories include anger, disgust, fear, happiness, sadness, surprise, and neutrality.

[0031] Furthermore, the training process of the Emo-ResNet network is as follows:

[0032] Face detection extraction was performed using the FER2013 dataset. Deep learning requires sufficient training data to ensure generality for a given recognition task. However, most publicly available facial expression recognition databases do not have a sufficient number of images for training. Therefore, data augmentation is an important step in deep learning research. This invention randomly crops the input samples from the center and four corners of face images after face correction and alignment, then horizontally flips and rotates them, resulting in a dataset ten times larger than the original training data. The augmented face images are then merged with the original dataset to form a new training set. The combination of multiple operations can generate more training samples, making the network more robust to face deviations and rotations, further expanding the size and diversity of the data, and alleviating overfitting.

[0033] Next, the new training set is used to iteratively train the Emo-ResNet network until the loss function converges, resulting in a trained recognition network model. The iterative function during training is the deep attention center loss function, the scheduler is a cosine scheduler, and the optimizer is a stochastic gradient descent (SGD) optimizer. The base learning rate is set to 0.01, the batch size is set to 32, and the learning rate is reduced when the loss function no longer improves after 30 small samples. The maximum number of training iterations is 300, and the evaluation metric is accuracy. After continuous fine-tuning of the network and comparing accuracy, the optimal trained network model is finally obtained.

[0034] Furthermore, the recognition result visualization step in step S5 combines the result determined by the facial expression recognition step with the corresponding time of the image stream transmitted from the data transmission step in the video stream, and outputs it on the front-end interface to visualize the recognition result, so that the expression of the person being recognized can be fed back in a timely manner.

[0035] A second objective of this invention is to provide a facial expression recognition device based on Emo-ResNet, the facial expression recognition device comprising:

[0036] The video capture module acquires the real-time dynamic video stream of the person to be identified;

[0037] The data transmission module extracts the frame intervals of the real-time dynamic video stream and saves it as a static image stream, acquiring an image including the face of the person to be identified.

[0038] The face detection and extraction module identifies the face of the person to be identified from the image containing the face of the person to be identified, performs face correction and face alignment, and then crops out the face image of each person to be identified;

[0039] The facial expression recognition module inputs the face image of each person to be recognized into a trained Emo-ResNet network to obtain the probability of each person's facial expression category. The facial expression with the highest probability is taken as the facial expression recognition result of the person to be recognized. The facial expression categories include anger, disgust, fear, happiness, sadness, surprise, and neutral. The main framework of the Emo-ResNet network is an improved ResNet50. In the first bottleneck layer of the fourth convolutional block, the multi-head self-attention mechanism MHSA is replaced with the corresponding convolutional layer and normalized network layer. At the same time, the deep attention center loss is used as the loss function.

[0040] The recognition result visualization module outputs the determined facial expression category and the time of the facial expression in the real-time dynamic video on the display device, thus visualizing the recognition result.

[0041] A third objective of this invention is to provide a computer device including a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements a facial expression recognition method based on Emo-ResNet.

[0042] The fourth objective of this invention is to provide a storage medium storing a program that, when executed by a processor, implements a facial expression recognition method based on Emo-ResNet.

[0043] The present invention has the following advantages and effects compared with the prior art:

[0044] (1) The dataset used for training in this invention is the facial expression dataset FER2013 in a natural environment. The recognition effect trained by this dataset is more suitable for facial expression recognition in real-world scenarios than many current facial expression datasets based on laboratory environments.

[0045] (2) This invention utilizes the advantages of residual networks, such as ease of optimization, the ability to increase network depth or embed an attention mechanism to improve accuracy, the skip connections in the residual blocks to alleviate network degradation (such as vanishing gradients and gradient degeneration) caused by increasing network depth, and the fact that fully connected layers act as classifiers for the learned "feature representations". It overcomes the limitations of ordinary convolutional neural networks, which can only learn local features and cannot learn the spatial features of images, lacking contextual information. Furthermore, excessively high layers in convolutional neural networks can lead to problems such as vanishing or exploding gradients.

[0046] (3) This invention integrates the advantages of residual networks, multi-head self-attention mechanism, and deep attention center loss[1] to achieve higher accuracy in classifying and recognizing facial expressions than simple residual networks;

[0047] (4) The present invention can be further expanded in application scenarios. It can be applied to recognize the other party’s facial expressions in various scenarios. Moreover, it is not limited to recognizing image input, but can also recognize video input in real time and provide feedback. It has stronger applicability in application. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0049] Figure 1 This is a flowchart of a facial expression recognition method based on Emo-ResNet disclosed in this invention;

[0050] Figure 2 This is a schematic diagram of the structure of the Emo-ResNet network recognition model in this invention;

[0051] Figure 3 This is a schematic diagram of the bottleneck layer in this invention;

[0052] Figure 4 This is a diagram showing the confusion matrix structure of facial expression classification on the test set of the FER2013 dataset, provided by the method of this invention.

[0053] Figure 5 This is a diagram showing the confusion matrix structure of facial expression classification on the test set of the RAF-DB dataset, provided by the method of this invention.

[0054] Figure 6 This is a structural block diagram of the ultrasound image processing device in Embodiment 4 of the present invention;

[0055] Figure 7 This is a structural block diagram of the computer device in Embodiment 5 of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1

[0058] Currently, many facial expression recognition systems are based on laboratory datasets, meaning the subjects' expressions are intentionally exaggerated and not natural. In contrast, most facial expressions, especially micro-expressions, are natural in most environments. Furthermore, ordinary convolutional neural networks can only learn local features and cannot learn the spatial features of the image, lacking contextual information. Also, if the number of layers in a convolutional neural network is too high, it can cause problems such as vanishing or exploding gradients. Therefore, ordinary convolutional neural networks cannot effectively suppress factors affecting facial expression recognition. Additionally, while simple residual networks extract image features better than ordinary convolutional neural networks, they can be further improved by embedding attention mechanisms and modifying the loss function, combining the advantages of both to achieve more accurate classification and recognition of facial expressions.

[0059] This embodiment provides a facial expression recognition method based on Emo-ResNet, including the following steps:

[0060] S1. Video capture step: The acquired video stream includes the dynamic video stream of the person to be identified, which can be a real-time online screen recording or a video transmitted in real-time from an offline camera, allowing us to grasp the other party's real-time emotions; or it can be a pre-recorded video.

[0061] S2, Data transmission step: Frame extraction operation is performed on the real-time dynamic video stream. The dynamic video stream is extracted at intervals according to the short time interval specified in advance and then saved as a static image stream of the person to be identified.

[0062] S3. Face detection and extraction step: Detect faces in each frame of the image including the face of the person to be identified. Use a multi-task convolutional neural network (MTCNN) (Zhang K, Zhang Z, Li Z, et al. Joint face detection and alignment using multitaskcascaded convolutional networks[J].IEEE signal processing letters,2016,23(10):1499-1503.) to extract face regions from the image. Simultaneously extract multiple regions of interest (ROIs) for the corresponding faces. The ROIs include the eye region, eyebrow region, mouth region, and facial contour region. Determine the center point of the face based on the eye region, and perform face correction on the entire face based on the center point. Then, align the face image with the Cartesian coordinate system to achieve face alignment. Finally, crop to obtain the face image of each person to be identified after face correction and face alignment. The process is as follows:

[0063] S31. Detect faces in each frame of a static image stream using the built-in face detection model of OpenCV (Khan M, Chakraborty S, Astya R, et al. Face detection and recognition using OpenCV[C] / / 2019 International Conference on Computing, Communication, and Intelligent Systems(ICCCIS).IEEE,2019:116-119.); S32. Extract face regions using a multi-task convolutional neural network, and simultaneously extract regions of interest for multiple corresponding faces.

[0064] S33. Based on the binocular regions, determine the location of the center point of the face, and perform face correction and alignment on the entire face based on the center point. The center position of the left eye in the binocular regions is set as (x... left_i ,y left_i The center position of the right eye is set to (x). right_i ,y right_i The tilt angle θ of the sample is calculated using equation (1), which is:

[0065] S34, Move the center position of the left eye (x) left_i ,y left_i ) and the center position of the right eye (x right_i ,y right_i Using the center point of the image as the center, perform an affine transformation on the image to obtain the corresponding frontal face image. The formula for the affine transformation is shown in equation (2):

[0066] in, The input sample is transformed into... Offset is A is the affine matrix related to the tilt angle θ, as follows:

[0067] S35. Cropping yields a face image of each person to be identified, after face correction and alignment.

[0068] S4. Facial Expression Recognition Step: Input each face image of the person to be recognized into the trained Emo-ResNet network to obtain the probability of each facial expression category for each person to be recognized. The facial expression corresponding to the highest probability is taken as the facial expression recognition result of the person to be recognized. The main framework of the Emo-ResNet network is an improved ResNet50, and the multi-head self-attention mechanism (MHSA) is replaced in the fourth convolutional block. At the same time, the deep attentive center loss (from Farzaneh AH, Qi X. Facial expression recognition in the wildvia deep attentive center loss[C] / / Proceedings of the IEEE / CVF winterconference on applications of computer vision.2021:2402-2411.) is used as the loss function. Facial expression categories include anger, disgust, fear, happiness, sadness, surprise, and neutral.

[0069] The Emo-ResNet network is as follows:

[0070] The input to the Emo-ResNet network is a face image of each subject after face correction and alignment. The Emo-ResNet network, from input to output, consists of sequentially connected convolutional layers (Conv1), a first module, a second module, a third module, a fourth module, a pooling layer (Avgpool), a fully connected layer (FC), and a softmax module. Convolutional layer Conv1 has a kernel size of 3, a stride of 1, and a padding width of 1. The basic units of the first, second, third, and fourth modules are all bottleneck layers. Each bottleneck layer has two branches. The first branch consists of sequentially connected convolutional layers (Conv01, BN01, RELU activation function, Conv02, BN02, RELU activation function, Conv03, and BN03). The second branch consists of sequentially connected convolutional layers (Conv01, BN01, RELU activation function, Conv02, BN02, RELU activation function, Conv03, and BN03). 04. The normalized network layer BN04 adds the outputs of the first and second branches and passes them through the activation function ReLU to obtain the output of the bottleneck layer. The number of bottleneck layers in the first, second, third, and fourth modules are 3, 4, 6, and 3, respectively. In the fourth module, the first bottleneck layer is replaced by the multi-head self-attention mechanism MHSA, which replaces the convolutional layer Conv04 and the normalized network layer BN04. In the first, second, third, and fourth modules, except for the first bottleneck layer, the second branches of the other bottleneck layers do not have the convolutional layer Conv04 and the normalized network layer BN04.

[0071] In this embodiment, each attention convergence in the Multi-Head Self-Attention (MHSA) mechanism is called a head, and the number of heads is set to 4. The width and height of the two-dimensional feature map are both set to 14. The calculation formula for the MHSA mechanism is expressed as follows:

[0072] MHSA(Q,K,V)=Concat(head1,head2,...,head n W 0 ,

[0073] The formula for calculating Attention is: Where n is the number of heads, Q refers to the query vector matrix, K refers to the "queried" vector matrix, Q and K are weight matrices used to calculate attention, V is the weight matrix used to sum the attention values ​​to obtain the final result, and d k Let K be the dimension of the matrix, and let... Through weights A linear transformation is performed, ultimately resulting in a learnable weight matrix W. 0 Multiple heads obtained by Attentionn The concatenation results in the final multi-head self-attention result MHSA(Q,K,V), calculated using Q, K, and V, which is used to extract features from the image. Contact(head1,head2,...,head) n This indicates that head1, head2, ..., head... n To perform concatenation, use Concat(head1, head2, ..., head...). n W 0 This indicates that head1, head2, ..., head n After concatenation, it is then combined with a learnable weight matrix W. o Perform a linear transformation to obtain the final Attention output.

[0074] The training process of the Emo-ResNet network is as follows:

[0075] S41. Perform face detection step extraction operation on the FER2013 dataset;

[0076] S42. Perform data augmentation such as flipping, rotating, and cutting on the face images after face correction and face alignment, and merge the data-augmented face images with the original dataset to form a new training set.

[0077] S43. Iteratively train the new training set on the Emo-ResNet network until the loss function converges, obtaining the trained recognition network model. The iterative function during training is the deep attention center loss function, the scheduler is a cosine scheduler, the optimizer is a stochastic gradient descent optimizer (SGD), the base learning rate is set to 0.01, the batch size is set to 32, and the learning rate is reduced when the loss function no longer improves after every 30 small samples. The maximum number of training iterations is 300, and the evaluation metric is accuracy.

[0078] S44. After continuous fine-tuning of the network and combining it with the accuracy comparison, the best trained network model is finally obtained.

[0079] S5. Visualization of Recognition Results: The results determined by the facial expression recognition module (seven categories: anger, disgust, fear, happiness, sadness, surprise, or neutral) are combined with the corresponding time of the image stream transmitted from the data processing module in the video stream and output on the front-end interface module, making the recognition results visible and providing timely feedback on the emotions of the person being recognized.

[0080] Example 2

[0081] In this embodiment, the experimental environment is based on PyTorch 1.7 and CUDA 10.2, using an NVIDIA Corporation GP104GL [Tesla P4] GPU (Graphics Processing Unit) with 8GB of memory, and the experimental development language is Python.

[0082] The hyperparameter settings for the experiment were as follows: batch size was set to 32, meaning 32 images were input into the network for each training iteration; the learning rate was set to 0.01; the scheduler was Cosine; the optimizer was Stochastic Gradient Descent (SGD); the learning rate was reduced if the loss function no longer improved after 30 small sample sets; and the number of iterations was set to 300. The facial expression recognition method described above was implemented, and the performance of the facial expression recognition method of this invention was evaluated using the Fer2013 public dataset.

[0083] This embodiment conducts ablation experiments on the accuracy of the Fer2013 public dataset—while keeping the corresponding parameters the same, the proposed model is compared with the original ResNet50, the improved ResNet50, the improved ResNet50+MHSA, the improved ResNet50+DACL, the improved ResNet50+MHSA+Softmax Loss, and the improved ResNet50+MHSA+Center Loss; at the same time, comparative experiments are conducted—while keeping the corresponding parameters the same, the proposed model is compared with common facial expression recognition algorithms such as GoogleNet, VGG+SVM, ResNet+CBAM, DNN, CNN+SVM, Inception, and DenseNet.

[0084] Specifically, the average results obtained by the method of this invention using the model proposed in this patent on the FER2013 dataset were compared with the accuracy of the original ResNet50, the improved ResNet50, the improved ResNet50+MHSA, the improved ResNet50+DACL, the improved ResNet50+MHSA+Softmax Loss, and the improved ResNet50+MHSA+Center Loss in ablation experiments. The results are shown in Table 1.

[0085] Table 1. Comparison of ablation experimental results on the FER2013 dataset

[0086] method Accuracy ACC ResNet50 72.10％ Improved ResNet50 72.58％ Improved ResNet50+MHSA 72.88％ Improved ResNet50+DACL 72.96％ Improved ResNet50+MHSA+Softmax Loss 72.93％ Improved ResNet50+MHSA+Center Loss 73.29％ Emo-ResNet (the method of this invention) 73.64％

[0087] As shown in Table 1, the facial expression recognition method constructed in this invention has higher accuracy than the original ResNet50, the improved ResNet50, the improved ResNet50+MHSA, the improved ResNet50+DACL, the improved ResNet50+MHSA+SoftmaxLoss, and the improved ResNet50+MHSA+Center Loss.

[0088] Specifically, the average results obtained by the method of this invention using the model proposed in this patent on the FER2013 dataset were compared with common facial expression recognition algorithms such as GoogleNet, VGG+SVM, ResNet+CBAM, DNN, CNN+SVM, Inception, and DenseNet. The results are shown in Table 2.

[0089] Table 2. Comparison of experimental results on the FER2013 dataset

[0090]

[0091]

[0092] As shown in Table 2, the facial expression recognition method constructed in this invention has a higher accuracy than common facial expression recognition algorithms such as GoogleNet, VGG+SVM, ResNet+CBAM, DNN, CNN+SVM, Inception, and DenseNet.

[0093] The confusion matrix generated by the method of this invention is as follows: Figure 5 As shown, the accuracy rate for recognizing the four expressions of happiness, surprise, disgust, and naturalness is relatively high, especially for happiness, where the accuracy rate is generally above 90%.

[0094] This embodiment constructs a facial expression recognition method for subjects based on Emo-ResNet. Combining the advantages of residual networks, MHSA, and DACL, it achieves high accuracy in real-time classification and recognition of facial expressions. Through ablation experiments and comparative experiments, it is demonstrated that the algorithm proposed in this embodiment is more suitable for facial expression recognition, and it also provides a method for constructing a facial expression recognition system for subjects.

[0095] Example 3

[0096] The experimental environment in this embodiment is based on PyTorch 1.7 and CUDA 10.2, using an NVIDIA Corporation GP104GL [Tesla P4] GPU (Graphics Processing Unit) with 8GB of memory, and the development language of the experiment is Python.

[0097] The hyperparameter settings for the experiment were similar to those in Example 1. The facial expression recognition method described above was implemented, and the performance of the facial expression recognition method of the present invention was evaluated using the RAF-DB public dataset.

[0098] This invention conducts ablation experiments on the RAF-DB public dataset to compare the accuracy of the proposed model with the original ResNet50, the improved ResNet50, the improved ResNet50+MHSA, the improved ResNet50+DACL, the improved ResNet50+MHSA+Softmax Loss, and the improved ResNet50+MHSA+CenterLoss, while keeping the relevant parameters the same. At the same time, comparative experiments are conducted to compare the proposed model with common facial expression recognition algorithms such as GoogleNet, VGG+SVM, ResNet+CBAM, DNN, CNN+SVM, Inception, and DenseNet, while keeping the relevant parameters the same.

[0099] Specifically, the average results obtained by the method of this invention using the model proposed in this embodiment on the RAF-DB dataset are compared with the accuracy of the original ResNet50, the improved ResNet50, the improved ResNet50+MHSA, the improved ResNet50+DACL, the improved ResNet50+MHSA+Softmax Loss, and the improved ResNet50+MHSA+Center Loss in ablation experiments. The results are shown in Table 1.

[0100] Table 3. Comparison of ablation experimental results on the RAF-DB dataset

[0101] method Accuracy ACC ResNet50 85.78％ Improved ResNet50 86.18％ Improved ResNet50+MHSA 86.54％ Improved ResNet50+DACL 87.18％ Improved ResNet50+MHSA+Softmax Loss 86.54％ Improved ResNet50+MHSA+Center Loss 87.06％ Emo-ResNet (the method of this invention) 88.34％

[0102] As shown in Table 3, the facial expression recognition method constructed in this invention has higher accuracy than the original ResNet50, the improved ResNet50, the improved ResNet50+MHSA, the improved ResNet50+DACL, the improved ResNet50+MHSA+SoftmaxLoss, and the improved ResNet50+MHSA+Center Loss.

[0103] Specifically, the average results obtained by the method of this invention using the model proposed in this embodiment on the RAF-DB dataset are compared with common facial expression recognition algorithms such as GoogleNet, VGG+SVM, ResNet+CBAM, DNN, CNN+SVM, Inception, and DenseNet. The results are shown in Table 4.

[0104] Table 4. Comparison of experimental results on the RAF-DB dataset

[0105]

[0106]

[0107] As shown in Table 4, the facial expression recognition method constructed in this invention has a higher accuracy than common facial expression recognition algorithms such as GoogleNet, VGG+SVM, ResNet+CBAM, DNN, CNN+SVM, Inception, and DenseNet.

[0108] The confusion matrix generated by the method of this invention is as follows: Figure 5 As shown, the accuracy rate for recognizing the four expressions of happiness, surprise, disgust, and naturalness is relatively high, especially for happiness, where the accuracy rate is generally above 90%.

[0109] The above embodiments construct a facial expression recognition method for subjects based on Emo-ResNet. Combining the advantages of residual networks, MHSA, and DACL, it achieves high accuracy in real-time classification and recognition of facial expressions. Through ablation experiments and comparative experiments, it is demonstrated that the algorithm proposed in this patent is more suitable for facial expression recognition, and it also provides a method for constructing a facial expression recognition system for subjects.

[0110] Example 4

[0111] like Figure 6 As shown, this embodiment provides a facial expression recognition device based on Emo-ResNet. The device includes a video acquisition module 601, a data transmission module 602, a face detection and extraction module 603, a facial expression recognition module 604, and a recognition result visualization module 605. The specific functions of each module are as follows:

[0112] The video acquisition module 601 acquires the real-time dynamic video stream of the person to be identified;

[0113] The data transmission module 602 performs frame interval extraction on the real-time dynamic video stream and saves it as a static image stream, acquiring an image including the face of the person to be identified.

[0114] The face detection and extraction module 603 identifies the face of the person to be identified from an image containing the face of the person to be identified, performs face correction and face alignment, and then crops out the face image of each person to be identified.

[0115] The facial expression recognition module 604 inputs the face image of each person to be recognized into the trained Emo-ResNet network to obtain the probability of each person's facial expression category. The facial expression corresponding to the highest probability is taken as the facial expression recognition result of the person to be recognized. The facial expression categories include anger, disgust, fear, happiness, sadness, surprise, and neutral. The main framework of the Emo-ResNet network is an improved ResNet50. In the first bottleneck layer of the fourth convolutional block, the multi-head self-attention mechanism MHSA is replaced with the corresponding convolutional layer and normalized network layer. At the same time, the deep attention center loss is used as the loss function.

[0116] The recognition result visualization module 605 outputs the determined facial expression category and the time of the facial expression in the real-time dynamic video to the display device, thus visualizing the recognition result.

[0117] The specific implementation of each module in this embodiment can be found in Embodiment 1 above, and will not be repeated here. It should be noted that the device provided in this embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0118] Example 5

[0119] This embodiment provides a computer device, which can be a computer, such as... Figure 7 As shown, the system is connected via a system bus 701 to a processor 702, a memory, an input device 703, a display 704, and a network interface 705. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 706 and internal memory 707. The non-volatile storage medium 706 stores the operating system, computer programs, and a database. The internal memory 707 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 702 executes the computer program stored in the memory, it implements the facial expression recognition method based on Emo-ResNet proposed in Embodiment 1 above. The process is as follows:

[0120] S1. Video capture step: Obtain the real-time dynamic video stream of the person to be identified;

[0121] S2, Data transmission step: The real-time dynamic video stream is subjected to frame interval extraction and saved as a static image stream to obtain an image including the face of the person to be identified.

[0122] S3, Face detection and extraction step: Identify the face of the person to be identified from the image containing the face of the person to be identified, and perform face correction and face alignment, and then crop to obtain the face image of each person to be identified;

[0123] S4. Facial Expression Recognition Step: Input the face image of each person to be recognized into the trained Emo-ResNet network to obtain the probability of each person's facial expression category. The facial expression corresponding to the highest probability is taken as the facial expression recognition result of the person to be recognized. The facial expression categories include anger, disgust, fear, happiness, sadness, surprise, and neutral. The main framework of the Emo-ResNet network is an improved ResNet50. In the first bottleneck layer of the fourth convolutional block, the multi-head self-attention mechanism MHSA is replaced with the corresponding convolutional layer and normalized network layer. At the same time, the deep attention center loss is used as the loss function.

[0124] S5. Visualization of recognition results: The recognized facial expression category and the time of the facial expression in the real-time dynamic video are output on the display device, thus visualizing the recognition results.

[0125] Example 6

[0126] This embodiment provides a storage medium, which is a computer-readable storage medium, storing a computer program. When the computer program is executed by a processor, it implements the facial expression recognition method based on Emo-ResNet described in Embodiment 1 above. The process is as follows:

[0127] S1. Video capture step: Obtain the real-time dynamic video stream of the person to be identified;

[0128] S2, Data transmission step: The real-time dynamic video stream is subjected to frame interval extraction and saved as a static image stream to obtain an image including the face of the person to be identified.

[0129] S3, Face detection and extraction step: Identify the face of the person to be identified from the image containing the face of the person to be identified, and perform face correction and face alignment, and then crop to obtain the face image of each person to be identified;

[0130] S4. Facial Expression Recognition Step: Input the face image of each person to be recognized into the trained Emo-ResNet network to obtain the probability of each person's facial expression category. The facial expression corresponding to the highest probability is taken as the facial expression recognition result of the person to be recognized. The facial expression categories include anger, disgust, fear, happiness, sadness, surprise, and neutral. The main framework of the Emo-ResNet network is an improved ResNet50. In the first bottleneck layer of the fourth convolutional block, the multi-head self-attention mechanism MHSA is replaced with the corresponding convolutional layer and normalized network layer. At the same time, the deep attention center loss is used as the loss function.

[0131] S5. Visualization of recognition results: The recognized facial expression category and the time of the facial expression in the real-time dynamic video are output on the display device, thus visualizing the recognition results.

[0132] The storage medium described in this embodiment can be a disk, optical disk, computer memory, random access memory (RAM), USB flash drive, portable hard drive, etc.

[0133] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A facial expression recognition method based on Emo-ResNet, characterized in that, The facial expression recognition method includes: S1. Video capture step: Obtain the real-time dynamic video stream of the person to be identified; S2, Data transmission step: The real-time dynamic video stream is subjected to frame interval extraction and saved as a static image stream to obtain an image including the face of the person to be identified. S3, Face detection and extraction step: Identify the face of the person to be identified from the image containing the face of the person to be identified, and perform face correction and face alignment, and then crop to obtain the face image of each person to be identified; S4. Facial Expression Recognition Step: Input the face image of each person to be recognized into the trained Emo-ResNet network to obtain the probability of each person's facial expression category. The facial expression corresponding to the highest probability is taken as the facial expression recognition result of the person to be recognized. The facial expression categories include anger, disgust, fear, happiness, sadness, surprise, and neutral. The main framework of the Emo-ResNet network is an improved ResNet50. In the first bottleneck layer of the fourth convolutional block, the multi-head self-attention mechanism MHSA is replaced with the corresponding convolutional layer and normalized network layer. At the same time, the deep attention center loss is used as the loss function. The Emo-ResNet network is as follows: The input to the Emo-ResNet network is a face image of each subject after face correction and alignment. The Emo-ResNet network, from input to output, consists of sequentially connected convolutional layers (Conv1), the first module, the second module, the third module, the fourth module, a pooling layer (Avgpool), a fully connected layer (FC), and a softmax module. Convolutional layer Conv1 has a kernel size of 3, a stride of 1, and a padding width of 1. The basic units of the first, second, third, and fourth modules are bottleneck layers. Each bottleneck layer has two branches. The first branch consists of sequentially connected convolutional layers (Conv01, BN01, RELU activation function, Conv02, BN02, RELU activation function, Conv03, and BN03). The second branch consists of sequentially connected convolutional layers (Conv01, BN01, RELU activation function, Conv02, BN02, RELU activation function, Conv03, and BN03).

04. The normalized network layer BN04 adds the outputs of the first and second branches and passes them through the activation function ReLU to obtain the output of the bottleneck layer. The number of bottleneck layers in the first, second, third, and fourth modules are 3, 4, 6, and 3, respectively. The first bottleneck layer in the fourth module is replaced by the multi-head self-attention mechanism MHSA, which replaces the convolutional layer Conv04 and the normalized network layer BN04. In the first, second, third, and fourth modules, except for the first bottleneck layer, the second branches of the other bottleneck layers do not have the convolutional layer Conv04 and the normalized network layer BN04. In the Multi-Head Self-Attention (MHSA) mechanism, each attention convergence is called a head, and the number of heads is set to 4. The width and height of the two-dimensional feature map are both set to 14. The calculation formula for MHSA is as follows: MHSA(Q,K,V) = Concat(head1, head2,..., head n )W 0 , The formula for calculating Attention() is: Where n is the number of heads, Q refers to the query vector matrix, K refers to the "queried" vector matrix, Q and K are weight matrices used to calculate attention, V is the weight matrix used to sum the attention values ​​to obtain the final result, and d k Let K be the dimension of the matrix, and let... Through weights A linear transformation is performed, ultimately resulting in a learnable weight matrix W. 0 Multiple heads obtained by Attention n The concatenation results in the final multi-head self-attention result MHSA(Q,K,V), calculated using Q, K, and V. This result is used to extract features from the image. Contact(head1,head2,...,head) n This indicates that head1, head2, ..., head... n To perform concatenation, use Concat(head1, head2, ..., head...). n W 0 This indicates that head1, head2, ..., head n After concatenation, it is then combined with a learnable weight matrix W. o Perform a linear transformation to obtain the final Attention output; S5. Visualization of recognition results: The recognized facial expression category and the time of the facial expression in the real-time dynamic video are output on the display device, thus visualizing the recognition results. 2.The Emo-ResNet-based facial expression recognition method of claim 1, wherein, The video acquisition step involves obtaining the real-time dynamic video stream of the person to be identified, which can be an online real-time screen recording, a video transmitted in real-time from an offline camera, or a pre-recorded video. 3.The Emo-ResNet-based facial expression recognition method of claim 1, wherein, In the data transmission step, the real-time dynamic video stream is subjected to frame extraction. The dynamic video stream is then extracted at intervals according to a pre-specified short time interval and saved as a static image stream of the person to be identified. 4.The Emo-ResNet-based facial expression recognition method of claim 1, wherein, In the face detection and extraction step, faces are detected in each frame of the static image stream. A multi-task convolutional neural network is used to extract face regions from the static images, and multiple regions of interest (ROIs) for each face are extracted simultaneously. These ROIs include the eye region, eyebrow region, mouth region, and facial contour region. The center point of the face is determined based on the eye region, and face correction is performed on the entire face based on the center point. Then, the face images are aligned relative to a Cartesian coordinate system to achieve face alignment. Finally, the images are cropped to obtain a face image for each subject after face correction and alignment. The process is as follows: S31. Use OpenCV's built-in face detection model to detect the faces included in each frame of a static image stream. S32. Use a multi-task convolutional neural network to extract face regions from the detected images, and extract multiple regions of interest for the corresponding faces at the same time. S33. Based on the binocular regions, determine the location of the center point of the face, and perform face correction and alignment on the entire face based on the center point. The center position of the left eye in the binocular regions is set as (x... left_i ,y left_i The center position of the right eye is set to (x). right_i ,y right_i The tilt angle θ of the sample is calculated using equation (1), which is: S34, affine transform the picture with the center point of the left eye center position (x left_i ,y left_i ) and the right eye center position (x right_i ,y right_i ) as the center to obtain the corresponding frontal face image, and the formula of the affine transform is as formula (2): wherein, is the input sample, is the output sample, and is the offset, A is an affine matrix related to the tilt θ, and is specifically as follows: S35. Cropping yields a face image of each person to be identified, after face correction and alignment. 5.The Emo-ResNet-based facial expression recognition method of claim 1, wherein, The training process of the Emo-ResNet network is as follows: The face detection steps were extracted using the FER2013 dataset. Data augmentation, including flipping, rotating, and cropping, was performed on the face images after face correction and alignment. The augmented face images were then merged with the original dataset to form a new training set. Next, the new training set is iteratively trained on the Emo-ResNet network until the loss function converges, resulting in a trained recognition network model. The iterative function during training is the deep attention center loss function, the scheduler is the cosine scheduler, and the optimizer is the stochastic gradient descent optimizer (SGD). The base learning rate is set to 0.01, the batch size is set to 32, and the learning rate is reduced when the loss function no longer improves after 30 small samples. The maximum number of training iterations is 300, and the evaluation metric is accuracy. After continuous fine-tuning of the Emo-ResNet network and comparing the accuracy, the optimal trained network model is finally obtained. 6.The Emo-ResNet-based facial expression recognition method of claim 1, wherein, In step S5, the visualization step of the recognition result combines the result determined by the facial expression recognition step with the corresponding time of the image stream transmitted from the data transmission step in the video stream, and outputs it on the front-end interface to visualize the recognition result and provide timely feedback on the expression of the person being recognized.

7. An ultrasonic image processing apparatus based on the Emo-ResNet-based facial expression recognition method according to any one of claims 1 to 6, characterized by The facial expression recognition device includes: The video capture module acquires the real-time dynamic video stream of the person to be identified; The data transmission module extracts the frame intervals of the real-time dynamic video stream and saves it as a static image stream, acquiring an image including the face of the person to be identified. The face detection and extraction module identifies the face of the person to be identified from the image containing the face of the person to be identified, performs face correction and face alignment, and then crops out the face image of each person to be identified; The facial expression recognition module inputs the face image of each person to be recognized into a trained Emo-ResNet network to obtain the probability of each person's facial expression category. The facial expression with the highest probability is taken as the facial expression recognition result of the person to be recognized. The facial expression categories include anger, disgust, fear, happiness, sadness, surprise, and neutral. The main framework of the Emo-ResNet network is an improved ResNet50. In the first bottleneck layer of the fourth convolutional block, the multi-head self-attention mechanism MHSA is replaced with the corresponding convolutional layer and normalized network layer. At the same time, the deep attention center loss is used as the loss function. The recognition result visualization module outputs the determined facial expression category and the time of the facial expression in the real-time dynamic video on the display device, thus visualizing the recognition result.

8. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the facial expression recognition method based on Emo-ResNet as described in any one of claims 1 to 6.

9. A storage medium storing a program, characterized by comprising: When the program is executed by the processor, it implements the facial expression recognition method based on Emo-ResNet as described in any one of claims 1 to 6.

Images