Facial expression recognition method based on interpretability and application

Abstract

The invention belongs to image processing and deep learning in the field of artificial intelligence, and particularly relates to a facial expression recognition method which is realized by utilizing a deep learning algorithm and has high transparency and interpretability and application of the facial expression recognition method. The objective of the invention is to solve the problems that an existing facial expression and emotion recognition technology is low in accuracy in a complex environment, and a model is opaque and does not have interpretability. The method comprises the following steps: firstly, acquiring a facial video stream through a camera or other equipment, enhancing data to improve the robustness and generalization ability of a model, then, carrying out facial detection by using an MTCNN model to obtain facial bounding box coordinates and key point information, then, carrying out expression classification by using an Xception and YOLOv5-cls double deep learning model, and finally, carrying out facial expression classification by using a facial expression classification model. And generating a saliency map through an RISE algorithm. According to the method, the robustness and generalization ability of recognition are improved through learning of the double deep learning models, the recognition accuracy is remarkably improved, the method is particularly excellent in performance in a complex environment, the significance graph is generated through the RISE algorithm, the decision process of the models is explained, and the transparency and the user trust degree of the models are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and to the field of facial recognition technology of image processing and deep learning, and in particular to a facial expression recognition method based on interpretability and its application. Background Art

[0002] With the continuous advancement of artificial intelligence and human-computer interaction technology, the application scope of emotion recognition technology continues to expand, covering many industries such as entertainment, transportation, healthcare, and retail. By identifying user emotions, these applications can provide personalized services and improve the naturalness and efficiency of human-computer interaction.

[0003] Among them, facial expression recognition technology has gradually attracted widespread attention in the field of emotion recognition. The success of deep convolutional neural networks (CNNs) has promoted the rapid development of facial expression recognition research. Despite certain achievements, people still have a poor understanding of how neural networks make decisions, how certain these decisions are, when they can be trusted, and when they need to be corrected. In areas where decisions may have significant consequences (such as medical diagnosis, autonomous driving, criminal justice, etc.), model transparency is particularly important. A large number of studies have shown that in cognitive psychology, philosophy, and machine learning, the key to understanding and building trust lies in the explanation of the decision-making process. Summary of the invention

[0004] In view of the deficiencies in the above-mentioned background technology, the present invention proposes a facial expression recognition method based on interpretability to solve the above-mentioned technical problems.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions: collect open source facial expression recognition data sets, use cameras or other image acquisition devices to collect facial expressions in real time. Data enhancement is achieved by adding random rotation, random jitter and random flipping of the picture; use target detection technology to detect the video stream picture, and obtain all facial bounding box coordinates and key point information detected in each frame of the image; use computer vision and deep learning algorithms to perform expression recognition and analysis on the collected facial images, and identify the happy, sad or angry emotional state; use interpretability algorithms to perform heat map and saliency interpretation on the facial recognition results.

[0006] In addition, the present invention also proposes a facial expression recognition device based on interpretability. Its structure includes the following modules: a data acquisition module, which is responsible for real-time acquisition of facial expressions using a camera or other image acquisition equipment, and performs enhanced processing by adding random rotation, random jitter of the picture, and random flipping of the picture; a facial detection module, which is used to obtain the bounding box coordinates of the face; an expression recognition and classification module, which detects facial expressions through deep learning and image recognition technology; and a saliency interpretation module, which generates a saliency map for the recognition result by means of random masking, thereby explaining the model execution and decision-making process.

[0007] The face detection module is formed by a face detection model MTCNN, which is composed of three sub-networks: P-Net, R-Net and O-Net. The P-Net receives an input image and generates a candidate face region. The P-Net outputs the confidence of each region. When the output confidence is higher than 0.9, the candidate region is retained and passed to the R-Net. The R-Net receives the candidate region passed by the P-Net and performs face detection in the candidate region. The R-Net outputs the confidence of each region and fine-tunes the bounding box. When the output confidence is higher than 0.9, it is determined to be a valid face region and passed to the O-Net. The O-Net receives the refined face region output by the R-Net and performs final face key point positioning. When the confidence output by the O-Net is higher than 0.9, the bounding box coordinates of the face are obtained and the key point positioning results are retained.

[0008] The emotion recognition and classification module is composed of an Xception model and a YOLOv5-cls model. The optimization process of the expression recognition and classification module is as follows: 15,000 expression images are selected from the FER2013 dataset, which contains more than 35,000 grayscale images and covers seven expressions of anger, disgust, fear, happiness, sadness, surprise, and calmness, and the KDEF dataset, which contains more than 4,900 color images and has seven different expressions, each of which is divided into five angles, to obtain an open source dataset; pre-process the open source dataset, trim the images in the dataset into 640x480 pixel labeled images, and scale them to 224x224 pixel square images to obtain 15000 expression images; 0 expression data; 15,000 images are divided into training set, validation set and test set in a ratio of 8:1:1; when receiving face information, the Xception model and YOLOv5-cls model are called for prediction at the same time, and the confidence threshold of each model is set to 0.7; if the prediction results of the two models are consistent, the results are directly passed to the saliency interpretation module; if the prediction results of the two models are inconsistent, the weighted average method is used to calculate the final judgment, where the Xception weight is 0.9 and the YOLOv5-cls model weight is 0.85. Calculate the weighted average of the prediction results of the two models, re-compare the confidence, and make the final judgment.

[0009] The saliency interpretation module consists of the RISE algorithm. In order to generate a saliency map for model prediction, RISE queries the black box model on multiple randomly masked versions of the input; after all queries are completed, the scores of all masks are averaged to obtain the final saliency map.

[0010] The beneficial effects of the present invention are:

[0011] (1) The facial expression recognition method proposed in the present invention constructs a dual deep learning network to perform task learning. The classification and recognition can adapt to complex environments and has high robustness. Compared with traditional emotion recognition methods, this method greatly improves the recognition accuracy;

[0012] (2) For the collected open source facial expression dataset, data enhancement is achieved by adding random rotation, random jitter and random flipping, which enhances the model's ability to learn information and improves the model's robustness and generalization ability;

[0013] (3) The present invention generates a saliency map through an interpretable algorithm, making the decision-making process of the model clearer and more transparent, improving the confidence of the model and enhancing the interpretability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the overall flow chart of the present invention;

[0015] Figure 2 A schematic diagram of a specific implementation network architecture of a facial expression recognition method based on interpretability provided by the present invention;

[0016] Figure 3 A schematic diagram of the structure of a facial expression recognition device based on interpretability provided by the present invention;

[0017] Figure 4 is a scene diagram of an emotion recognition device according to an embodiment of the present disclosure; DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0019] Implementation method 1: Figure 1 and Figure 3 As shown, the facial expression recognition method based on interpretability described in this embodiment, the system includes: a data acquisition module, a face detection module, an expression recognition and classification module, and a saliency interpretation module.

[0020] The data acquisition module is used to collect facial video streams and transmit the facial video streams to the facial detection module; the facial detection module is used to receive facial video streams, detect facial information in each frame of the facial video stream, and transmit the facial information to the expression recognition and classification module; the expression recognition and classification module classifies the facial expressions of the detected facial areas, and makes fusion judgments on the classification results to infer the emotional state; the saliency interpretation module generates a saliency map for the recognition results by means of random masks, thereby explaining the execution and decision-making process.

[0021] Traditional speech recognition or text analysis methods rely on the user's language expression to infer emotions, but these methods may be affected by factors such as the user's expressive ability, language style, and complexity of emotional expression, resulting in limited accuracy in emotion recognition. The system described in this embodiment uses facial expression recognition, which can directly capture subtle emotional changes on the user's face, does not rely on the user's language expression, and can provide more direct and detailed emotional data, thereby improving the accuracy of emotion recognition.

[0022] Traditional deep neural networks are often viewed as black box structures because they consist of multiple layers of nonlinear transformations that make it extremely difficult to directly understand how inputs affect outputs. The system described in this embodiment uses an interpretable algorithm to display the model's internal working judgments and decision-making processes through visualization technology, thereby improving model transparency and confidence.

[0023] Implementation method 2. This implementation method further limits the facial expression recognition method based on interpretability described in implementation method 1. The frame rate of the facial video stream collected in the data acquisition module is 30 frames per second and the resolution is 640x480 pixels. In this implementation method, a frame rate of 30 frames per second is used to ensure the smoothness of the video stream, so that the slight changes in facial expressions can be accurately captured, avoiding the loss or miss of expression details caused by too low a frame rate. A sufficient frame rate can ensure that the system captures the user's emotional fluctuations in real time and provides timely data support for emotional state inference.

[0024] Furthermore, the use of a resolution of 640x480 pixels provides a good balance between image quality and computing resources, avoiding the computing burden of too high a resolution while still clearly capturing the key details of facial expressions. Compared to higher resolutions, 640x480 pixels can reduce the computing resources required for image processing and expression recognition, improving the processing speed and response efficiency of the system.

[0025] Implementation method three, this implementation method is a further limitation of the facial expression recognition method based on interpretability described in implementation method one, the facial detection module is formed by a facial detection model MTCNN, and the facial detection model MTCNN is composed of three sub-networks: P-Net, R-Net and O-Net; the P-Net receives an input image and generates candidate face regions, wherein the input image is an image of 640x480 pixels; the P-Net outputs the confidence of each region, and when the output confidence is higher than 0.9, the candidate region is retained and passed to the R-Net; the R -Net receives the candidate area transmitted by P-Net, the candidate area is cropped and scaled to 24x24 pixels, and face detection is performed in the candidate area. The R-Net outputs the confidence of each area and fine-tunes the bounding box. When the output confidence is higher than 0.9, it is determined to be a valid face area, and the valid face area is transmitted to O-Net. The O-Net receives the refined face area output by R-Net and performs the final facial key point positioning. When the confidence output by O-Net is higher than 0.9, the bounding box coordinates of the face are obtained, and the key point positioning results are retained.

[0026] In this embodiment, by decomposing the facial detection task into three sub-networks (P-Net, R-Net, O-Net), the detection results can be gradually refined at each stage, thereby improving the accuracy of facial detection and key point positioning. Each sub-network focuses on different tasks when processing facial areas, from the generation of candidate areas to refinement to the precise positioning of key points, to achieve step-by-step optimization and reduce false detections and missed detections. By setting the confidence threshold, low-confidence areas can be effectively filtered out, improving processing efficiency and reducing computational burden.

[0027] Specifically, P-Net is responsible for generating candidate face regions and preliminarily evaluating the confidence of these regions. Through preliminary screening, the areas that need further processing can be quickly reduced, reducing the amount of computation. R-Net performs more refined detection and fine-tuning of bounding boxes on the candidate regions generated by P-Net to improve the accuracy of the regions and reduce false positives. O-Net further refines the bounding boxes and performs final key point positioning to ensure accurate extraction of facial features. Setting a confidence threshold (>0.9) ensures that the results output at each step have high reliability, thereby improving the accuracy of the final face detection and key point positioning. By processing and screening layer by layer, the computational burden of subsequent networks can be reduced.

[0028] P-Net and R-Net first screen and optimize the regions, and only the regions with high confidence are passed to O-Net, which can significantly reduce the number of regions that O-Net needs to process, thereby improving processing efficiency. O-Net is responsible for key point positioning in the final stage, and can accurately obtain the location of facial features, which is crucial for expression recognition, because subtle changes in facial expressions require accurate key point data for analysis.

[0029] Implementation method 4. This implementation method further limits the facial expression recognition method based on interpretability described in implementation method 1, and the emotion recognition classification module is composed of an Xception model and a YOLOv5-cls model. The entire Xception network can be divided into the following main parts: Initial layer: The model starts with a regular convolution layer using a 3x3 convolution kernel. Depth-wise separable convolution block: The outlet stream contains two depth-wise separable convolution blocks, where the structure of the first block is similar to the block in the inlet stream, but the last convolution layer is replaced by a 3x3 maximum pooling layer for downsampling.

[0030] YOLOv5-cls consists of the following parts: Backbone: The main feature extractor, which uses a structure similar to CSPNet to build the Backbone, which helps improve the efficiency and performance of the model. Head structure: Maps the features extracted by the Backbone to the category label, including a global average pooling layer to compress the feature map into a vector of fixed length, followed by a fully connected layer to output the category probability.

[0031] Implementation method 5. This implementation method is a further limitation of the facial expression recognition method based on interpretability described in implementation method 1. In order to generate an importance map, the input image is first subsampled by a random binary mask, and the model's response to each masked image is recorded. These responses calculate the combination weights based on the probability of the model's predicted output. The smaller binary mask is then upsampled to a higher resolution through bilinear interpolation. In this process, the mask is transformed from a binary state to a state containing intermediate values. Finally, to increase diversity, we randomly translate the pixels of all masks in two spatial directions. The final importance map is formed by linearly combining these processed masks, where the weights reflect the model's predicted probability of the masked image.

Claims

1. A facial expression recognition method and application based on interpretability, characterized in that: The system includes: a data acquisition module, a face detection module, an expression recognition and classification module, and a saliency interpretation module. The data acquisition module is used to collect face video streams and transmit the face video streams to the face detection module; the face detection module is used to receive face video streams and detect face information in each frame of the face video streams and transmit the face information to the expression recognition and classification module; the expression recognition and classification module classifies the detected face area, and makes a fusion judgment on the classification results to infer the emotional state; the saliency interpretation module generates a saliency map for the recognition results by means of random masking, so as to explain the decision-making process.

2. A facial expression recognition method and application based on interpretability, characterized in that: The following steps are involved: S1, facial expression acquisition: real-time facial video stream acquisition through a camera or other video acquisition equipment; S2, facial expression detection: The facial detection module detects the video stream and obtains the coordinates and key point information of all facial bounding boxes detected in each frame; S3, Expression analysis and detection: Using computer vision and deep learning algorithms, the collected facial images are used to recognize and analyze expressions and identify emotional states such as happiness, sadness, or anger; S4. Saliency interpretation of emotion detection: The saliency interpretation module generates a heat map for facial recognition results and performs saliency interpretation.

3. The method and application of facial expression recognition based on interpretability according to claim 1, characterized in that: The frame rate of the facial video stream collected in the data collection module is 30 frames per second and the resolution is 640x480 pixels.

4. The method and application of facial expression recognition based on interpretability according to claim 1, characterized in that: The face detection module is formed by a face detection model MTCNN, which is composed of three sub-networks: P-Net, R-Net and O-Net. The P-Net receives an input image and generates a candidate face region. The P-Net outputs the confidence of each region. When the output confidence is higher than 0.9, the candidate region is retained and passed to the R-Net. The R-Net receives the candidate region passed by the P-Net and performs face detection in the candidate region. The R-Net outputs the confidence of each region and fine-tunes the bounding box. When the output confidence is higher than 0.9, it is determined to be a valid face region and passed to the O-Net. The O-Net receives the refined face region output by the R-Net and performs final face key point positioning. When the confidence output by the O-Net is higher than 0.9, the bounding box coordinates of the face are obtained and the key point positioning results are retained.

5. The method and application of facial expression recognition based on interpretability according to claim 1, characterized in that: The expression recognition and classification module consists of an Xception model and a YOLOv5-cls model.

6. The method and application of facial expression recognition based on interpretability according to claim 1, characterized in that: The optimization process of the expression recognition and classification module is as follows: 15,000 expression images are selected from the FER2013 dataset, which contains more than 35,000 grayscale images and covers seven expressions of anger, disgust, fear, happiness, sadness, surprise, and calmness, and the KDEF dataset, which contains more than 4,900 color images and has seven different expressions, each of which is divided into five angles, to obtain an open source dataset; Preprocess the open source dataset, trim the images in the dataset into 640x480 pixel annotated images, and scale them into 224x224 pixel square images, and obtain 15,000 expression data; divide the 15,000 images into training set, validation set and test set in a ratio of 8:1:1; when receiving face information, call the Xception model and YOLOv5-cls model for prediction at the same time, and set the confidence threshold of each model to 0.7; if the prediction results of the two models are consistent, the results are directly passed to the saliency interpretation module; if the prediction results of the two models are inconsistent, the weighted average method is used to calculate the final judgment, where the Xception weight is 0.9 and the YOLOv5-cls model weight is 0.

85. Calculate the weighted average of the prediction results of the two models, re-compare the confidence, and make the final judgment.

7. A method for facial expression recognition based on interpretability and its application, comprising a system for the method for facial expression recognition based on interpretability as claimed in any one of claims 1 to 6, characterized in that: It also includes computers or embedded devices, screens or displays, and cameras.

Images