A method and apparatus for emotion recognition based on facial expression and human posture

Abstract

The application discloses a kind of based on facial expression and human posture emotional recognition method and device, method includes the following steps: using Dlib network to the facial expression image is detected to obtain face partial image, using VGG network to face partial image is detected to obtain facial expression emotional recognition result;Using Yolo network to the human posture image is detected to obtain human partial image, using AlphaPose network to human partial image is detected to obtain human key point information, using the SE-GCN network of fusion attention mechanism to human key point information is detected to obtain human posture emotional recognition result;Facial expression and human posture emotional recognition result are fused to obtain final emotional recognition result.The application extracts features to two kinds of modalities of facial expression and human posture and carries out feature fusion, realizes the emotional recognition of double modalities, can effectively improve the accuracy of emotional recognition.

Description

Technical Field

[0001] This invention belongs to the field of emotion computing technology in computer vision, specifically relating to an emotion recognition method and device based on facial expressions and human posture. Background Technology

[0002] Emotion recognition refers to the process by which computers preprocess and extract features from emotional signals, and automatically identify the types of human emotions. It falls under the category of affective computing. Affective computing involves many fields such as social psychology, cognitive science, neurology, physiology, and computer science. Its aim is to establish a harmonious human-computer interaction environment and enable computers to have higher intelligence by giving them the ability to recognize, understand, and simulate human emotions. Among all these research areas, emotion recognition is one of the most fundamental and important.

[0003] Emotion recognition methods generally include steps such as raw image input, data preprocessing, manual feature extraction, and emotion classification, with feature extraction being the most crucial step. Traditional methods rely on manual feature extraction, but in real-world applications, the large dimensionality and computational complexity of emotion features make it difficult to handle data from real-world scenarios. However, with the rapid development of deep learning technology, multi-layered neural networks are used to extract deep feature information, significantly improving the effectiveness of emotion recognition.

[0004] Human emotional expression manifests externally through changes in facial expressions, body posture, and tone of voice, while internally through changes in physiological signals such as heart rate, blood pressure, and brain waves, indicating that emotional expression is essentially multimodal.

[0005] Facial expressions are the strongest outward manifestation of emotions. Currently, most emotion recognition methods focus solely on facial expression image information, identifying emotions through feature extraction and achieving some success. However, in real-world scenarios, limitations such as lighting and distance often result in small faces in images, leading to lower facial feature accuracy and impacting recognition results. Furthermore, research indicates that facial expressions can be easily confused, such as sadness, fear, and disgust. Therefore, relying solely on single-modal facial features to determine a person's emotional state has limitations.

[0006] Human posture is also an outward expression of emotion. When expressing emotions, people often accompany them with related actions, i.e., body language. For example, when happy, a person will hold their head high and chest out, their body relaxed; when sad, they will lower their head, let their hands hang down, and their body contracted. In real-life scenarios, human posture is easier to capture and changes more clearly than facial expressions, providing a wealth of emotional information.

[0007] Existing research on bimodal emotion recognition based on facial expressions and body posture is relatively limited. Compared to single-modal emotion recognition methods, bimodal emotion recognition using facial expressions and body posture can fully utilize information from both, resulting in more accurate results. Furthermore, when information from one modality is missing, such as when the face is occluded, information from other modalities can be used to determine the emotion of the target individual, enhancing the robustness of the method. Therefore, further in-depth research on bimodal emotion recognition technology based on facial expressions and body posture is needed to further improve the accuracy of emotion recognition. Summary of the Invention

[0008] In view of the above, the purpose of this invention is to provide an emotion recognition method and apparatus based on facial expressions and human posture, which achieves high accuracy in emotion recognition by performing feature analysis and fusion on images of facial expressions and human posture of the target person.

[0009] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0010] In a first aspect, embodiments of the present invention provide an emotion recognition method based on facial expressions and human posture, comprising the following steps:

[0011] Acquire the facial expression image and human posture image to be identified, and perform preprocessing;

[0012] The Dlib face object detection network is used to detect the preprocessed facial expression image to obtain the face image, and the VGG facial expression emotion recognition network is used to detect the face image to obtain the facial expression emotion recognition result.

[0013] The YOLO human target detection network is used to detect the preprocessed human pose image to obtain the human part image. The AlphaPose human key point detection network is used to detect the human part image to obtain human key point information. The SE-GCN graph convolutional neural network with fused attention mechanism is used to detect the human key point information to obtain the human pose emotion recognition result.

[0014] The facial expression emotion recognition results and the human posture emotion recognition results are fused to obtain the target person's emotion recognition results.

[0015] Specifically, the step of acquiring the facial expression image and human posture image to be identified, and performing preprocessing, includes:

[0016] The acquired facial expression image and human posture image are images of the same target person at the same time. One camera captures a clear facial expression image of the target person with a large proportion of the face area, while another camera captures a human posture image of the target person's complete human torso, including the head. Then, the facial expression image and human posture image are preprocessed, including grayscale transformation and filtering noise reduction.

[0017] Specifically, the step of using the Dlib face detection network to detect facial expression images in the preprocessed facial expression image to obtain face portion images includes:

[0018] The Dlib face detection network based on ResNet is used to detect face objects in facial expression images, output the coordinates of the face bounding boxes in the image, and then the face part is cropped, resized and grayscaled to obtain a grayscale face image.

[0019] Specifically, the step of using the VGG facial expression emotion recognition network to detect facial features in a face image and obtain facial expression emotion recognition results includes:

[0020] The face image is input into the VGG facial expression emotion recognition network. After multiple convolutional layers and multiple fully connected layers, the deep features of the face image are extracted. Then, the softmax function is used for classification, and the final output is the facial expression emotion recognition result.

[0021] Specifically, the step of using the YOLO human target detection network to detect the preprocessed human pose image to obtain a human portion image includes:

[0022] The YOLO object detection network is used to detect human targets in human pose images, output the coordinates of the human bounding box in the image, and then the image within the human bounding box is cropped and resized to obtain the RGB human part image.

[0023] Specifically, the step of using the AlphaPose human keypoint detection network to detect human keypoint information in a human body image includes:

[0024] The Alphapose human keypoint detection network is used to extract and output human keypoint information, including the coordinates and confidence scores of human keypoints, from human body images.

[0025] Specifically, the method of using an SE-GCN graph convolutional neural network with a fusion attention mechanism to detect key point information of the human body to obtain human posture and emotion recognition results includes:

[0026] Human body key point information is input into the SE-GCN graph convolutional neural network with an attention mechanism. The deep features of the human body pose image are extracted through multiple graph convolutional layers using the SENet graph attention module and at least one fully connected layer. After classification by the softmax function, the final output is the human body pose emotion recognition result.

[0027] Specifically, the fusion of facial expression emotion recognition results and human posture emotion recognition results to obtain the target person's emotion recognition results includes:

[0028] The facial expression emotion recognition results and the human posture emotion recognition results are integrated at the decision level, and the emotion recognition results of the target person are obtained through weighted calculation, and the recognized emotion category is output.

[0029] Secondly, in order to achieve the above-mentioned objectives, the present invention also provides an emotion recognition device based on facial expressions and human posture, including: an image preprocessing module, a facial expression emotion recognition module, a human posture emotion recognition module, and a multimodal fusion emotion recognition module.

[0030] The image preprocessing module is used to acquire facial expression images and human posture images to be identified, and to perform preprocessing.

[0031] The facial expression emotion recognition module is used to detect the preprocessed facial expression image through the Dlib face target detection network to obtain the face portion image, and to detect the face portion image through the VGG facial expression emotion recognition network to obtain the facial expression emotion recognition result.

[0032] The human posture emotion recognition module is used to detect the preprocessed human posture image through the YOLO human target detection network to obtain the human body part image, to detect the human body part image through the AlphaPose human key point detection network to obtain human key point information, and to detect the human key point information through the SE-GCN graph convolutional neural network with fused attention mechanism to obtain the human posture emotion recognition result.

[0033] The multimodal fusion emotion recognition module is used to fuse facial expression emotion recognition results and human posture emotion recognition results to obtain the emotion recognition result of the target person.

[0034] Thirdly, to achieve the above-mentioned objectives, embodiments of the present invention also provide an emotion recognition device based on facial expressions and human posture, comprising: a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned emotion recognition method based on facial expressions and human posture when executing the computer program.

[0035] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0036] (1) When performing feature extraction and emotion recognition on facial expression images, this invention improves the robustness of face detection in complex backgrounds by using the Dlib face target detection network and enhances the accuracy of emotion recognition by using the VGG facial expression emotion recognition network to extract multi-level deep features of the face image.

[0037] (2) When performing feature extraction and emotion recognition on human pose images, this invention can quickly and accurately identify human body regions by using the YOLO human target detection network and accurately capture information of various key points of the human body by using the AlphaPose human key point detection network, providing a rich data foundation for subsequent emotion recognition. By combining the SE-GCN graph convolutional neural network constructed with the attention mechanism, the network's ability to pay attention to important nodes can be improved, thereby obtaining more accurate emotion recognition results.

[0038] (3) By using decision-level fusion to fuse the emotion recognition results of the two modalities, this invention can make full use of the information of facial expressions and postures, and further improve the accuracy of emotion recognition. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the emotion recognition method based on facial expressions and human posture provided in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the framework of the emotion recognition method based on facial expressions and human posture provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of the emotion recognition device based on facial expressions and human posture provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0044] The inventive concept of this invention is as follows: Addressing the problem that existing research on bimodal emotion recognition based on facial expressions and human posture is relatively limited and its accuracy needs improvement, this invention provides an emotion recognition method and apparatus based on facial expressions and human posture. It combines the Dlib face object detection network and the VGG facial expression emotion recognition network for facial expression emotion recognition, and combines the YOLO human object detection network, the AlphaPose human keypoint detection network, and the SE-GCN graph convolutional neural network with an attention mechanism for human posture emotion recognition. By fusing the facial expression emotion recognition results and the human posture emotion recognition results, the final emotion category is obtained, thus achieving bimodal emotion recognition and effectively improving the accuracy of emotion recognition.

[0045] Figure 1 This is a flowchart illustrating the emotion recognition method based on facial expressions and human posture provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the framework of the emotion recognition method based on facial expressions and human posture provided in an embodiment of the present invention. Figure 1 and Figure 2 As shown, the embodiment provides an emotion recognition method based on facial expressions and human posture, including the following steps:

[0046] S1: Acquire the facial expression image and human posture image to be identified, and perform preprocessing.

[0047] Specifically, the facial expression image and the human posture image are images of the same target person at the same time. One camera captures a clear facial expression image of the target person with a large proportion of the face area, while another camera captures a human posture image of the target person's complete human torso, including the head.

[0048] Preprocessing is performed on facial expression images and human pose images, including grayscale transformation and filtering for noise reduction.

[0049] (1.1) Gray-scale transformation: Gamma transformation is used to enhance the low-brightness areas of the image. The calculation method of Gamma transformation is as follows:

[0050] s = cr γ

[0051] Where s is the output value after Gamma transformation, c is a constant (c is 1 in this embodiment), r is the original brightness value, and γ is the Gamma factor (γ is 0.85 in this embodiment). If the input is a grayscale image, the brightness value of the pixel is subjected to Gamma transformation; if the input is an RGB image, the R, G, and B components of the pixel are subjected to Gamma transformation respectively.

[0052] (1.2) Filtering and noise reduction: Gaussian filtering is used to remove image noise. The Gaussian function G(x,y) is as follows:

[0053]

[0054] Where σ is the standard deviation, and x and y are the pixel coordinates of the image row and column, respectively. In this embodiment, σ is set to 1.5 and discretized into a 5×5 Gaussian convolution kernel. Since the Gaussian function is linearly separable, in this embodiment, the convolution calculation of the input image with the two-dimensional Gaussian convolution kernel is equivalently transformed into first calculating the convolution result G(x) of each row of pixels with the one-dimensional Gaussian convolution kernel, and then calculating the convolution result G(y) of each column of pixels with the one-dimensional Gaussian convolution kernel, so as to achieve the goal of accelerating the calculation.

[0055] S2, the Dlib face object detection network is used to detect the preprocessed facial expression image to obtain the face portion image, and the VGG facial expression emotion recognition network is used to detect the face portion image to obtain the facial expression emotion recognition result.

[0056] (2.1) The Dlib face detection network based on ResNet was used to detect face targets in facial expression images. The Dlib face detection network was trained using datasets such as FaceScrub and VGG and contained a total of 29 convolutional layers. The coordinates of the face bounding boxes in the output image were then processed by cropping, resizing, and grayscale conversion of the image within the face bounding boxes to obtain a grayscale face image of size 48×48 pixels.

[0057] (2.2) The VGG-19 facial expression emotion recognition network was used to detect facial features. The VGG-19 facial expression emotion recognition network was trained on the FER2013 dataset. A 48×48 pixel grayscale facial image was used as input. After extracting the depth feature information of the facial image through 16 convolutional layers and 3 fully connected layers, the image was then classified using the softmax function. Finally, discrete emotion prediction values ​​were output as the facial expression emotion recognition results.

[0058] S3 uses the YOLO human target detection network to detect human body parts in the preprocessed human pose image, uses the AlphaPose human keypoint detection network to detect human keypoint information in the human body parts image, and uses the SE-GCN graph convolutional neural network with attention fusion to detect human keypoint information to obtain human pose emotion recognition results.

[0059] (3.1) The YOLOv5 object detection network was used to detect human targets in human pose images. The YOLOv5 object detection network was trained using the COCO dataset. After single-stage object detection and non-maximum suppression, the coordinates of the human bounding box in the image were output. Then, the image within the human bounding box was cropped and resized to adjust the size of the human region to a 256×192 pixel RGB human part image.

[0060] (3.2) The Alphapose human keypoint detection network, which is a FastPose-DCN model based on ResNet50-DCN, is used to detect human body parts. The Alphapose human keypoint detection network takes a 256×192 pixel RGB human body image as input, passes through a symmetric spatial transformation network, a single-person pose estimation network, and non-maximum suppression, and outputs the coordinates (x, y, y) of 136 human keypoints, including the head, shoulders, elbows, wrists, hips, knees, and ankles. i ,y i ) and confidence level f i , where i is the index of the human body key point.

[0061] (3.3) The SE-GCN graph convolutional neural network with fusion attention mechanism is used to detect human key point information. The SE-GCN graph convolutional neural network with fusion attention mechanism includes three layers of GCN graph convolutional layers using SENet graph attention module and one fully connected layer to extract the depth features of human pose image. After classification by softmax function, the discrete emotion prediction value is finally output as the human pose emotion recognition result.

[0062] The main architecture of SE-GCN is based on GCN, and a graph attention module based on Squeeze-Excitation Networks (SENet) is added before each GCN convolutional layer to improve the network's ability to pay attention to important nodes, thereby obtaining more accurate emotion recognition results.

[0063] For the i-th layer of SE-GCN, the input is the feature map. Its adjacency matrix Where N i Let D be the number of nodes in the graph. i The number of feature dimensions for each node. The SENet graph attention module first focuses on X... i Use compression operation F sq (·) Calculate the descriptor The compressed global features are obtained. i The j-th element z ij The calculation method is as follows:

[0064]

[0065] Get descriptor z i Then, it is subjected to an excitation operation F. ex (·,W), to obtain the descriptor The weights of each feature dimension are represented as follows:

[0066] s i =F ex (z i ,W)=σ(W2δ(W1z i ))

[0067] in, and Let r be two linear transformation matrices, which are hyperparameters set to reduce the number of parameters and complexity of the operation. σ(·) is the sigmoid activation function, and δ(·) is the ReLU activation function.

[0068] Using attention weights i For X i Recalibrate to obtain the recalibrated feature matrix. The j-th element The calculation method is as follows:

[0069] GCN convolutional layers use Perform graph convolution to obtain the feature map X of the (i+1)th layer. i+1 :

[0070]

[0071] in, I is the identity matrix. for The degree matrix, It is the weight matrix of the i-th layer.

[0072] S4 fuses the facial expression emotion recognition results and the human posture emotion recognition results to obtain the target person's emotion recognition results.

[0073] The facial expression emotion recognition results and the human posture emotion recognition results are fused at the decision-making level, and the emotion recognition result of the target person is obtained through weighted calculation:

[0074] p i =αp i,face +βp i,body

[0075] Where, p i Let p be the confidence level for the i-th emotion type. i,face For the confidence score of the i-th type of facial expression emotion, pi,face Let α be the confidence score of the i-th type of human posture emotion, and β be the weighting factors, with α+β=1. The final output is the recognized emotion category.

[0076] In summary, an emotion recognition method based on facial expressions and human posture improves the speed and accuracy of recognition by performing facial expression emotion recognition and human posture emotion recognition separately using different networks. Furthermore, by extracting features from the two modalities and fusing them, a bimodal emotion recognition method is achieved. Compared with traditional emotion recognition methods such as unimodal methods, this method can effectively improve the accuracy of emotion recognition.

[0077] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides an emotion recognition device 300 based on facial expressions and human posture, including: an image preprocessing module 310, a facial expression emotion recognition module 320, a human posture emotion recognition module 330, and a multimodal fusion emotion recognition module 340.

[0078] The image preprocessing module 310 is used to acquire the facial expression image and human posture image to be recognized, and to perform preprocessing.

[0079] The facial expression emotion recognition module 320 is used to detect the preprocessed facial expression image through the Dlib face object detection network to obtain the face portion image, and then use the VGG-19 facial expression emotion recognition network to detect the face portion image to obtain the facial expression emotion recognition result.

[0080] The human posture emotion recognition module 330 is used to detect human body parts in the preprocessed human posture image through the YOLOv5 human target detection network, detect human body key point information in the human body parts image through the AlphaPose human key point detection network, and detect human body key point information through the SE-GCN graph convolutional neural network with attention mechanism to obtain the human posture emotion recognition result.

[0081] The multimodal fusion emotion recognition module 340 is used to fuse the facial expression emotion recognition results and the human posture emotion recognition results to obtain the emotion recognition results of the target person.

[0082] Based on the same inventive concept, embodiments of the present invention also provide an emotion recognition device based on facial expressions and human posture, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described emotion recognition method based on facial expressions and human posture when executing the computer program.

[0083] It should be noted that the emotion recognition device and the emotion recognition equipment based on facial expressions and human posture provided in the above embodiments belong to the same inventive concept as the emotion recognition method based on facial expressions and human posture. For details of their specific implementation process, please refer to the embodiments of the emotion recognition method based on facial expressions and human posture, which will not be repeated here.

[0084] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An emotion recognition method based on facial expressions and human posture, characterized in that, Includes the following steps: The process involves acquiring facial expression images and human posture images to be identified, and performing preprocessing, including: the acquired facial expression images and human posture images are images of the same target person at the same time; a clear facial expression image of the target person with a large proportion of the face area is captured by one camera; and a human posture image of the target person's complete human torso, including the head, is captured by another camera; and then the facial expression images and human posture images are preprocessed, including grayscale transformation and filtering noise reduction. The Dlib face object detection network is used to detect the preprocessed facial expression image to obtain the face image, and the VGG facial expression emotion recognition network is used to detect the face image to obtain the facial expression emotion recognition result. The YOLO human target detection network is used to detect human body parts in the preprocessed human pose image to obtain human body parts. The AlphaPose human keypoint detection network is used to detect human keypoints in the human body parts image to obtain human keypoint information. The SE-GCN graph convolutional neural network with fusion attention mechanism is used to detect human keypoint information to obtain human pose emotion recognition results. Specifically, the human keypoint information is input into the SE-GCN graph convolutional neural network with fusion attention mechanism. After multiple graph convolutional layers using SENet graph attention module and at least one fully connected layer, the depth features of the human pose image are extracted. After classification by the softmax function, the human pose emotion recognition results are finally output. The facial expression emotion recognition results and the human posture emotion recognition results are fused to obtain the target person's emotion recognition results.

2. The emotion recognition method based on facial expressions and human posture according to claim 1, characterized in that, The process of using the Dlib face detection network to detect facial expression images in the preprocessed facial expression images to obtain face portion images includes: The Dlib face detection network based on ResNet is used to detect face objects in facial expression images, output the coordinates of the face bounding boxes in the image, and then the face part is cropped, resized and grayscaled to obtain a grayscale face image.

3. The emotion recognition method based on facial expressions and human posture according to claim 1, characterized in that, The method of using the VGG facial expression emotion recognition network to detect facial expressions in a partial image to obtain facial expression emotion recognition results includes: The face image is input into the VGG facial expression emotion recognition network. After multiple convolutional layers and multiple fully connected layers, the deep features of the face image are extracted. Then, the softmax function is used for classification, and the final output is the facial expression emotion recognition result.

4. The emotion recognition method based on facial expressions and human posture according to claim 1, characterized in that, The process of using the YOLO human target detection network to detect human body parts in the preprocessed human pose image to obtain human body images includes: The YOLO object detection network is used to detect human targets in human pose images, output the coordinates of the human bounding box in the image, and then the image within the human bounding box is cropped and resized to obtain the RGB human part image.

5. The emotion recognition method based on facial expressions and human posture according to claim 1, characterized in that, The method of using the AlphaPose human keypoint detection network to detect human keypoints in a human body image to obtain human keypoint information includes: The Alphapose human keypoint detection network is used to extract and output human keypoint information, including the coordinates and confidence scores of human keypoints, from human body images.

6. The emotion recognition method based on facial expressions and human posture according to claim 1, characterized in that, The process of fusing facial expression emotion recognition results and human posture emotion recognition results to obtain the target person's emotion recognition results includes: The facial expression emotion recognition results and the human posture emotion recognition results are integrated at the decision level, and the emotion recognition results of the target person are obtained through weighted calculation, and the recognized emotion category is output.

7. An emotion recognition device based on facial expressions and human posture, implemented using the emotion recognition method based on facial expressions and human posture as described in any one of claims 1-6, characterized in that, include: Image preprocessing module, facial expression emotion recognition module, human posture emotion recognition module, and multimodal fusion emotion recognition module; The image preprocessing module is used to acquire facial expression images and human posture images to be identified, and to perform preprocessing. The facial expression emotion recognition module is used to detect the preprocessed facial expression image through the Dlib face target detection network to obtain the face portion image, and to detect the face portion image through the VGG facial expression emotion recognition network to obtain the facial expression emotion recognition result. The human posture emotion recognition module is used to detect the preprocessed human posture image through the YOLO human target detection network to obtain the human body part image, to detect the human body part image through the AlphaPose human key point detection network to obtain human key point information, and to detect the human key point information through the SE-GCN graph convolutional neural network with fused attention mechanism to obtain the human posture emotion recognition result. The multimodal fusion emotion recognition module is used to fuse facial expression emotion recognition results and human posture emotion recognition results to obtain the emotion recognition result of the target person.

8. An emotion recognition device based on facial expressions and human posture, comprising a memory and one or more processors, wherein the memory is used to store computer programs, characterized in that, The processor is configured to implement the emotion recognition method based on facial expressions and human posture as described in any one of claims 1-6 when executing the computer program.

Images