Emotion recognition method and device based on facial expressions and human body postures
By combining Dlib, VGG, Yolo, AlphaPose and SE-GCN networks to extract facial expressions and human posture characteristics, and fuse information for emotion recognition, the problem of insufficient research on bimodal emotion recognition and low recognition accuracy in the existing technology is solved, and the problem of insufficient recognition research and low recognition accuracy is achieved. Higher accuracy and robustness.
Patent Information
- Application Number
- CN202411022205.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-29
AI Technical Summary
In the prior art, there are few researches on bimodal emotion recognition based on facial expressions and human postures. In real scenes, the results of single-modal facial feature recognition are low and robust. It is poor in nature and is difficult to make effective judgments when the modal information is missing.
By obtaining facial expressions and human posture images for preprocessing, the SE-GCN picture volume using Dlib face object detection network, VGG facial expression emotion recognition network, Yolo human object detection network, AlphaPose human key point detection network and SE-GCN picture volume of fusion attention mechanism using Dlib face object detection network, VGG facial expression emotion recognition network, Yolo human object detection network, AlphaPose human body key point detection network and SE-GCN picture volume of fusion attention mechanism Accumulate neural networks to extract and fuse facial expressions and human posture feature information for emotional recognition.
It improves the accuracy and robustness of emotion recognition, can accurately identify faces and human postures in complex backgrounds, enhances the ability to discriminate when modal information is missing, and provides richer emotional characteristics.
Smart Images

Figure CN119027997A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of emotion computing of computer vision, and in particular relates to an emotion recognition method and device based on facial expressions and human body postures. Background Art
[0002] Emotion recognition refers to the process of computer preprocessing and feature extraction of emotional signals and automatically identifying the internal emotional types of human beings, which belongs to the category of emotional computing. Emotional computing involves many fields such as social psychology, cognitive science, neurology, physiology and computer science. It aims to establish a harmonious human-computer interaction environment and make computers more intelligent by giving computers the ability to recognize, understand and simulate human emotions. Among them, emotion recognition is one of the most basic and important contents of all research.
[0003] Emotion recognition methods generally include the steps of raw image input, data preprocessing, manual feature extraction and emotion classification, among which feature extraction is the most important step. Traditional methods use manual feature extraction, but in actual application scenarios, it is difficult to process data in real scenarios due to the large dimension of emotion features and complex calculations. With the rapid development of deep learning technology, multi-layer neural networks are used to extract deep feature information, and the effect of emotion recognition has been significantly improved.
[0004] The external manifestations of human emotional expression include changes in facial expressions, body posture, voice intonation and other features, while the internal manifestations include changes in physiological signals such as heart rate, blood pressure, and EEG, indicating that the expression of emotion is multimodal in nature.
[0005] Facial expressions are the strongest external manifestations of emotions. At present, most emotion recognition methods only focus on facial expression image information, and recognize emotions by extracting facial expression features, and have achieved certain results. However, in real scenes, due to limitations such as lighting and distance, the faces in the images are often small, and the accuracy of facial features is low, which has a certain impact on the recognition results. At the same time, studies have shown that facial expressions are easily confused, such as sadness, fear, disgust, etc. Therefore, there are limitations to the results obtained by relying solely on single-modal facial features to judge a person's emotional state.
[0006] Human body posture is also an external expression of emotion. When people express their emotions, they often have related actions, that is, body language. For example, when people are happy, they will raise their heads and chests, and their bodies will be stretched out. When they are sad, they will lower their heads, lower their hands, and their bodies will be contracted. In real scenes, human body posture is easier to collect than facial expressions, and the changes are more obvious, so it can provide rich emotional characteristics.
[0007] There are relatively few studies on dual-modal emotion recognition based on facial expressions and human postures. Compared with emotion recognition methods under a single modality, dual-modal emotion recognition using facial expressions and human postures can make full use of the information of expressions and postures to obtain more accurate results. At the same time, when faced with the lack of information in one modality, such as when the face is occluded, other modal information can be used to discriminate the target person's emotions, which enhances the robustness of the method. Therefore, it is necessary to strengthen the in-depth research on dual-modal emotion recognition technology based on facial expressions and human postures to further improve the accuracy of emotion recognition. Summary of the invention
[0008] In view of the above, the purpose of the present invention is to provide a method and device for emotion recognition based on facial expressions and human postures, which achieves emotion recognition with high accuracy by performing feature analysis and fusion on facial expressions and human posture images of target persons.
[0009] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:
[0010] In a first aspect, an embodiment of the present invention provides an emotion recognition method based on facial expression and human posture, comprising the following steps:
[0011] Obtain facial expression images and human posture images to be recognized and perform preprocessing;
[0012] The Dlib face target detection network is used to detect the preprocessed facial expression image to obtain the partial face image, and the VGG facial expression emotion recognition network is used to detect the partial face image to obtain the facial expression emotion recognition result;
[0013] The preprocessed human posture image is detected using the Yolo human target detection network to obtain a partial human image. The partial human image is detected using the AlphaPose human key point detection network to obtain human key point information. The SE-GCN graph convolutional neural network with an attention mechanism is used to detect human key point information to obtain human posture emotion recognition results.
[0014] The facial expression emotion recognition results and the human posture emotion recognition results are fused to obtain the target person's emotion recognition results.
[0015] Specifically, the acquiring of facial expression images and human posture images to be recognized and preprocessing thereof include:
[0016] The acquired facial expression image and body posture image are images of the same target person to be identified at the same moment. A clear facial expression image of the target person with a large face area is obtained by one camera, and a body posture image of the target person's complete torso including the head is obtained by another camera. Then, the facial expression image and the body posture image are preprocessed including grayscale transformation and filtering noise reduction.
[0017] Specifically, the method of using the Dlib face target detection network to detect the preprocessed facial expression image to obtain a partial face image includes:
[0018] Use the Dlib face target detection network based on ResNet to detect the face target in the facial expression image, output the coordinates of the face bounding box in the image, and then crop, resize and grayscale the face part of the picture to obtain a grayscale face part image.
[0019] Specifically, the method of using the VGG facial expression emotion recognition network to detect a partial face image to obtain a facial expression emotion recognition result includes:
[0020] The partial face image is input into the VGG facial expression emotion recognition network. After multiple convolutional layers and multiple fully connected layers to extract the deep features of the face image, it is classified by the softmax function and finally the facial expression emotion recognition result is output.
[0021] Specifically, the method of using the Yolo human target detection network to detect the preprocessed human posture image to obtain a human body part image includes:
[0022] The Yolo target detection network is used to detect human targets in human posture images, output the coordinates of the human bounding box in the image, and then crop and resize the image within the human bounding box to obtain an RGB human part image.
[0023] Specifically, the method of using the AlphaPose human key point detection network to detect a human body part image to obtain human body key point information includes:
[0024] Use the Alphapose human key point detection network to extract and output human key point information including the coordinates and confidence of the human key points from the human body part image.
[0025] Specifically, the SE-GCN graph convolutional neural network using the fusion attention mechanism detects the key point information of the human body to obtain the human posture emotion recognition result, including:
[0026] The key point information of the human body is input into the SE-GCN graph convolutional neural network with integrated attention mechanism. After multiple layers of graph convolutional layers using the SENet graph attention module and at least one fully connected layer, the deep features of the human posture picture are extracted. After softmax function classification, the human posture emotion recognition result is finally output.
[0027] Specifically, the facial expression emotion recognition result and the human body posture emotion recognition result are integrated to obtain the target person emotion recognition result, including:
[0028] The facial expression emotion recognition results and the human posture emotion recognition results are fused at the decision layer, and the emotion recognition results of the target person are obtained through weighted calculation, and the recognized emotion category is output.
[0029] In a second aspect, in order to achieve the above-mentioned invention object, an embodiment of the present invention further provides an emotion recognition device based on facial expression and human posture, comprising: an image preprocessing module, a facial expression emotion recognition module, a human posture emotion recognition module and a multimodal fusion emotion recognition module;
[0030] The image preprocessing module is used to obtain facial expression images and human posture images to be recognized, and perform preprocessing;
[0031] The facial expression emotion recognition module is used to detect the preprocessed facial expression image through the Dlib face target detection network to obtain a partial face image, and to detect the partial face image through the VGG facial expression emotion recognition network to obtain a facial expression emotion recognition result;
[0032] The human posture emotion recognition module is used to detect the preprocessed human posture image through the Yolo human target detection network to obtain a human body partial image, detect the human body partial image through the AlphaPose human key point detection network to obtain human body key point information, and detect the human body key point information through the SE-GCN graph convolutional neural network that integrates the attention mechanism to obtain a human posture emotion recognition result;
[0033] The multimodal fusion emotion recognition module is used to fuse the facial expression emotion recognition results and the human body posture emotion recognition results to obtain the target person's emotion recognition results.
[0034] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides an emotion recognition device based on facial expressions and human body postures, comprising: a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned emotion recognition method based on facial expressions and human body postures when executing the computer program.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] (1) When performing feature extraction and emotion recognition on facial expression images, the present invention can improve the robustness of face detection in complex backgrounds by using the Dlib face target detection network, and can enhance the accuracy of emotion recognition by extracting multi-level deep features of partial face images by using the VGG facial expression emotion recognition network.
[0037] (2) When performing feature extraction and emotion recognition on human posture images, the present invention can quickly and accurately identify human body areas by using the Yolo human target detection network, and can accurately capture the information of each key point of the human body by using the AlphaPose human key point detection network, providing a rich data foundation for subsequent emotion recognition. The SE-GCN graph convolutional neural network constructed by combining the attention mechanism can improve the network's ability to focus on important nodes, thereby obtaining more accurate emotion recognition results.
[0038] (3) The present invention fuses the emotion recognition results under the two modes by using the decision layer fusion method, which can make full use of the information of expression and posture and further improve the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0040] Figure 1 is a flow chart of an emotion recognition method based on facial expression and human body posture provided by an embodiment of the present invention;
[0041] Figure 2 is a schematic diagram of a framework of an emotion recognition method based on facial expression and human posture provided by an embodiment of the present invention;
[0042] Figure 3 It is a schematic diagram of the structure of an emotion recognition device based on facial expression and human posture provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0043] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.
[0044] The inventive concept of the present invention is: in view of the problem that the research on dual-modal emotion recognition based on facial expression and human posture in the prior art is relatively small and the accuracy needs to be improved, the embodiment of the present invention provides an emotion recognition method and device based on facial expression and human posture, combines the Dlib face target detection network and the VGG facial expression emotion recognition network to perform facial expression emotion recognition, combines the Yolo human target detection network, the AlphaPose human key point detection network and the SE-GCN graph convolutional neural network with the fusion attention mechanism to perform human posture emotion recognition, and fuses the facial expression emotion recognition results and the human posture emotion recognition results to obtain the final emotion category, thereby realizing dual-modal emotion recognition, which can effectively improve the accuracy of emotion recognition.
[0045] Figure 1 is a flow chart of an emotion recognition method based on facial expression and human posture provided by an embodiment of the present invention, Figure 2 Schematic diagram of the framework of the emotion recognition method based on facial expression and human posture provided by the embodiment of the present invention. Figure 1 and Figure 2 As shown, the embodiment provides an emotion recognition method based on facial expression and human body posture, comprising the following steps:
[0046] S1, obtain facial expression images and human posture images to be recognized, and perform preprocessing.
[0047] Specifically, the acquired facial expression image and body posture image are images of the same target person to be identified at the same moment. A facial expression image with a large and clear face area of the target person is obtained by one camera, and a body posture image of the target person's complete torso including the head is obtained by another camera.
[0048] Preprocess facial expression images and human posture images, including grayscale transformation and filtering noise reduction.
[0049] (1.1) Grayscale transformation: Gamma transformation is used to enhance the low brightness area of the image. The calculation method of Gamma transformation is as follows:
[0050] s=cr γ
[0051] Wherein, s is the output value after Gamma transformation, c is a constant, in the embodiment, c is 1, r is the original brightness value, γ is the Gamma factor, in the embodiment, γ is 0.85. If the input is a grayscale image, the brightness value of the pixel is Gamma transformed, and if the input is an RGB image, the R, G, and B components of the pixel are Gamma transformed respectively.
[0052] (1.2) Filtering and noise reduction: Use Gaussian filtering to remove image noise. The Gaussian function G(x,y) is as follows:
[0053]
[0054] Wherein, σ is the standard deviation, x and are the pixel coordinates of the image row and column respectively, and in the embodiment, σ is set to 1.5 and discretized into a 5×5 Gaussian convolution kernel. Since the Gaussian function is linearly separable, in the embodiment, the convolution calculation of the input image and the two-dimensional Gaussian convolution kernel is equivalently transformed into first calculating the convolution result G(x) of each row of pixels and the one-dimensional Gaussian convolution kernel, and then calculating the convolution result G(y) of each column of pixels of the result and the one-dimensional Gaussian convolution kernel, so as to achieve the goal of accelerating the calculation.
[0055] S2, use the Dlib face target detection network to detect the preprocessed facial expression image to obtain a partial face image, and use the VGG facial expression emotion recognition network to detect the partial face image to obtain the facial expression emotion recognition result.
[0056] (2.1) Use the Dlib face detection network based on ResNet to detect face targets in facial expression images. The Dlib face detection network is trained using datasets such as FaceScrub and VGG, and contains a total of 29 convolutional layers. Output the coordinates of the face bounding box in the image, then crop, resize, and grayscale the image within the face bounding box to obtain a grayscale face image of 48×48 pixels.
[0057] (2.2) Use the VGG-19 facial expression emotion recognition network to detect the face part. The VGG-19 facial expression emotion recognition network is trained using the FER2013 dataset. It takes a 48×48 pixel grayscale face part image as input, extracts the deep feature information of the face image through 16 convolutional layers and 3 fully connected layers, and then uses the softmax function for classification. Finally, it outputs a discrete emotion prediction value as the facial expression emotion recognition result.
[0058] S3, uses the Yolo human target detection network to detect the preprocessed human posture image to obtain the human body partial image, uses the AlphaPose human key point detection network to detect the human body partial image to obtain the human body key point information, and uses the SE-GCN graph convolutional neural network with fusion attention mechanism to detect the human body key point information to obtain the human posture emotion recognition result.
[0059] (3.1) Use the Yolov5 target detection network to detect human targets in human pose images. The Yolov5 target detection network is trained using the COCO dataset. After single-stage target detection and non-maximum suppression, it outputs the coordinates of the human bounding box in the image. Then, the image within the human bounding box is cropped and resized, and the size of the human area is adjusted to a 256×192 pixel RGB human part image.
[0060] (3.2) Use the Alphapose human key point detection network, which is a FastPose-DCN model based on ResNet50-DCN, to detect human parts. The Alphapose human key point detection network takes a 256×192 pixel RGB human part image as input, and outputs 136 human key point information coordinates (x ) including the head, shoulders, elbows, wrists, hips, knees, and ankles through a symmetric space transformation network, a single-person pose estimation network, and non-maximum suppression. i ,y i ) and confidence f i , where i is the index of the key point of the human body.
[0061] (3.3) Use the SE-GCN graph convolutional neural network with fusion attention mechanism to detect the key point information of the human body. The SE-GCN graph convolutional neural network with fusion attention mechanism includes 3 layers of GCN graph convolutional layers using SENet graph attention module and 1 layer of fully connected layer to extract the deep features of the human posture picture, and finally output the discrete emotion prediction value as the human posture emotion recognition result after softmax function classification.
[0062] The main architecture of SE-GCN is based on GCN, and a graph attention module based on Squeeze-Excitation Networks (SENet) is added before each GCN convolutional layer to improve the network's ability to focus on important nodes, thereby obtaining more accurate emotion recognition results.
[0063] For the i-th layer of SE-GCN, the input is the feature map Its adjacency matrix Where N i is the number of nodes in the graph, D i is the number of feature dimensions for each node. The SENet graph attention module first i Use the compression operation F sq (·) Calculate descriptors Get the compressed global features. i The jth element z ij The calculation method is:
[0064]
[0065] Get descriptor z i After that, the excitation operation F is performed ex (·,W), get the descriptor Represents the weight of each dimension feature:
[0066] s i =F ex (z i ,W)=σ(W2δ(W1z i ))
[0067] in, and are two linear transformation matrices, r is a hyperparameter, which is set to reduce the number of parameters and complexity of the operation, σ(·) is the sigmoid activation function, and δ(·) is the ReLU activation function.
[0068] Use the attention weight s i X i Recalibrate and get the recalibrated feature matrix The jth element The calculation method is:
[0069] GCN convolutional layer usage Perform graph convolution calculation to obtain the feature graph X of the i+1th layer i+1 :
[0070]
[0071] in, I is the identity matrix, for The degree matrix of is the weight matrix of the i-th layer.
[0072] S4, fusing the facial expression emotion recognition results and the human body posture emotion recognition results to obtain the target person emotion recognition results.
[0073] The facial expression emotion recognition results and the human posture emotion recognition results are integrated at the decision layer, and the emotion recognition results of the target person are obtained through weighted calculation:
[0074] p i =αp i,face +βp i,body
[0075] Among them, p i is the confidence of the i-th emotion, p i,face is the confidence level of the facial expression emotion of the i-th category, pi,face is the confidence of the i-th human posture emotion, α and β are weight factors, and α+β=1, and the final output is the recognized emotion category.
[0076] In summary, an emotion recognition method based on facial expression and human posture improves the recognition speed and accuracy by performing facial expression emotion recognition and human posture emotion recognition based on each network respectively, and then extracts features from the two modalities and performs feature fusion to realize dual-modal emotion recognition. Compared with traditional emotion recognition methods such as single modality, it can effectively improve the accuracy of emotion recognition.
[0077] Based on the same inventive concept, Figure 3 As shown, an embodiment of the present invention further provides an emotion recognition device 300 based on facial expression and human posture, comprising: an image preprocessing module 310, a facial expression emotion recognition module 320, a human posture emotion recognition module 330 and a multimodal fusion emotion recognition module 340.
[0078] The image preprocessing module 310 is used to obtain facial expression images and human posture images to be recognized and perform preprocessing.
[0079] The facial expression emotion recognition module 320 is used to detect the preprocessed facial expression image through the Dlib face target detection network to obtain a partial face image, and to detect the partial face image through the VGG-19 facial expression emotion recognition network to obtain a facial expression emotion recognition result.
[0080] The human posture emotion recognition module 330 is used to detect the preprocessed human posture image through the Yolov5 human target detection network to obtain a partial human body image, detect the partial human body image through the AlphaPose human key point detection network to obtain human body key point information, and detect the human body key point information through the SE-GCN graph convolutional neural network that integrates the attention mechanism to obtain a human posture emotion recognition result.
[0081] The multimodal fusion emotion recognition module 340 is used to fuse the facial expression emotion recognition results and the human posture emotion recognition results to obtain the target person's emotion recognition results.
[0082] Based on the same inventive concept, an embodiment of the present invention also provides an emotion recognition device based on facial expressions and human body postures, comprising a memory and one or more processors, the memory being used to store computer programs, and the processor being used to implement the above-mentioned emotion recognition method based on facial expressions and human body postures when executing the computer program.
[0083] It should be noted that the emotion recognition apparatus based on facial expressions and human body postures and the emotion recognition device based on facial expressions and human body postures provided in the above-mentioned embodiments belong to the same inventive concept as the emotion recognition method based on facial expressions and human body postures. The specific implementation process is detailed in the embodiment of the emotion recognition method based on facial expressions and human body postures, which will not be repeated here.
[0084] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An emotion recognition method based on facial expression and human posture, characterized in that: The following steps are involved: Obtain facial expression images and human posture images to be recognized and perform preprocessing; The Dlib face target detection network is used to detect the preprocessed facial expression image to obtain the partial face image, and the VGG facial expression emotion recognition network is used to detect the partial face image to obtain the facial expression emotion recognition result; The preprocessed human posture image is detected using the Yolo human target detection network to obtain a partial human image. The partial human image is detected using the AlphaPose human key point detection network to obtain human key point information. The SE-GCN graph convolutional neural network with an attention mechanism is used to detect human key point information to obtain human posture emotion recognition results. The facial expression emotion recognition results and the human posture emotion recognition results are fused to obtain the target person's emotion recognition results.
2. The emotion recognition method based on facial expression and human posture according to claim 1, characterized in that: The step of obtaining the facial expression image and the human body posture image to be recognized and performing preprocessing includes: The acquired facial expression image and body posture image are images of the same target person to be identified at the same moment. A clear facial expression image of the target person with a large face area is obtained by one camera, and a body posture image of the target person's complete torso including the head is obtained by another camera. Then, the facial expression image and the body posture image are preprocessed including grayscale transformation and filtering noise reduction.
3. The emotion recognition method based on facial expression and human posture according to claim 1, characterized in that: The method of using the Dlib face target detection network to detect the preprocessed facial expression image to obtain a partial face image includes: Use the Dlib face target detection network based on ResNet to detect the face target in the facial expression image, output the coordinates of the face bounding box in the image, and then crop, resize and grayscale the face part of the picture to obtain a grayscale face part image.
4. The emotion recognition method based on facial expression and human posture according to claim 1, characterized in that: The method of using the VGG facial expression emotion recognition network to detect a partial face image to obtain a facial expression emotion recognition result includes: The partial face image is input into the VGG facial expression emotion recognition network. After multiple convolutional layers and multiple fully connected layers to extract the deep features of the face image, it is classified by the softmax function and finally the facial expression emotion recognition result is output.
5. The emotion recognition method based on facial expression and human posture according to claim 1, characterized in that: The method of using the Yolo human target detection network to detect the preprocessed human posture image to obtain a human body part image includes: The Yolo target detection network is used to detect human targets in human posture images, output the coordinates of the human bounding box in the image, and then crop and resize the image within the human bounding box to obtain an RGB human part image.
6. The emotion recognition method based on facial expression and human posture according to claim 1, characterized in that: The method of using the AlphaPose human key point detection network to detect a human body part image to obtain human key point information includes: Use the Alphapose human key point detection network to extract and output human key point information including the coordinates and confidence of the human key points from the human body part image.
7. The emotion recognition method based on facial expression and human posture according to claim 1, characterized in that: The SE-GCN graph convolutional neural network with fusion attention mechanism is used to detect the key point information of the human body to obtain the human posture emotion recognition result, including: The key point information of the human body is input into the SE-GCN graph convolutional neural network with integrated attention mechanism. After multiple layers of graph convolutional layers using the SENet graph attention module and at least one fully connected layer, the deep features of the human posture picture are extracted. After softmax function classification, the human posture emotion recognition result is finally output.
8. The emotion recognition method based on facial expression and human posture according to claim 1, characterized in that: The facial expression emotion recognition results and the human body posture emotion recognition results are integrated to obtain the target person emotion recognition results, including: The facial expression emotion recognition results and the human posture emotion recognition results are fused at the decision layer, and the emotion recognition results of the target person are obtained through weighted calculation, and the recognized emotion category is output.
9. An emotion recognition device based on facial expression and human posture, characterized in that: include: Image preprocessing module, facial expression emotion recognition module, human posture emotion recognition module and multimodal fusion emotion recognition module; The image preprocessing module is used to obtain facial expression images and human posture images to be recognized, and perform preprocessing; The facial expression emotion recognition module is used to detect the preprocessed facial expression image through the Dlib face target detection network to obtain a partial face image, and to detect the partial face image through the VGG facial expression emotion recognition network to obtain a facial expression emotion recognition result; The human posture emotion recognition module is used to detect the preprocessed human posture image through the Yolo human target detection network to obtain a human body partial image, detect the human body partial image through the AlphaPose human key point detection network to obtain human body key point information, and detect the human body key point information through the SE-GCN graph convolutional neural network that integrates the attention mechanism to obtain a human posture emotion recognition result; The multimodal fusion emotion recognition module is used to fuse the facial expression emotion recognition results and the human body posture emotion recognition results to obtain the target person's emotion recognition results.
10. An emotion recognition device based on facial expression and human posture, comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the emotion recognition method based on facial expression and human body posture described in any one of claims 1-8 when executing the computer program.
Citation Information
Patent Citations
Facial expression recognition method based on Dlib and artificial neural network
CN112560730A
Facial expression and posture fused emotion recognition method and system
CN117036877A
Cited By
Pet posture and expression tracking method and device, electronic equipment and storage medium
CN120260080A
Pet posture and expression tracking method and device, electronic equipment and storage medium
CN120260080B
