Character emotion recognition method and device in video

Abstract

The invention provides a character emotion recognition method and device in a video. The method comprises the following steps: extracting a face image in a video and a sound spectrogram and a subtitletext corresponding to the face image, extracting to obtain an image feature vector, a sound feature vector and a text feature vector, carrying out feature fusion on the image feature vector, the sound feature vector and the text feature vector to obtain a multi-modal joint feature vector; compared with the face image feature, the sound feature vector and the text feature vector are increased in the multi-modal joint feature vector, the richness of the character emotion features is improved, the joint feature vectors are input into a character emotion recognition model obtained by training atleast one machine learning model through a joint feature vector training data set in advance for recognition processing, and a more accurate character emotion recognition result can be obtained.

Description

technical field [0001] The present invention relates to the technical field of video data analysis, and more specifically, to a method and device for character emotion recognition in videos. Background technique [0002] Character emotion recognition is an important part of human-computer interaction and affective computing research. Common and important emotion categories in videos include happiness, anger, disgust, fear, sadness, surprise, etc. Emotion is an important part of video content. By identifying emotion, the emotion and emotion expressed in video clips can be analyzed to derive emotion-related video applications. [0003] Most of the existing video emotion recognition technologies focus on the way of emotion recognition based on the visual features of the face, that is, through face detection to locate, analyze and identify the image of the face area, and classify the emotion according to the visual features of the image of the face area. The visual features of...

AI Technical Summary

Problems solved by technology

The visual features of the face area image are indeed the visual features that can best reflect the facial emotion, but there are many interference factors in the face image in the video, such as blurred face image, poor lighting conditions, angle deviation, etc. Therefore, based on the visual feature The accuracy of character emotion recognition is low

Images