Multi-modal emotion recognition method based on time domain convolutional network

Abstract

The invention discloses a multi-modal emotion recognition method based on a time domain convolutional network, and the method comprises the steps: carrying out the interval sampling of video modal data in an audio and video sample, carrying out the face detection and key point positioning, and obtaining a gray face image sequence; performing short-time Fourier transform and obtaining a Mel spectrogram through a Mel filter bank; enabling the gray-scale face image sequence and the Mel spectrogram to pass through a face image convolutional network and a spectrogram image convolutional network respectively, and performing feature fusion; inputting the fusion feature sequence into a time domain convolutional network to obtain an advanced feature vector; the advanced feature vectors are subjected to a full connection layer and Softmax regression, the prediction probability of each emotion category is obtained, cross entropy loss is calculated between the prediction probability and actual probability distribution, the whole network is trained through back propagation, and a trained neural network model is obtained. The emotion can be predicted through audios and videos, the training time is short, and the recognition accuracy is high.

Description

technical field [0001] The present invention relates to audio and video processing, pattern recognition, and deep learning technology, and in particular to a multimodal emotion recognition method based on time-domain convolutional network. Background technique [0002] In 1997, Professor Picard first proposed the concept of "affective computing", which involves psychology, cognition, pattern recognition, speech signal processing, physiology, sociology, computer vision, and artificial intelligence. Facial expressions, voice and other information to identify the emotional state shown by humans, so that the machine can better understand human emotions and behaviors, so as to bring a smoother and more efficient interactive experience. Multimodal emotion recognition aims to use expression and voice modal information to identify basic human emotions, which are generally divided into 6 categories, which are happy (Happy), sad (Sad), surprised (Surprise), angry (Angry), Fear (Fear)...

AI Technical Summary

Problems solved by technology

Decision fusion is to make a final judgment on the final result with certain rules after the emotion recognition results are obtained by each modal model, which has high flexibility and strong real-time performance. Less volume, lower relative accuracy

Images