A multimodal speech emotion recognition method based on enhanced residual neural network

Abstract

The invention discloses a multimodal speech emotion recognition method based on an enhanced depth residual neural network, which relates to the technical fields of video stream image processing, speech signal analysis and the like, and solves the emotion recognition problem of human-computer interaction. The invention mainly comprises the following steps: extracting the feature expression of video(sequence data) and speech, including converting the speech data into the corresponding speech spectrum expression, and encoding the time sequence data; Convolution neural network is used to extractthe emotional features of the original data for classification. The model accepts multiple inputs and the input dimensions are different. The cross-convolution layer is proposed to fuse the data features of different modes, and the overall network structure of the model is enhanced depth residual neural network. After the model is initialized, the multi-classification model is trained with speechspectrum, sequence video information and corresponding affective tagging. After the training, the unlabeled speech and video are predicted to obtain the probability value of affective prediction, andthe maximum value of probability is selected as the affective category of the multi-modal data. The invention improves the recognition accuracy on the problem of multi-modal emotion recognition.

Description

technical field [0001] A multi-modal speech emotion recognition method based on an enhanced deep residual neural network, which involves the technical fields of video stream image processing and speech signal analysis, and solves the problem of emotion recognition in human-computer interaction. Background technique [0002] With the rapid development of computer technology, human beings are increasingly dependent and demanding on computers. How to better realize the anthropomorphism of computers has become a research hotspot, and having "emotion" has become the research goal of the next generation of computers. Emotions can be conveyed through various means of communication, such as text, voice, video, etc. It is often impossible to understand emotions well through a single piece of information, so emotion recognition from multimodal data is currently the main field of vision for pattern recognition. [0003] Traditional multimodal research methods mainly rely on facial exp...

AI Technical Summary

Problems solved by technology

However, since speech and image information are often high-dimensional data, traditional computing methods cannot perform feature learning well.

Images