Multi-modal emotion recognition method based on feature-time attention mechanism

Abstract

The invention discloses a multi-modal emotion recognition method based on a feature-time attention mechanism, and the method comprises the following steps: constructing a neural network model, and obtaining an audio / video sample containing emotion information and a video primary feature matrix; acquiring an audio primary feature matrix; obtaining a fusion feature matrix according to the video primary feature matrix and the audio primary feature matrix, and inputting the fusion feature matrix into a feature self-attention mechanism module; inputting the processed fusion feature matrix into a bidirectional gating cycle unit network to obtain output vectors at all moments and a state vector of the last hidden layer; obtaining an attention weight, and obtaining an advanced feature vector according to the attention weight; obtaining a neural network model capable of performing sentiment classification on audio and video samples after training; and collecting to-be-detected audios and videos, and inputting the to-be-detected audios and videos into the trained neural network model to obtain an emotion classification result. According to the invention, the accuracy of face emotion recognition in audios and videos can be improved.

Description

technical field [0001] The invention relates to the technical field of pattern recognition, in particular to a multimodal emotion recognition method based on a feature-time attention mechanism. Background technique [0002] As an important basis of human life experience, emotion affects human cognition, perception and daily life. In 1971, psychologists Ekman and Friesen divided human emotions into six basic emotional categories through cross-cultural research, which are happy (Happy), sad (Sad), surprised (Surprise), angry (Angry), fear ( Fear) and Disgust (Disgust), these six types of emotion categories are universal, and more fine-grained sub-level emotion categories can be synthesized on this basis. In 1997, Professor Picard first proposed the concept of "affective computing", which involves psychology, cognition, pattern recognition, speech signal processing, physiology, sociology, computer vision, and artificial intelligence. Facial expressions, voice and other inform...

AI Technical Summary

Problems solved by technology

Decision fusion is to make a final judgment on the final result with certain rules after the emotion recognition results are obtained by each modal model, which has high flexibility and strong real-time performance. Less volume, lower relative accuracy

Images