A multi-modal emotion recognition method and system

Abstract

The present disclosure provides a multi-modal emotion recognition method and system. The scheme uses a novel and robust endpoint detection algorithm for speech components in emotional video samples, and utilizes the prediction residue generated during the sample reconstruction process under the compressed sensing theory. The difference conditional entropy parameter is used to calculate the difference value of the residual conditional entropy during the iterative process of the Orthogonal Matching Pursuit (OMP) algorithm, to complete the endpoint detection according to the empirical threshold, and to complete the feature learning of the emotional speech of the vocal segment based on the reconstruction sample; at the same time, through the emotional The endpoint detection results of the voice, the facial expression images are screened, and only the facial expression images with active emotional voices in the same time period are retained to achieve the purpose of enhancing the emotional separability of the facial expression data set and reducing redundancy; emotional voice Features and facial expression features are fused through features to train an effective multi-modal emotion recognition model to achieve effective multi-modal emotion recognition.

Description

technical field [0001] The present disclosure belongs to the technical field of emotion recognition, and in particular, relates to a multimodal emotion recognition method and system. Background technique [0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art. [0003] Emotion recognition is a research hotspot in the field of emotional computing. The emotional signals of the two modalities, emotional speech and facial expression images, have the characteristics of convenient collection and large amount of emotional information. They are important and highly correlated in emotion recognition research. Two types of data sources. [0004] The inventor found that the emotional speech data and facial expression images currently used in the multi-emotion recognition field are usually obtained by separately saving the emotional speech component and the image component in the emotion...

AI Technical Summary

Problems solved by technology

Under the slow-changing characteristics of facial expressions, this non-discrimination and non-filtering method of facial expression image collection ignores the connection between different modal emotional expressions, and does not consider the emotional differentiation of facial expression images, resulting in the collected The emotional recognizability of facial expression images is low and the redundancy is large, which leads to the poor performance of the model trained and learned in the follow-up emotion recognition research.

Images