Recognition method and system based on dual-modal emotion fusion of voice and facial expression

Abstract

The invention relates to a voice-and-facial-expression-based identification method for dual-modal emotion fusion. The method comprises: S1, audio data and video data of a to-be-identified object are obtained; S2, a face expression image is extracted from the video data and segmentation of an eye region, a nose region, and a mouth region is carried out; S3, a facial expression feature in each regional image is extracted from images of the three regions; S4, PCA analysis and dimensionality reduction is carried out on voice emotion features and the facial expression features; and S5, naive Bayesian emotion voice classification is carried out on samples of two kinds of modes and decision fusion is carried out on a conditional probability to obtain a final emotion identification result. According to the invention, fusion of the voice emotion features and the facial expression features is carried out by using a decision fusion method, so that accurate data can be provided for corresponding conditional probability calculation carried out at the next step; and an emotion state of a detected object can be obtained precisely by using the method, so that accuracy and reliability of emotion identification can be improved.

Description

technical field [0001] The invention belongs to the field of emotion recognition, and more specifically relates to a method and system for dual-mode emotion fusion and recognition based on speech and facial expressions. Background technique [0002] With the improvement of living standards and the rapid development of information technology, people's demand for intelligent life is getting higher and higher, and the ability of human-computer interaction is getting more and more attention. As an important part of human-computer interaction, affective computing has become a research topic. hotspot. At present, the research on emotion recognition at home and abroad is mainly divided into two categories, one is emotion recognition based on single modality, and the other is emotion recognition based on multi-modality. The main difference between the two types of emotion recognition is that multi-modal emotion recognition analyzes the information collected by multiple channels, an...

AI Technical Summary

Problems solved by technology

[0003] Existing emotion recognition is mainly based on single-modal emotion recognition. Most of the research on facial expression recognition is still on the emotion recognition of basic facial expressions, and the recognition effect on more subtle expressions is not good.

The research on speech emotion recognition is relatively mature, but when performing single-modal emotion recognition for speech, if the speech channel is blocked, the result of emotion recognition will be greatly affected

[0004] Ye Liang et al. proposed a speech feature screening method for mixed speech emotion recognition, which can well extract the best feature set in a series of acoustic features, but does not consider the situation where the acoustic channel is blocked

Zhao Xiaoming et al. proposed a robust speech emotion recognition method based on compressed sensing, which expanded the extraction of feature parameters from prosodic features and sound quality features to Mel frequency cepstral coefficient MFCC, which improved the anti-interference performance of feature signals, but still Unresolved emotion recognition when the acoustic channel cannot acquire the signal

Images