Emotion recognition method, system and device based on multi-modal feature fusion and medium

Abstract

The invention discloses an emotion recognition method, system and device based on multi-modal feature fusion and a medium, and the method comprises the steps: obtaining preset first voice information and corresponding first visual information, and carrying out the feature extraction of the first voice information and the first visual information, and obtaining a voice feature image and an expression feature image; performing feature fusion on the voice feature image and the expression feature image to obtain a first multi-modal feature, and constructing a training data set according to the first multi-modal feature; inputting the training data set into a pre-constructed convolutional neural network for training to obtain a trained multi-modal feature recognition model; and identifying the emotion of the person to be tested according to the multi-modal feature identification model. On one hand, the model complexity is reduced, the model training and emotion recognition efficiency is improved, on the other hand, the influence of the voice features and the expression features on the emotion recognition result of the model is considered, the emotion recognition accuracy is improved, and the emotion recognition method can be widely applied to the technical field of emotion recognition.

Description

technical field [0001] The present invention relates to the technical field of emotion recognition, in particular to an emotion recognition method, system, device and medium based on multimodal feature fusion. Background technique [0002] Emotion recognition is an important part of realizing full human-computer interaction. Emotion recognition can be applied in many different fields. For example, emotion recognition can be used to monitor and predict fatigue status. The task of emotion recognition is challenging because human emotions lack temporal boundaries and different people express emotions in different ways. Despite the current rich experience in emotion recognition for inferring the subject's emotion from speech or other forms, such as visual information (facial gestures), the accuracy of single-modal emotion recognition is not high, and the generalization ability is poor . [0003] With the advent of deep neural networks in the past decade, there have been many b...

AI Technical Summary

Problems solved by technology

However, in existing technologies, multiple neural network models are often used to perform emotion recognition on speech, expression and other features, and then make comprehensive judgments based on the respective recognition results. On the one hand, this method needs to train multiple recognition models for each The characteristics of different types are recognized, the model complexity is high, and the recognition efficiency is low. On the other hand, each feature is recognized separately, and the influence of the association between each feature on the emotion recognition result is not considered, so the accuracy of emotion recognition is relatively low. Low

Images