Multimodal emotion recognition method and system based on neural network and transfer learning

Abstract

The invention relates to the field of multi-mode emotion calculation, discloses a neural network and transfer learning-based multi-mode emotion recognition method and system, and aims at solving the problems that corresponding recognition models cannot be sufficiently trained and multi-mode emotion recognition correctness cannot satisfy requirements as emotion data is difficult to obtain and label. The method comprises the following steps of: obtaining an audio feature extractor and a video feature extractor through transfer learning on the basis of a large-scale data training deep neural network; extracting audio features and video features of multi-mode emotion data so as to recognize a probability of each voice emotion category and a probability of each video emotion category; and judging a final emotion category through probability values. The method is capable of effectively fusing two modes of audio and video, so as to improve the correctness of multi-mode emotion recognition.

Description

technical field [0001] The invention belongs to the field of multimodal emotion computing, and in particular relates to a multimodal emotion recognition method and system based on neural network and transfer learning. Background technique [0002] Affective computing is trying to endow machines with the ability to observe, understand and generate various emotions. The purpose of its research is to explore and understand the role of emotion in biological weight, and to propose corresponding models and methods to establish the emotional capabilities of machines. , to enhance their autonomy, adaptability and sociability. Affective computing is an important research direction in the field of expressive human-computer interaction and artificial intelligence, involving multiple fields such as intelligence science, mathematics, neurology, and physiological science. [0003] Emotion recognition mainly includes two steps of feature extraction and classifier classification. There is...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the above-mentioned problems in the prior art, that is, in order to solve the problem that the emotional data is difficult to obtain and label, so that the corresponding recognition model cannot be fully trained, resulting in the problem that the accuracy of multi-modal emotion recognition cannot meet the demand, one aspect of the present invention, A multimodal emotion recognition method based on neural network and transfer learning is proposed, including the following steps:

Images