Multi-modal emotion recognition method based on fusion attention network

Abstract

The invention discloses a multi-modal emotion recognition method based on a fusion attention network. The method comprises: extracting high-dimensional features of three modes of text, vision and audio, and aligning and normalizing according to the word level; then, inputting the signals into a bidirectional gating circulation unit network for training; extracting state information output by the bidirectional gating circulation unit network in the three single-mode sub-networks to calculate the correlation degree of the state information among the multiple modes; calculating the attention distribution of the plurality of modalities at each moment; wherein the state information is the weight parameter of the state information at each moment; and weighting and averaging state information ofthe three modal sub-networks and the corresponding weight parameters to obtain a fusion feature vector as input of the full connection network, a to-be-identified text, inputting vision and audio intothe trained bidirectional gating circulation unit network of each modal, and obtaining final emotion intensity output. According to the method, the problem of weight consistency of all modes during multi-mode fusion can be solved, and the emotion recognition accuracy under multi-mode fusion is improved.

Description

technical field [0001] The present invention relates to the fields of text processing, audio processing, visual processing, feature extraction, deep learning, cyclic neural network, emotion recognition, etc., and in particular relates to a multi-modal emotion recognition method. Background technique [0002] Emotion recognition is a research hotspot in the field of natural language processing. The main challenge of emotion recognition is to be able to conduct continuous and real-time analysis of the speaker's emotion. Multimodal emotion recognition research has made great progress in a variety of tasks and has become an emerging research field of artificial intelligence. Recognizing human emotions using information such as human facial expressions, voice intonation, and body gestures is an interesting and challenging problem. In the research of multimodal emotion recognition involving video, text, vision and audio are often used as the main multimodal information. The purpo...

AI Technical Summary

Problems solved by technology

The irrationality of the above three methods of linear extraction and fusion of feature vectors lies in the weight consistency of each mode during multi-modal fusion, that is, selecting special state information from the state information output as the encoded state information, and only considering any one The state information itself will have a certain impact on the final emotional intensity output, without considering the importance of each state information is not the same

Images