Multi-modal emotion recognition method based on attention enhancing mechanism

Abstract

The invention belongs to the technical field of emotion calculation and relates to a multi-modal emotion recognition method based on an attention enhancement mechanism. The method comprises steps of obtaining a voice coding matrix through a multi-head attention mechanism, and obtaining a text coding matrix through a pre-trained BERT model; performing point multiplication on the coding matrixes ofthe voice and the text respectively to obtain alignment matrixes of the voice and the text, and calibrating the alignment matrixes with original modal coding information to obtain more local interaction information; and finally, splicing the coding information, the semantic alignment matrix and the interaction information of each mode as features to obtain a feature matrix of each mode; aggregating the voice feature matrix and the text feature matrix by using a multi-head attention mechanism; converting the aggregated feature matrix into vector representation through an attention mechanism; and splicing the vector representations of the voice and the text, and obtaining a final emotion classification result by using a full connection network. According to the method, a problem of multi-modal interaction is solved, and accuracy of multi-modal emotion recognition is improved.

Description

technical field

[0001] The invention belongs to the technical field of emotion computing, in particular to a multimodal emotion recognition method based on an enhanced attention mechanism. Background technique

[0002] As early as 1995, the concept of affective computing has been proposed. Affective computing aims to endow machines with the ability to observe, understand and express various emotions. In recent years, although we have made great progress in image processing, speech recognition, and natural speech understanding, there is still an insurmountable gap to establish a highly harmonious human-computer interaction environment. Modeling the complex emotional expressions of humans is very challenging, but it is also the most fundamental problem of human-computer interaction that needs to be solved urgently.

[0003] With the continuous development of social networks, people express their emotions in more and more diverse forms. The traditional single emotion recogniti...

Images