Attention mechanism-based multi-modal emotion feature learning and recognition method

Abstract

The invention relates to an attention mechanism-based multi-modal emotion feature learning and recognition method, and the method comprises the steps: carrying out the feature extraction of an audio and text sample, and obtaining an FBank acoustic feature and a word vector feature; taking the obtained features as original input features of an audio emotion feature encoder and a text emotion feature encoder respectively, and extracting emotion semantic features of different modes through the encoders; performing audio attention, modal jump attention and text attention learning on the obtained emotion semantic features respectively, and extracting four complementary emotion features including an audio feature with remarkable emotion, an audio feature with semantic alignment, a text feature with semantic alignment and a text feature with remarkable emotion; and fusing the four features and then classifying to obtain corresponding emotion categories. According to the invention, the problemof low emotion recognition rate caused by intra-modal emotion irrelevant factors and inter-modal emotion semantics inconsistency in traditional multi-modal emotion recognition is solved, and the multi-modal emotion recognition accuracy can be effectively improved.

Description

technical field [0001] The invention belongs to the field of emotional computing, and in particular relates to a method for learning and identifying multi-modal emotional features based on an attention mechanism. Background technique [0002] In people's daily interactions, emotions often play a very important role, and the perception of emotional information helps people understand each other's mental states and behaviors. Likewise, emotional information is crucial to sustain long-term interactions between humans and machines, and automatic speech emotion recognition is an effective way to bridge the communication gap between humans and computers. With the rapid development and popularization of the Internet, people have put forward higher requirements for human-computer interaction systems, and people expect the machines they interact with to have the ability to observe, understand and generate emotional features similar to humans. Therefore, multimodal emotion recognitio...

AI Technical Summary

Problems solved by technology

[0004] Although the traditional multimodal emotion recognition methods can promote the final emotion recognition by fusing the emotional features of different modalities, most of the multimodal emot

Images