Feature-level fusion method for multi-modal emotion detection

Abstract

The invention discloses a feature-level fusion method for multi-modal emotion detection. The method comprises the following steps: obtaining a transcript in a text form from a public data set; applying a single-layer CNN on the vector sequence X; extracting audio information from an audio file in the data set by using an open source tool openSMILE, and mapping a high-dimensional vector into a dense neural layer to obtain an audio feature vector; using the multi-dimensional self-attention as a feature fusion method for calculating the attention score probability of the single-peak feature; performing a weighted addition using the attention score probability to create a fusion vector; mapping the generated fusion vector su to another dense neural layer; calculating the classification probability of the fusion vector by using a softmax function; and calculating the batch loss Loss of the training based on back propagation by using the classification cross entropy to obtain an optimal emotion prediction node. According to the method, a self-attention mechanism is introduced, proper weights are allocated to the two modal features, and fusion features are obtained, so that the emotion recognition accuracy is improved.

Description

technical field [0001] The present invention proposes a new feature-level fusion method that is different from traditional fusion methods. This method extracts the features of text and audio modalities separately, introduces a self-attention mechanism, assigns appropriate weights to the two modal features, and obtains fusion features, thereby improving the accuracy of emotion recognition. The specific method involved is: a feature-level fusion method based on a self-attention mechanism. Background technique [0002] Emotion detection is a hot research field with broad application prospects. Machines can enhance human-computer interaction by accurately recognizing human emotions and responding to these emotions. Emotion recognition is also important in the fields of medicine, education, marketing, security, and surveillance. Applications. [0003] Traditional emotion detection methods only use a single visual or audio signal, which has certain limitations. Compared with si...

AI Technical Summary

Problems solved by technology

[0003] Traditional emotion detection methods only use a single visual or audio signal, which has certain limitations

Images