Emotion recognition method and system based on multi-modal adversarial fusion in heterogeneous space

Abstract

The invention discloses an emotion recognition method and system based on multi-modal adversarial fusion in a heterogeneous space. The method comprises the steps of S100, acquiring multi-modal data composed of text information and image information; S200, extracting word feature vectors of the text information to obtain a text feature set; S300, extracting a regional feature vector of the image information to obtain an image feature set; S400, mapping the multi-modal data in the heterogeneous space; S500, carrying out adaptive fusion based on adversarial learning; and S600, utilizing a softmax layer to carry out emotion category prediction on a fusion result. Complementation of multivariate heterogeneous data in emotion recognition is realized, and richer and more accurate emotion recognition results can be obtained.

Description

technical field [0001] The invention relates to the technical field of emotion recognition, in particular to an emotion recognition method and system based on multi-modal confrontation fusion in a heterogeneous space. Background technique [0002] Big data is multi-source and heterogeneous. With the rapid development of information technology today, multi-modal data has become the main form of data resources recently. Natural phenomena are rich in features, so information from a single modality is often difficult to provide complete knowledge of the phenomenon of interest. Therefore, how to fuse the information of each modality has become a new challenge in many fields. Multimodal fusion can provide complementary information and improve the accuracy of overall decision-making. Existing multi-modal fusion methods map the data of different modalities from their independent representation spaces to a third-party public space, which inevitably loses the semantic information of...

AI Technical Summary

Problems solved by technology

Existing multimodal fusion methods map data of different modalities from their independent representation spaces to a third-party public space, which will inevitably lose the semantic information of the modalities, thereby affecting the fusion effect

Moreover, the current model does not fully consider the interaction between modalities in the fusion stage, and cannot produce better clustering effects in the representation space.

[0003] Especially in the field of emotion recognition, most of the emotions are recognized based on text information at present, but when the text information contains ironic meanings, it may lead to inaccurate emotion recognition; in multimedia networks, emoticon images are widely used, and it can be considered to combine emoticon images to assist text information recognize emotion

Images