A multi-modal sentiment analysis method and system fusing image caption and BERT

Abstract

The application discloses a kind of multi-modal sentiment analysis method and system of fusion image caption and BERT, it is related to multi-modal sentiment analysis technical field including extracting the image generation description image of multi-modal data set;Image feature is obtained by ResNet, the hidden layer representation of target is calculated using BERT encoder, and the final visual representation is obtained based on target image matching layer;Image description and text of multi-modal data are input BERT encoder to calculate the feature representation of image description and the feature representation of text, and the final text feature representation is obtained;Calculate multi-modal hidden layer representation, and the final sentiment classification is obtained by pooling layer fully connected layer and Softmax.This application extracts multi-modal data set image, realizes cross-modal sentiment analysis by combining more than two modalities, increases the understanding and recognition ability to image content, effectively solves the limitation of single mode, converts the information of image into visual representation with more expressive ability and semantic information, and improves the stability of multi-modal system.

Images