A Multimodal Sentiment Analysis Method Based on Multidimensional Attention Fusion Network

Abstract

The invention discloses a multi-modal emotion analysis method based on a multi-dimensional attention fusion network, which includes: extracting voice preprocessing features, video preprocessing features, text Preprocessing features; then construct the multi-dimensional attention fusion network for each modality, use the autocorrelation feature extraction module inside the network to extract the first-level autocorrelation feature and the second-level autocorrelation feature, and then combine the three modalities Combining autocorrelation information, using the cross-modal fusion module inside the network to obtain the cross-modal fusion features of the three modalities; then using the second-level autocorrelation features and cross-modal fusion features to combine to obtain modal multi-dimensional features ;Finally, splicing the multi-dimensional features of the modalities, determining the sentiment score, and performing sentiment analysis; sentiment analysis.

Description

technical field [0001] The invention belongs to the field of multi-modal emotion computing, and more particularly, relates to a multi-modal emotion analysis method based on a multi-dimensional attention fusion network. Background technique [0002] Sentiment analysis has many applications in daily life. With the development of big data and multimedia technology, the use of multimodal sentiment analysis technology to analyze the different modalities of voice, video and text of data is more conducive to mining the shallow meaning behind the data. For example, in the return survey, through the comprehensive analysis of the user's voice, face and speech content, the user's satisfaction with the service or product can be known. [0003] At present, the difficulty of multimodal sentiment analysis lies in how to effectively integrate multimodal information, because the acquisition methods of voice, video and text features are completely different. When describing the same content...

AI Technical Summary

Problems solved by technology

This method is only a simple result integration of the three modalities, and does not take into account the correlation information between the modalities, and it is easy to cause model overfitting due to information redundancy.

Another approach is based on modal labeling alignment, that is, when performing data labeling, the three modalities are forced to align in the time dimension based on text or phonemes, thus ensuring the corresponding relationship of the three modalities in time. Then use the recurrent neural network, convolutional neural network, attention mechanism, and Seq2Seq framework for modal fusion. This method is expensive to label and is not conducive to the actual production and living environment.

Images