Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A Cross-Modal Sentiment Classification Method Based on Compact Bilinear Fusion of Image and Text

An emotion classification and bilinear technology, applied in text database clustering/classification, character and pattern recognition, unstructured text data retrieval, etc., can solve problems such as cross-modal emotion classification of images and texts, and achieve good images Effects of Emotional Traits

Active Publication Date: 2018-05-25
HUAQIAO UNIVERSITY
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, researchers only pay attention to the research of models, and cannot perform cross-modal sentiment classification of graphics and texts well.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Cross-Modal Sentiment Classification Method Based on Compact Bilinear Fusion of Image and Text
  • A Cross-Modal Sentiment Classification Method Based on Compact Bilinear Fusion of Image and Text
  • A Cross-Modal Sentiment Classification Method Based on Compact Bilinear Fusion of Image and Text

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] Such as figure 1 Shown is the overall structure diagram of the present invention. exist figure 1Among them, the pre-trained 152-layer residual network is used to extract the feature representation of the image, and the two-layer LSTM model is used to extract the feature representation of the text, and then the two features are stitched together, and the soft attention map is generated through two layers of convolutional layers and softmax. The product of the corresponding position of the soft attention map and the image feature representation is summed to obtain the attention feature representation of the image, then the MCB algorithm is used to fuse the image attention feature representation and the text feature representation, and finally the softmax classifier is used to complete the cross-modality of the image and text Sentiment classification.

[0034] Such as figure 2 Shown is the MCB fusion algorithm model diagram of the present invention. This figure illu...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention provides a graphic-text cross-modal emotion classification method based on compact bilinear fusion, comprising the following six steps: (1) extraction of image feature representation; (2) extraction of text feature representation; (3) soft attention (4) Generation of image attention feature representation; (5) Multimodal compact bilinear fusion algorithm to fuse image attention feature representation and text feature representation; (6) Image-text emotion classification. The use of the soft attention map and the multimodal compact bilinear fusion algorithm in the method of the present invention can effectively improve the accuracy of emotion classification.

Description

technical field [0001] The invention relates to a cross-modal emotion classification method combining graphics and text, in particular to a cross-modal emotion classification method based on compact bilinear fusion of graphics and text. Background technique [0002] The object of cross-modal emotion classification research is to determine whether the sender is positive or negative based on the images sent by users and the corresponding text information on platforms such as Twitter, Facebook, Weibo, and e-commerce. manner. The application fields of graphic-text cross-modal sentiment classification mainly include: public opinion monitoring, user behavior analysis, product attribute evaluation, etc. With the advancement and development of science and technology, and the continuous improvement of software and hardware levels, many platforms can support users to send multimedia information such as voice, image, and video at the same time, and most of them choose to send both ima...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30G06K9/62
CPCG06F16/35G06F18/2411G06F18/214
Inventor 陈锻生吴琼吴扬扬雷庆张洪博
Owner HUAQIAO UNIVERSITY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products