Multi-modal data feature extraction and association method for public opinion analysis

Abstract

The invention discloses a multi-modal data feature extraction and association method for public opinion analysis. The method comprises the following steps: constructing a first neural network and a second neural network; extracting text features of the text public opinion information by the first neural network, and extracting image features of the image public opinion information by the second neural network; constructing graph structures A and B comprising nodes and edges, wherein the graph structure A is used for encoding text features, and nodes on the graph structure A correspond to the text features, and the graph structure B is used for encoding image features, and nodes on the graph structure B correspond to the image features; matching the image public opinion information with thetext public opinion information one by one to form a plurality of image-text pairs, and jointly constraining and training through a semantic loss function and a triple loss function; and evaluating and sequencing the matching association degree of the plurality of image-text pairs containing the same image public opinion information or the same text public opinion information. According to the method, a graph structure is adopted to model the relationship between objects and words in an image text pair, and the matching performance is improved through a reordering mechanism.

Description

technical field [0001] The invention relates to the fields of cross-modal retrieval and semantic analysis, in particular to a multi-modal data feature extraction and association method for public opinion analysis. Background technique [0002] At present, with the advent of the era of big data, the data volume of Internet public opinion is growing explosively, and the data format is diversified. Internet public opinion is affected by massive, heterogeneous, and multi-dimensional information sources, and the main data formats are text and images. In recent years, the task of text-image matching has become a hot research topic and attracts increasing interest, which connects the fields of computer vision and natural language processing, and involves two different modalities. How to extract public opinion text features and image features, and establish a mapping between heterogeneous feature spaces, has become the key to effectively analyze public opinion data and use multi-sou...

AI Technical Summary

Problems solved by technology

[0005] Although a lot of work has been done in the field of image-text matching, and researchers have focused more on mapping to the public space, there is still no some lack

Images