An information matching method and system based on hierarchical multimodal information fusion

Abstract

The invention provides an information matching method and system based on hierarchical multi-modal information fusion, which extracts the feature representation of user multi-source heterogeneous multi-modal data; The graph structure captures the external and internal relationships of users in social media, and obtains accurate representations of different sources of users; according to the representations obtained from different sources of users, adversarial learning is introduced to perform semantic discrimination, further enhance user representation, and realize user identity information match. The present invention can improve the modeling performance of user identification across social media.

Description

technical field [0001] The invention belongs to the technical field of information processing, and relates to an information matching method and system based on hierarchical multimodal information fusion. Background technique [0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art. [0003] Multimodal information refers to different types of data such as text, images, and audio. Generally speaking, there are complementary semantic relationships between multimodal information, which can reflect the expressed content from different perspectives and reinforce each other. Taking social media as an example, users involve heterogeneous multimodal data in social media, such as text, pictures, spatiotemporal information, and social relationships. These heterogeneous multimodal data can refract users' daily life from different angles and reflect their attribute characteristics from di...

AI Technical Summary

Problems solved by technology

Although the internal multi-modal publishing content will display users from different aspects, there is a complementary semantic relationship between them for user representation

At the same time, for external social relations, due to different degrees of intimacy, different followers usually have different confidence levels for user characterization. How to capture the hierarchical relationship of user-related data in social media to effectively represent users is a difficult task. challenge;

[0005] At the same time, due to the different data distributions among different social media, even the representation of the same person in different social media usually has the problem of semantic gap, resulting in inaccurate user identification

Images