A Deeply Supervised Cross-modal Adversarial Learning Method Based on Attention Mechanism

Abstract

The invention relates to a deep-supervised cross-modal confrontation learning method based on an attention mechanism. A deep learning network is constructed for each mode to obtain deep features, a generative confrontation network is introduced, and the cross-discrimination between modes is used to continuously refine with the help of an attention mechanism. The generated features of the modal feature network use the label information to perform deep supervised learning on the modal data in the label space while measuring the heterogeneous data in the common subspace. The network thus constructed enables the trained cross-modal depth-supervised adversarial model based on the attention mechanism to have good image and text mutual retrieval performance; in the retrieval process, the trained network model M is used to treat the query image (text) and candidate database. Feature extraction and cosine distance calculation are performed on the text (image) in the database, so as to obtain the image (text) to be queried and the text (image) data in the candidate database with higher similarity, and realize cross-modal retrieval.

Description

technical field [0001] The invention relates to the technical field of multimodal learning and information retrieval, and more particularly, to a deep-supervised cross-modal confrontation learning method based on an attention mechanism. Background technique [0002] Different modal data such as images and texts widely exist in the Internet. However, the problem of "heterogeneous gap" leads to inconsistent data distribution and representation of different modalities, and it is difficult to achieve semantic correlation, which is a problem for users in the massive Internet data between different modalities. It is inconvenient to retrieve useful information. In the prior art, cross-modal retrieval can be used to retrieve data in different modalities (image, text, voice, video, etc.), such as retrieving text through images, retrieving audio through text, retrieving video through audio, and so on. Cross-modal retrieval is commonly used in search engines and big data management. ...

AI Technical Summary

Problems solved by technology

[0003] However, multi-modal data has the characteristics of low-level feature heterogeneity and high-level semantic correlation. Therefore, the shortcomings of the existing cross-modal retrieval technology are: in the case of inconsistent data representation, the internal correlation between different modalities is poor. The degree of semantic association is low; the cross-modal similarity measurement of the existing technology faces great challenges

Images