Picture text cross-modal retrieval method based on self-supervised adversarial

Abstract

The invention discloses a picture text cross-modal retrieval method based on self-supervised adversarial, which comprises the following steps of: constructing a shared network in an embedded generation network to serve as picture features and text features to be respectively mapped into shared subspaces of picture codes and text codes, so that the separation between the picture features and the text features is reduced, and meanwhile, constructing a self-supervised adversarial network, performing adversarial training under the assistance of the self-supervised adversarial network to embed a generative network, filtering redundant information in picture features, so that different modal features are better mapped to the same space, the different modal features can be better aligned, and therefore the cross-modal retrieval effect of picture texts is improved.

Description

technical field [0001] The invention belongs to the technical field of image text cross-modal retrieval, and more specifically, relates to a self-supervised confrontation-based image text cross-modal retrieval method. Background technique [0002] With the development of cloud computing, Internet of Things, mobile phones, social media and other information technologies, the data on the Internet is growing explosively, and the era of big data has arrived. In the era of big data, how to perform fast image text cross-modal retrieval is the focus of people's attention. [0003] Self-supervised learning uses the labels of its own attributes to train the model to obtain the features of the picture, avoiding expensive manual labeling, but the learned features are often only related to the visual information of the picture. Adversarial training alternately trains the generator and the discriminator, so that the samples generated by the generator cannot be successfully discriminated...

AI Technical Summary

Problems solved by technology

Adversarial training alternately trains the generator and the discriminator so that the samples generated by the generator cannot be successfully discriminated by the discriminator

[0004] Due to the large gap between image features and text features, it is difficult to map the features of the two modalities into the same space for retrieval in traditional image-text cross-modal retrieval methods, which hinders efficient cross-modal retrieval

In particular, there is redundant information in image features, and different modal features cannot be well mapped to the same space, which affects the effect of icon text cross-modal retrieval

Images