Cross-modal generalized zero sample retrieval method based on dual learning generative adversarial network

Abstract

The invention provides a cross-modal generalized zero sample retrieval method based on a dual learning generative adversarial network. The method comprises: constructing a generative adversarial network based on dual learning; mapping the high-dimensional visual features of different modes to a common low-dimensional semantic embedding space; secondly, constructing multiple constraint mechanisms to perform cyclic consistency constraint, generative adversarial constraint and classifier constraint so as to maintain visual-semantic consistency and generated feature-source feature consistency, andperforming cross-modal retrieval after training of the whole network, so that the model is more powerful in performance in generalization of zero-sample retrieval. Meanwhile, in the whole training process, paired multimedia data pairs on the pixel level do not need to serve as training samples, only paired data on the category are needed, so that the complexity and expensive cost of data set collection are reduced, the retrieval effect is better, and performance improvement is more obvious in the zero-sample generalization retrieval problem.

Description

technical field [0001] The invention belongs to the technical field of cross-modal retrieval in computer vision, and in particular relates to a retrieval method for multimedia data, that is, a cross-modal generalization zero-sample retrieval method based on dual learning generative adversarial networks. Background technique [0002] Cross-media retrieval means that users can retrieve semantically related data in all media types by inputting query data of any media type. With the increasing amount of multimedia data such as text, images, and videos on the Internet, retrieval across different modalities has become a new trend in information retrieval. The goal of traditional cross-modal zero-shot retrieval is to perform cross-modal retrieval on unseen new category data; the test set of cross-modal generalization zero-shot retrieval includes unknown classes and some known classes, but its class labels Unknown to the model, it is more difficult to retrieve. [0003] Currently,...

AI Technical Summary

Problems solved by technology

[0004] 1. There are "semantic gaps" and intra-class differences between different modal data, which lead to inconsistencies between data distribution and feature representation of different media types, so it is difficult to directly measure the similarity between multiple media data sex;

[0005] 2. The zero-sample retrieval problem involves transferring the knowledge learned from known classes to unknown classes. There are information loss and over-fitting problems in the learning and transfer process, so generalized zero-sample retrieval is often better than traditional zero-sample retrieval. Retrieval is more difficult

[0007] The main problem of existing methods is that there is a loss of semantic information when mapping from high-dimensional visual features to low-dimensional semantics, and the adversarial generative network is often unstable during training; The feature and semantic information are encoded definitely, so that the model will tend to the known classes that have appeared in the training process during the retrieval process.

Images