Zero-sample cross-modal retrieval method based on multi-modal feature synthesis

Abstract

The invention discloses a zero-sample cross-modal retrieval method based on multi-modal feature synthesis. The method comprises the steps: employing two adversarial generative networks, synthesizing feature representations of different modals through class embedding shared by two modal data, then mapping original modal data and synthesized modal data to a common subspace, and carrying out the aligned distribution of the original modal data and the synthesized modal data. Therefore, the relation between different modal data of the same category is established, and knowledge is migrated to unseen categories. The cyclic consistency constraint further reduces the difference between the original semantic feature and the reconstructed semantic feature, and well establishes the association between the original representation and the semantic feature in each modal, so that the common semantic space is more robust, and the accuracy of zero-sample cross-modal retrieval is improved.

Description

technical field [0001] The invention belongs to the technical field of cross-modal retrieval, and more specifically relates to a zero-sample cross-modal retrieval method based on multimodal feature synthesis. Background technique [0002] The goal of cross-modal retrieval is to search for semantically similar instances in another modality (such as image) by using a query from one modality (such as text). The distributions and feature representations of different modality data are inconsistent, so it is difficult to directly measure the similarity between different modality data. The solution of existing methods is usually to establish a common subspace, map the data of different modalities to this common subspace to obtain a unified representation, and then use some measurement methods to calculate the similarity between different modal data, and The search results are those with the largest similarity of the search targets, thus achieving cross-modal search. [0003] Howe...

AI Technical Summary

Problems solved by technology

Existing zero-shot learning methods of this kind are usually used to solve traditional classification problems, are not optimized for cross-modal retrieval problems, and often focus only on the mapping from raw data representations to category embeddings, ignoring their interrelationships

Images