Cross-modal retrieval method based on deep self-supervised sorting hash

Abstract

The invention provides a cross-modal retrieval method based on deep self-supervised sorting hash. The method comprises the following steps of: firstly, learning a label network for reserving a similarity relationship between semantic features and corresponding hash codes; according to the label network, effectively using multi-label information to bridge semantic relevance among different modes; and designing an end-to-end feature learning network for the image and the text, and carrying out feature learning; in one aspect, a semantic correlationship between the label network and an image text network is kept. And on the other hand, the learned features can be perfectly compatible with a specific cross-modal retrieval task. In order to solve the problem that binary partition function coding is very sensitive to partition threshold values, a coding function based on sorting is adopted. The relative order of each dimension is not changed, and the value of the Hash code is not changed, so that the Hash function is not very sensitive to some threshold values, and the robustness of the obtained Hash code is better.

Description

technical field [0001] The invention relates to pattern recognition, in particular to a cross-modal retrieval method based on deep self-supervised sorting and hashing. Background technique [0002] Cross-modal retrieval has become a compelling topic in recent years due to the explosion of multimedia data on various search engines and social media. Cross-modal retrieval aims to use data from one modality (e.g. text) to search for semantically similar instances in another modality (e.g. image). Since data from different modalities usually have incomparable feature representations and distributions, it is necessary to map them into a common feature space. To meet the requirements of low storage cost and high query speed in practical applications, hashing has attracted much attention in the field of cross-modal retrieval. It maps high-dimensional multi-modal data to the public Hamming space, and after obtaining the hash code, the similarity between multi-modal data can be calc...

AI Technical Summary

Problems solved by technology

[0003] At present, cross-modal retrieval has been greatly developed. Many shallow cross-modal hash retrieval methods have been proposed. These shallow methods are based on handcrafted features for hash learning. A common shortcoming of them is that handcrafted features The crafting process and the hashing learning process are completely independent, and thus handcrafted features may not be fully compatible with the hashing learning process

Another reason for the unsatisfactory retrieval performance is that most existing deep hashing cross-modal retrieval methods discard the full labels of the data and only use the cross-modal similarity matrix for supervised learning, so that the learned hash codes lack Semantic information, not accurate enough

In addition, most cross-modal retrieval methods are encoded using binary space partitioning functions, which can produce large encoding errors and also lead to poor retrieval performance.

Images