Cross-modal retrieval method based on modal specificity and shared feature learning

Abstract

The invention discloses a cross-modal retrieval method based on modal specificity and shared feature learning, which comprises the following steps: S1, acquiring a cross-modal retrieval data set, and dividing the cross-modal retrieval data set into a training set and a test set; S2, respectively carrying out feature extraction on the text and the image; S3, extracting modal specific features and modal sharing features; S4, generating a hash code corresponding to the modal sample through a hash network; S5, training the network by combining the loss function of the adversarial auto-encoder network and the loss function of the Hash network; and S6, performing cross-modal retrieval on the samples in the test set by using the network trained in the step S5. According to the method, a Hash network is designed, encoding features of image channels, encoding features of text channels and modal sharing features are projected into a Hamming space, and modeling is performed by using label information, modal specificity and sharing features, so that output Hash codes have better semantic discrimination between modals and in the modals.

Description

technical field [0001] The invention relates to a cross-modal retrieval method, in particular to a cross-modal retrieval method based on modality-specific and shared feature learning. Background technique [0002] In recent years, massive amounts of multimodal data have flooded our lives. Take the news on the Internet as an example, which usually includes text introductions, and sometimes some photos taken by reporters are also typeset on the page, and there are even some exclusive video and audio reports. Multimodal data such as text, image, video, audio, etc. is an important means for us to efficiently obtain the same information from multiple angles. Users not only need to retrieve between single-modal data, but also need a more flexible retrieval method: accurately retrieve related data from one modality to another modality. In recent years, the work of cross-modal retrieval has become a hot topic widely discussed in academia. However, multimodal data usually have rel...

AI Technical Summary

Problems solved by technology

In addition, the existing cross-modal retrieval work rarely considers modality-specific information and modality-shared information when doing feature extraction, resulting in the loss of effective information

Images