Cross-modal retrieval method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a cross-modal retrieval method. The method comprises the steps of obtaining first modal data and to-be-retrieved second modal data; inputting the first modaldata and the second modal data into a trained cross-modal retrieval network; extracting a first feature of the first modal data and a second feature of the second modal data through a cross-modal retrieval network, and outputting a retrieval result according to the matching degree of the first feature and the second feature; wherein the retrieval result is used for representing data matched with the second modal data and retrieved from the first modal data; wherein the cross-modal retrieval network comprises a network trained by using a neural network based on original data and generated datain a data set; wherein the generated data comprises data generated by a trained generator in the generative adversarial network based on the original data; and the generator and a discriminator in thegenerative adversarial network are trained at the same time, and the discriminator is used for generating data belonging to the first mode from data belonging to the second mode in the original data.

Description

technical field [0001] The present application relates to the field of computers, and in particular to a cross-modal retrieval method, a cross-modal retrieval device, a cross-modal retrieval device and a computer-readable storage medium. Background technique [0002] With the popularity of smart mobile terminals, each user can publish and receive various modal data including text, picture, video, audio, etc. anytime and anywhere, which brings rich content. However, rich content brings more pain to users, often making choices, and it becomes more and more difficult to obtain the information they really need. Rich information needs to be presented in front of users through precise screening. For the retrieval system, it is necessary to provide precise retrieval and services centered on users. Therefore, cross-modal retrieval systems have become a hot spot of concern today. [0003] For example, image-to-text cross-modal retrieval is an important and challenging classical com...

Images