Cross-modal retrieval method and device for multi-modal data, equipment and medium

Abstract

The invention discloses a cross-modal retrieval method and device for multi-modal data, cross-modal retrieval equipment for multi-modal data, and a computer readable storage medium. The method comprises the steps: inputting training sample data of different modals into deep neural networks corresponding to all modals in batches, and obtaining the sample data features of all training sample data; respectively mapping each sample data feature into a common space, and calculating a corresponding loss function according to the intra-class low-rank loss constraint and semantic consistency constraint of each training sample data of different modes of the same class; adjusting network parameters of the deep neural network by using a loss function, and determining a target feature extraction model; and, after target data and to-be-retrieved data of different modes are obtained, calling the target feature extraction model to perform cross-modal retrieval operation, and then obtaining a retrieval sorting result of the to-be-retrieved data corresponding to the target data, so that the target feature extraction model can extract data features with higher quality, thereby improving the accuracyof cross-modal retrieval of the multi-modal data.

Description

technical field [0001] The present invention relates to the field of data retrieval, in particular to a cross-modal retrieval method, device, equipment and computer-readable storage medium for multi-modal data. Background technique [0002] With the continuous advancement of the information society, cross-modal retrieval of multi-modal data is more and more widely used in real life. For example, the image information corresponding to the description of the voice information is retrieved according to a piece of voice information; or the voice information corresponding to the description of the text information is retrieved according to the text information. [0003] In the process of cross-modal retrieval, it is necessary to obtain the common data characteristics of cross-modal data so that they can be compared directly. The quality of data features extracted from multimodal data using deep neural networks will directly affect the accuracy of cross-modal retrieval of multimo...

AI Technical Summary

Problems solved by technology

However, there is not absolutely a corresponding relationship in the sample data. Therefore, according to the method of the prior art, the method of adjusting the network parameters of the deep

Images