Cross-modal retrieval method and device for multi-modal data, equipment and medium

Abstract

The invention discloses a cross-modal retrieval method and device for multi-modal data, cross-modal retrieval equipment for multi-modal data, and a computer readable storage medium. The method comprises the steps: inputting training sample data of different modals into deep neural networks corresponding to all modals in batches, and obtaining the sample data features of all training sample data; respectively mapping each sample data feature into a common space, and calculating a corresponding loss function according to the intra-class low-rank loss constraint and semantic consistency constraint of each training sample data of different modes of the same class; adjusting network parameters of the deep neural network by using a loss function, and determining a target feature extraction model; and, after target data and to-be-retrieved data of different modes are obtained, calling the target feature extraction model to perform cross-modal retrieval operation, and then obtaining a retrieval sorting result of the to-be-retrieved data corresponding to the target data, so that the target feature extraction model can extract data features with higher quality, thereby improving the accuracyof cross-modal retrieval of the multi-modal data.

Description

technical field [0001] The present invention relates to the field of data retrieval, in particular to a cross-modal retrieval method, device, equipment and computer-readable storage medium for multi-modal data. Background technique [0002] With the continuous advancement of the information society, cross-modal retrieval of multi-modal data is more and more widely used in real life. For example, the image information corresponding to the description of the voice information is retrieved according to a piece of voice information; or the voice information corresponding to the description of the text information is retrieved according to the text information. [0003] In the process of cross-modal retrieval, it is necessary to obtain the common data characteristics of cross-modal data so that they can be compared directly. The quality of data features extracted from multimodal data using deep neural networks will directly affect the accuracy of cross-modal retrieval of multimo...

AI Technical Summary

Problems solved by technology

However, there is not absolutely a corresponding relationship in the sample data. Therefore, according to the method of the prior art, the method of adjusting the network parameters of the deep neural network according to the ranking results by using the correlation ranking still has inaccurate adjustments, so that Leads to reduced accuracy for cross-modal retrieval of multimodal data

Images