A cross-modal retrieval method that can directly measure the similarity between different modal data

Abstract

The invention discloses a cross-modal retrieval method that can directly measure the similarity between different modal data. It includes the following steps: 1) feature extraction; 2) model building and learning; 3) cross-media data retrieval; 4) result evaluation. The present invention can directly perform similarity comparison between different modal data, and for cross-modal retrieval tasks, users can submit text, images, sounds, etc. of any modal to retrieve the corresponding modal results they need. The difference between the present invention and the traditional cross-media retrieval method is that the similarity comparison between different modal data can be directly performed, which meets the requirements of cross-media retrieval and realizes the user's retrieval intention more directly. Compared with cross-media retrieval algorithms based on modal similarity, this method has strong anti-noise ability and the ability to express loosely related cross-modal data, which makes the retrieval effect better.

Description

technical field [0001] The invention relates to cross-modal retrieval, in particular to a cross-modal retrieval method that can directly measure the similarity between different modal data. Background technique [0002] Nowadays, the types of electronic data tend to be more and more colorful, and various types of data, such as text, image, sound, map, etc., exist widely on the Internet. Often the same semantic content can be described by data of one modality, or by data of other modalities. Sometimes, for the description of one type of data with a certain semantic, we hope to find the corresponding description of other types of data. For example, search for pictures with similar meanings to the text based on the text, or search for news reports related to the pictures based on the pictures. However, existing retrieval methods are generally aimed at unimodal data, such as text retrieval for text and image retrieval for images. There are also some multimodal or multimedia r...

AI Technical Summary

Problems solved by technology

However, existing retrieval methods are generally aimed at unimodal data, such as text retrieval for text, image retrieval for image

There are also some multimodal or multimedia retrieval methods, but most of these multimodal retrieval methods measure the similarity between the same modalities, and then calculate the similarity between cross-media data through function mapping, and rarely directly compare the differences. Modal Similarity Retrieval Method

The shortcoming of the cross-media retrieval method that measures the similarity between the same modalities is that it cannot learn the relationship between cross-modal data, and needs to rely on the pre-specified matching relationship in the database, and it is loose for multimedia data. The corresponding relationship, the query effect is not ideal

The difficulty of directly comparing the similarity of different modal data is that the characteristics of different modal data are quite different, and generally speaking, the dimension is high, and there is a problem of "semantic gap"

Images