A Multimodal Search Method Based on Neighbor Graph

Abstract

The invention relates to a multi-modal search method based on a neighbor graph. Firstly, each modal data of each reference object in a reference data set is generated into a eigenvector, and then independently calculated according to each eigenvector, and then calculated by an aggregation function fusion to obtain each eigenvector. Query the fusion distance between objects to construct a neighbor graph of reference objects. Then, a query vector containing multiple feature vectors is generated according to the query content, and the query vector is used to perform a multimodal search on the neighbor graph to obtain the most similar query target. The method of the present invention simultaneously queries multiple modalities of the object by querying the fusion distance, and can change the influence weight of different modalities on the fusion distance by adjusting the aggregation function, thereby realizing the importance of the modalities in the search process Flexible manipulation, and improve the efficiency and accuracy of the search.

Description

technical field [0001] The invention belongs to the technical field of search, and in particular relates to a multi-modal search method based on a neighbor graph. Background technique [0002] With the continuous development of the Internet, various applications continue to generate and eventually gather massive amounts of text, pictures, audio, and video data. The multimodality and massiveness of data pose a huge challenge to information retrieval. With the research and progress of intelligent technology, various modal data can be extracted by artificial intelligence and converted into feature vectors for various similarity calculations and various extended applications. Therefore, the multimodal search method Research is very important. [0003] The current multimodal search mainly adopts two methods: the first is to perform single-modal search on each modality in the multimodal data, and then merge the search results. Obviously, the efficiency of this method will increas...

AI Technical Summary

Problems solved by technology

[0003] The current multimodal search mainly adopts two methods: the first is to perform single-modal search on each modality in the multimodal data, and then merge the search results. It is obvious that the efficiency of this method will increase. It decreases significantly with the increase of the number of modalities and the amount of data; the second is to map the data of each modality to a unified multimodal vector space through the learning method and then search. The disadvantage of this method is After the mapping model is trained, the importance of a modality cannot be freely controlled in the search phase, which is not flexible enough and the recall rate is very low

Images