A multi-modal retrieval method based on deep learning and hash algorithm

By employing a multimodal retrieval method based on deep learning and hashing algorithms, utilizing multi-path Transformer pre-trained models and encoder technology, and combining speech recognition and ElasticSearch, high-efficiency and high-precision retrieval on large-scale multimodal datasets is achieved, solving the problems of low efficiency and insufficient accuracy in existing technologies.

CN116204694BActive Publication Date: 2025-12-16BEIJING UNIV OF POSTS & TELECOMM
0 Cites 0 Cited by

Patent Information

Application Number
CN202310126081.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-15
Publication Date
2025-12-16
Estimated Expiration
2043-02-15

AI Technical Summary

Technical Problem

Existing multimodal retrieval models suffer from low retrieval efficiency and low accuracy on large-scale datasets. In particular, models based on fusion encoders need to calculate the similarity between user input data and all image and text data in the database, resulting in high hardware resource consumption. On the other hand, models based on dual encoders cannot guarantee information interaction and sharing between different modal data, leading to insufficient retrieval accuracy.

Method used

A multimodal retrieval method based on deep learning and hashing algorithms is adopted. A dual encoder and a fusion encoder are constructed through a pre-trained model based on multi-path Transformer to realize the mapping of text and image data in a unified vector subspace. Combined with speech recognition technology, ElasticSearch technology and gray-level comparison method, text semantic, image and video retrieval are performed. The model is trained using cosine similarity and cross-entropy loss function to achieve "coarse recall" and "precise ranking".

Benefits of technology

High-efficiency and high-precision multimodal retrieval is achieved on large-scale datasets. Hash encoding reduces hardware resource consumption and improves retrieval efficiency. A multi-encoder model ensures retrieval accuracy, achieving high-efficiency and high-precision cross-modal retrieval of text and images.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The application provides a multi-modal retrieval method based on deep learning and a hash algorithm, comprising: obtaining multi-modal data to be retrieved; performing text semantic retrieval on the multi-modal data based on a deep learning and hash coding method to obtain text data; performing picture retrieval on the multi-modal data based on a gray scale comparison method and an ElasticSearch technology to obtain picture data; mapping the text data and the picture data into a unified vector subspace by using a model based on a double encoder; obtaining a candidate set by calculating the similarity distance of the text data and the picture data in the vector subspace through a cosine similarity; and calculating the similarity on the candidate set by using a model based on a fusion encoder and performing sorting to obtain a retrieval result of the multi-modal data. By the method, the efficient and high-precision multi-modal retrieval task on a large-scale data set is realized.
Need to check novelty before this filing date? Find Prior Art