Cross-modal image audio retrieval method based on deep heterogeneous correlation learning

Abstract

The invention discloses a cross-modal image audio retrieval method based on deep heterogeneous correlation learning. The method mainly solves the problem that heterogeneous correlation information of images and audios is not sufficiently utilized in an existing method. According to the invention, firstly, a new cross-modal pairwise construction strategy is designed to select effective image and audio pairs, which is beneficial to capturing heterogeneous correlation between images and audios. According to the invention, the relationship between the image and the audio is established by utilizing the heterogeneous correlation of the depth features, the Hash code is generated by bridging the deep feature correlation between the image and the audio for image and audio retrieval, and the quantization error between the Hash-like code and the Hash code is reduced by using the regularization constraint. According to the invention, the heterogeneous correlation of the depth features is fully utilized, and the retrieval performance is further improved.

Description

technical field [0001] The invention belongs to the field of image retrieval, and in particular relates to a cross-modal image and audio retrieval method based on deep heterogeneous correlation learning. Background technique [0002] With the explosive growth of various images, texts, audio and video data on the Internet, cross-modal image and audio retrieval has been widely used in the fields of computer vision and natural language processing, such as search engines and driverless vehicles. Typical application scenarios. The task of cross-modal image-audio retrieval is to retrieve relevant images with audio, or retrieve relevant audio with images. However, due to the heterogeneity of multimodal data, it is difficult for users to obtain favorable information quickly and accurately. Therefore, how to improve retrieval efficiency and solve the heterogeneous problem of multimodal data are two great challenges for cross-modal retrieval tasks. [0003] At present, some deep lea...

AI Technical Summary

Problems solved by technology

Although the existing cross-modal remote sensing image-sound retrieval methods have made some progress, however, the existing cross-modal image-audio retrieval methods still have some limitations: (1) Existing methods do not fully learn heterogeneous correlation relations , leading to underutilization of heterogeneous correlations in cross-modal learning

(2) Existing image and audio retrieval methods use high-dimensional real-valued features for cross-modal retrieval, which require a large amount of storage space

(3) Some existing cross-modal retrieval methods cannot effectively select good cross-modal paired samples, which will eventually affect the effectiveness of cross-modal correlation learning

Images