An Unsupervised Cross-Modal Retrieval Method Based on Attention Mechanism Enhancement

Abstract

The invention belongs to the technical field of artificial intelligence smart community applications, and relates to an unsupervised cross-modal retrieval method based on attention mechanism enhancement, which first enhances the visual semantic features of images, then aggregates feature information of different modalities, and combines the fused Multi-modal features are mapped to the same semantic feature space, and then based on the generative confrontation network, image modality and text modality features are respectively adversarially learned with the same semantic feature after multi-modal fusion, and the semantic features of different modalities are aligned. Finally, the The different modal features after generating the confrontation network are aligned to generate hash codes respectively, and carry out the similarity measurement learning of intra-modal features and their hash codes and the similarity measurement of inter-modal features and their hash codes, which reduces the difference between different modalities. The heterogeneous semantic gap problem of the method enhances the dependency relationship between different modal features, thereby narrowing the semantic gap between different modal data, and can more robustly characterize the semantic common features between different modalities.

Description

technical field [0001] The invention belongs to the technical field of artificial intelligence smart community applications, and relates to an unsupervised cross-modal retrieval method based on attention mechanism enhancement, which can effectively handle cross-modal retrieval between large-scale images and texts in a smart community. Background technique [0002] Cross-modal retrieval is to use data from one modality to search for related data in another modality, for example, use a text description to retrieve images related to the text description in the image database. This technology is often used in daily life, such as Baidu search map, Taobao shopping and so on. Traditional cross-modal retrieval is divided into supervised cross-modal retrieval and unsupervised cross-modal retrieval. Due to the remarkable effect of deep neural network in the field of computer vision, deep cross-modal retrieval has become the mainstream of current research. With the rapid development ...

AI Technical Summary

Problems solved by technology

However, a non-negligible problem with these methods is that the semantic information obtained from images and texts is not equal, which increases the indirectness of the heterogeneous semantic gap between different modalities, which leads to the failure of retrieving data of different modalities.

Images