Unsupervised cross-modal retrieval method based on attention mechanism enhancement

Abstract

The invention belongs to the technical field of artificial intelligence smart community application, and relates to an unsupervised cross-modal retrieval method based on attention mechanism enhancement, which comprises the following steps of: enhancing visual semantic features of an image, then aggregating feature information of different modals, mapping fused multi-modal features to the same semantic feature space, then, on the basis of a generative adversarial network, adversarial learning is carried out on the image modal and text modal features and the same semantic feature after multi-modal fusion, aligning semantic features of different modals, and finally, generating hash codes by the different modal features after alignment of the generative adversarial network; and performing intra-modal feature and Hash code similarity measurement learning and inter-modal feature and Hash code similarity measurement, so a heterogeneous semantic gap problem between different modalities is reduced, a dependency relationship between different modal features is enhanced, a semantic gap between different modal data is reduced, and semantic common characteristics among different modes can be represented more robustly.

Description

technical field [0001] The invention belongs to the technical field of artificial intelligence smart community applications, and relates to an unsupervised cross-modal retrieval method based on attention mechanism enhancement, which can effectively handle cross-modal retrieval between large-scale images and texts in a smart community. Background technique [0002] Cross-modal retrieval is to use data from one modality to search for related data in another modality, for example, use a text description to retrieve images related to the text description in the image database. This technology is often used in daily life, such as Baidu search map, Taobao shopping and so on. Traditional cross-modal retrieval is divided into supervised cross-modal retrieval and unsupervised cross-modal retrieval. Due to the remarkable effect of deep neural network in the field of computer vision, deep cross-modal retrieval has become the mainstream of current research. With the rapid development ...

AI Technical Summary

Problems solved by technology

However, a non-negligible problem with these methods is that the semantic information obtained from images and texts is not equal, which increases the indirectness of the heterogeneous semantic gap between different modalities, which leads to the failure of retrieving data of different modalities.

Images