Cross-modal retrieval method based on graph convolutional neural network

Abstract

The invention discloses a cross-modal retrieval method based on a graph convolutional neural network. The cross-modal retrieval method comprises four processes of network construction, data set preprocessing, network training and retrieval and precision testing. Semantic representations in an image mode and a text mode are respectively learned by using a graph convolutional neural network; the cross-modal retrieval method can help to process the potential relationship among modal features, introduces the associated data of the third modal into the cross-modal retrieval method to reduce the semantic gap among the modals, and can significantly improve the accuracy and stability of cross-modal retrieval, thereby realizing accurate cross-modal retrieval.

Description

Technical field [0001] The present invention relates to the field of multimodal retrieval, in particular to an end-to-end cross-modal retrieval method. Background technique [0002] Cross-modal retrieval is a method that uses one modal data to query and returns retrieval results in other different modalities. It is widely used in matching image and text data. For example, in a traditional image-to-text cross-modal retrieval task, the most similar text is output as the output. In recent years, with the rapid development of deep learning, the current cross-modal retrieval methods mostly use neural networks to directly retrieve multi-modal data without relying on tags, but their methods simply combine cross-modal retrieval algorithms with deep neural networks. The combination of networks, such as selecting several features in the image to perform dimensionality reduction operations, etc., most of them do not make full use of the potential depth information in the multi-modal data, ...

AI Technical Summary

Problems solved by technology

[0003] In order to overcome the shortcomings of the existing cross-modal retrieval methods, such as insufficient utilization of multi-modal data, poor ability to represent data between different modalities, and low retrieval accuracy, the present invention provides a high-precision, fully-utilized multi-modal The cross-modal retrieval method based on graph convolutional neural network with high data and strong representation ability adopts the latest advanced neural network technology based on graph learning, which can not only extract deep semantic features more effectively, but also mine features in the model. potential correlations in

Images