Image and text retrieval method based on graph neural network structure modeling

Abstract

The invention discloses an image and text retrieval method based on graph neural network structure modeling, and the method comprises the steps: employing an attention mechanism to represent the fine-grained vision extracted from a picture and a text, and enabling text elements to calculate the similarity of the picture and the text better; a graph structure is constructed in a self-adaptive modethrough visual and text elements, features are updated through a graph convolution method, and the intra-modal and inter-modal relation of the visual and text elements can be better considered; a constraint mechanism is introduced between different pictures and text pairs and in the visual and text element alignment process, fine-grained text elements can correspond to corresponding picture areaseasily, then the reliability of picture and text level similarity calculation is improved, and the accuracy of picture and text retrieval is improved.

Description

technical field [0001] The invention relates to the technical field of multimedia retrieval, in particular to an image and text retrieval method based on graph neural network structure modeling. Background technique [0002] With the influx of massive multimedia data into the Internet, multimedia retrieval technologies spanning multiple different modal data (visual, text, voice, etc.) play an increasingly important role. [0003] Traditional image retrieval techniques often use tags to retrieve images. This process is often unidirectional and can only utilize discrete labeled data. The two-way retrieval of images and texts contains richer semantics and is more in line with the human habit of using natural language. However, there is a big difference between the data of the two different modalities of vision and text. In order to achieve cross-modal retrieval of images and texts, it is necessary to integrate computer vision and natural language understanding well. [0004...

AI Technical Summary

Problems solved by technology

However, these methods seldom consider fine-grained alignment between visual elements, text elements

This limits the overall similarity calculation of images and texts, affecting the final retrieval accuracy

Images