Image text matching method based on multi-relation perceptual reasoning

Abstract

Aiming at an image text matching task, the invention designs an image text matching method based on multi-relation perceptual reasoning, and relates to the two fields of computer vision and natural language processing. Sufficient mining of features of visual and text modals and alignment of features of different modals are the key to overcome the difficulty of an image text matching task. Reasonable utilization of the graph convolutional neural network is beneficial for improving the performance of the model, the method designs a multi-relation perception reasoning module for the image based on the graph convolutional neural network, pays attention to the semantic relation and spatial position relation of the image, extracts richer visual feature representation, and realizes better alignment with text semantic information; and a BERT-GRU-based text encoder is combined, so that deep semantic information of the sentences can be comprehensively expressed, and good alignment with image visual representation is realized. The method has certain significance in the practical application aspects of automatic image-text mutual retrieval, children intelligence development education, visual impaired person assistance and the like.

Description

technical field [0001] The present invention relates to two fields of computer vision and natural language processing, and specifically involves using a multi-relationship perception reasoning module to focus on the spatial position relationship and semantic relationship between image regions, and using a BERT-based text encoder to focus on text representations containing contextual semantic information . Background technique [0002] The image-text matching task (Image-text matching) aims to measure the similarity between an image and a piece of text in a cross-modal embedding space. This task involves the learning of two modalities, vision and text, and is a bridge for joint computer vision and natural language processing. [0003] Early image-text matching models mainly used a standard dual-branch embedding architecture to extract image and text features and map them into the embedding space for matching. This method has been proven to be useful, but only focuses on the...

AI Technical Summary

Problems solved by technology

Due to the richness of sentence semantics and the diversity of structures, these methods use a fixed vector for each word, which cannot solve the problem of polysemy

Images