A visual question answering method based on the fusion of fine-grained image features and external knowledge

Abstract

The invention discloses a visual question answering method based on the fusion of fine-grained image features and external knowledge. The method consists of four steps: fine-grained image feature extraction, text processing and feature extraction, question knowledge retrieval based on external knowledge base and Multimodal feature fusion and answer prediction. Fine-grained image feature extraction is used to extract the sub-regional visual features of the image; text processing and feature extraction process visual question sentences and obtain the overall features of the questions; question knowledge retrieval based on external knowledge bases is introduced as a model of Freebase knowledge graph The external knowledge base for visual question answer prediction supplements necessary common sense or specific knowledge; multimodal feature fusion and answer prediction use similarity-based feature fusion method for multimodal feature fusion, and use the fused visual question features Make predictions about the answer to the question. The proposed method has better performance, with higher prediction accuracy for answers to visual questions.

Description

technical field [0001] The invention belongs to the field of intelligent information processing, and in particular relates to a visual question answering method. Background technique [0002] Visual Question Answering (VQA) is an interdisciplinary subject combining computer vision and natural language processing research. Its research goal is to enable computers to predict the answers to visual questions. The specific process is to input an image and an open question related to the image to the computer. The visual question answering system first needs to understand the semantics of the visual question text, and then combine the visual information of the image related to the question to predict the answer. The visual question answering task requires the computer to be able to deeply understand the content of the image in the visual question and the semantics of the question. The answer to some questions also requires the computer to master relevant common sense or specific k...

AI Technical Summary

Problems solved by technology

The visual question answering task requires the computer to be able to deeply understand the content of the image in the visual question and the semantics of the question. The answer to some questions also requires the computer to master relevant common sense or specific knowledge. Therefore, the research on visual question answering involves many artificial intelligence technologies, including detailed Granularity recognition, object recognition, behavior recognition and natural language processing, etc., which make visual question answering have higher requirements and greater challenges in image semantic understanding than traditional computer vision research.

[0003] There have been some studies on visual question answering in the prior art, but they use global image features, and cannot obtain fine-grained visual features that are highly correlated with the question text, and have poor applicability to fine-grained visual questions; most methods only focus on The content of the visual problem itself is greatly limited in its application scenarios; at the same time, the answer to the visual problem of fine-grained images is not good, and it is impossible to make certain reasoning on the basis of the visual problem

Images