Visual question-answering method and system based on external knowledge aggregation

Abstract

The invention discloses a visual question-answering method and system based on external knowledge aggregation. The method comprises the steps that firstly, an external knowledge graph subgraph relatedto a scene is extracted, then knowledge aggregation is conducted on the knowledge graph subgraph to obtain knowledge entity representation, and finally the entity representation and a traditional knowledge question-answering system are organically fused to finally obtain a question answer. According to the method, the external knowledge graph is introduced into a traditional visual question-answering system, the method can be applied to traditional visual questions and visual questions needing external knowledge, extra strong supervision information is not needed in the model training process, the method has high applicability, and better visual question accuracy can be obtained on multiple benchmark data sets.

Description

technical field [0001] The invention relates to the technical field of computer vision question answering, in particular to a visual question answering method and system based on external knowledge aggregation. Background technique [0002] Such as figure 1 As shown, the visual question answering task refers to: given a picture and its corresponding natural language description question, how to use the information in the picture to get the correct answer to the question. In actual scenarios, in addition to image information, it is often necessary to introduce external common sense knowledge to assist in answering visual questions. [0003] Most of the existing visual question answering methods are only based on the image and the content of the question text itself. At present, there is limited work on introducing external knowledge graphs into visual question answering. According to the degree of integration of external knowledge graphs, there are three main types of relat...

AI Technical Summary

Problems solved by technology

[0007] For example, related technologies Narasimhan M, Schwing A G. Straight to the facts: Learning knowledge base retrieval for factual visual question answering[C] / / Proceedings of the European Conference on Computer Vision(ECCV).2018:451-468, in this method The external knowledge base used is constructed based on a specific environment, with a limited scale, and does not need to extract knowledge subgraphs; each piece of knowledge in the external knowledge base is considered separately, and the structural characteristics of the graph are not used for knowledge aggregation; it is a knowledge map. The model for information retrieval cannot be combined with the traditional visual question answering system, and additional supervision information is required during training.

[0008] In summary, the existing visual question answering methods that integrate external knowledge graphs have poor applicability and are difficult to apply in open scenarios with complex and diverse questions.

Images