Questioning and answering method and device for open domain of fusing external knowledge

Abstract

The invention discloses a questioning and answering method and device for an open domain of fusing external knowledge. The method comprises the steps that according to a vision problem, discretized external knowledge represented by an explicit expression is extracted from a preset knowledge graph, and the vision problem comprises image information and a problem text; through a knowledge embedded mode maintained by a structure, the discretized external knowledge is embedded in semantic space represented by an implicit expression to obtain a high-dimensional continuous spatial vector; through adynamic memory network and an attention mechanism, auxiliary inference knowledge representation extraction is conducted on the high-dimensional continuous spatial vector, and an image feature is fusedto obtain the answer of the vision problem. According to the method, the superiority of a deep neural network model is kept, a large amount of structural external knowledge is introduced to assist inanswering the vision problem of the 'open domain', the dynamic memory network and the memory mechanism are utilized, the knowledge representation of effectively assisting inference is obtained, and therefore the reliability and effectivity of vision questioning and answering are effectively improved.

Description

technical field [0001] The invention relates to the technical fields of computer vision, visual question answering, and multimodal fusion, and in particular to an open-domain visual question answering method and device for fusing external knowledge. Background technique [0002] At present, answering "open domain" visual questions often requires a lot of external knowledge to assist reasoning, in addition to the information in the picture. Most of the existing visual question answering methods are only based on the image and the content of the question text itself, and the related work on introducing external knowledge in visual question answering is relatively limited. [0003] In related technologies, the visual question answering model based on deep neural network is difficult to answer "open domain" visual questions. A visual question answering model that performs explicit reasoning on a structured knowledge graph; this method expresses the content of the picture in a s...

AI Technical Summary

Problems solved by technology

[0003] In related technologies, the visual question answering model based on deep neural network is difficult to answer "open domain" visual questions

A visual question answering model that performs explicit reasoning on a structured knowledge graph; this method expresses the content of the picture in a structured form, which is easy to lose a lot of visual information, so it has great limitations; a visual question answering model that simply introduces unstructured external knowledge , which introduces a large amount of redundant information at the same time, and does not make full use of external knowledge

Images