Visual question-answering method and system based on external knowledge aggregation

An external knowledge, question answering system technology, applied in computer parts, character and pattern recognition, text database clustering/classification, etc., can solve problems such as poor applicability, limited scale, difficult application problems, complex and diverse open scenarios, etc. The effect of excellent visual question answering accuracy and high applicability

An external knowledge, question answering system technology, applied in computer parts, character and pattern recognition, text database clustering/classification, etc., can solve problems such as poor applicability, limited scale, difficult application problems, complex and diverse open scenarios, etc. The effect of excellent visual question answering accuracy and high applicability

CN111475656AActive Publication Date: 2020-07-31TSINGHUA UNIV
2 Cites 8 Cited by

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Visual question-answering method and system based on external knowledge aggregation
  • Visual question-answering method and system based on external knowledge aggregation
  • Visual question-answering method and system based on external knowledge aggregation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0036] The following describes the visual question answering method and system based on external knowledge aggregation according to the embodiments of the present invention with reference to the accompanying drawings. First, the visual question answering method based on external knowledge aggregation according to the embodiments of the present invention will be described with reference to the accompanying drawings.

[0037] figure 2 It is a flowchart of a visual question answering method based on external knowledge aggregation a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a visual question-answering method and system based on external knowledge aggregation. The method comprises the steps that firstly, an external knowledge graph subgraph relatedto a scene is extracted, then knowledge aggregation is conducted on the knowledge graph subgraph to obtain knowledge entity representation, and finally the entity representation and a traditional knowledge question-answering system are organically fused to finally obtain a question answer. According to the method, the external knowledge graph is introduced into a traditional visual question-answering system, the method can be applied to traditional visual questions and visual questions needing external knowledge, extra strong supervision information is not needed in the model training process, the method has high applicability, and better visual question accuracy can be obtained on multiple benchmark data sets.

Description

technical field [0001] The invention relates to the technical field of computer vision question answering, in particular to a visual question answering method and system based on external knowledge aggregation. Background technique [0002] Such as figure 1 As shown, the visual question answering task refers to: given a picture and its corresponding natural language description question, how to use the information in the picture to get the correct answer to the question. In actual scenarios, in addition to image information, it is often necessary to introduce external common sense knowledge to assist in answering visual questions. [0003] Most of the existing visual question answering methods are only based on the image and the content of the question text itself. At present, there is limited work on introducing external knowledge graphs into visual question answering. According to the degree of integration of external knowledge graphs, there are three main types of relat...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
31 Jul 2020
Publication
CN111475656A
IPC
G06F16/36; G06F16/35; G06K9/62
CPC
G06F16/367; G06F16/35; G06F18/25; Y02D10/00
Inventors
朱文武; 李国豪