Text visual question-answering system and method based on concept interaction and associated semantics

Abstract

The invention provides a text visual question-answering system and method based on concept interaction and associated semantics. The system comprises an object position extraction module, a first fullconnection layer, a text information extraction module, a second full connection layer, an OCR-object graph convolutional network, a multi-gate-step mechanism graph convolutional network, a converternetwork and a bidirectional converter representation encoder BERT. According to the invention, modeling is carried out by using a position relationship between an object and text information in an image, then modeling is performed on text information and object information through the OCR-object graph convolutional network, thus learning abundant and directional features for relationship coding through a gate mechanism, and finally, precisely paying attention to objects and texts in an image through a converter network, thereby obtaining a more accurate answer.

Description

Technical field [0001] The invention belongs to the technical field of visual question answering, and in particular relates to a text visual question answering system and method based on concept interaction and associated semantics. Background technique [0002] With the development of visual and language interaction, text visual question answering algorithms have made great progress in recent years. As a branch of visual question answering, the text visual question answering algorithm focuses on how to mine the relationship between text and objects in pictures and use it to support question answering. It is also widely used in real applications, such as visual assistants for people with disabilities. Educational assistants for young children, etc. Compared with the traditional visual question answering algorithm, the goal of the text visual question answering algorithm is to require the model to understand the visual information and text information in the image at the same tim...

AI Technical Summary

Problems solved by technology

[0004] In view of the above shortcomings in the prior art, the present invention provides a text visual question answering system and method based on concept interaction and associated semantics, which solves the problem of ignoring objects and visual relationships in the prior art

Images