Visual question-answering method and system based on semantic alignment and storage medium

Abstract

The invention provides a visual question-answering method and system based on semantic alignment and a storage medium, and relates to the technical field of visual question-answering. According to theembodiment of the invention, the method comprises the steps: firstly obtaining and preprocessing a data set, extracting original image features and target position features according to an original image, generating an image description statement according to the target position features, obtaining an image description word, question features and image description statement features, and carryingout the semantic alignment of the original image features and the image description word; and obtaining a first image feature, obtaining a second image feature according to the original image featureand the image description statement feature, obtaining a third image feature according to the original image feature and the question feature, fusing the three image features, the image description statement feature and the question feature to obtain a comprehensive feature, and predicting a final answer result. Therefore, the importance of the image information is highlighted, the information involved in the feature fusion process is perfected, and the finally generated answer result is more accurate.

Description

technical field [0001] The present invention relates to the technical field of visual question answering, in particular to a method and system for visual question answering based on semantic alignment, and a storage medium. Background technique [0002] Visual question answering is a learning task involving computer vision and natural language processing. It is to let the computer learn the input pictures and questions to output an answer that conforms to the rules of natural language and the content is logical. It only focuses on a certain part of the picture according to the different questions. objects, and some questions require a certain amount of common sense reasoning to get the answers. Therefore, visual question answering has higher requirements on the semantic understanding of images than general picture-based speaking, and it also faces greater challenges. [0003] At present, the existing visual question answering technology usually uses the attention mechanism t...

AI Technical Summary

Problems solved by technology

[0006] Aiming at the deficiencies of the prior art, the present invention provides a visual question answering method, system and storage medium based on semantic alignment, which solves the problem that the existing visual question answering technology only involves the original image features and question features in the feature fusion process, and the information is insufficient. Improvement, technical problems that lead to inaccurate answers in the final generation

Images