Visual question and answer and visual question and answer model training method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a visual question-answering and visual question-answering model training method and device, electronic equipment and a computer storage medium. The visual question-answering model training method comprises the steps: receiving and inputting a training sample of a visual question-answering model through an input part, wherein the training sample comprises a sample image and a plurality of text questions corresponding to the sample image; through a feature extraction part of the visual question and answer model, performing feature extraction on the plurality of text questions to obtain a plurality of corresponding semantic vectors, and performing feature extraction on the sample image to obtain a corresponding image feature vector; in an expression learning part of the visual question and answer model, processing the image feature vector and the semantic vectors by using an attention mechanism to obtain an image feature expression vector and a question feature expression vector; finally, through an output part of the visual question and answer model, performing question result prediction according to the image feature expression vector and the question feature expression vector, and performing training of the visual question and answer model according to a question result prediction result.

Description

technical field [0001] The embodiments of the present invention relate to the field of computer technology, and in particular, to a training method of a visual question answering model and a visual question answering method, and devices, electronic equipment, and computer storage media corresponding to the training method of the visual question answering model and the visual question answering method respectively. Background technique [0002] Visual Question Qanswering (VQA) is a learning task involving computer vision and natural language processing. A VQA system takes as input an image and a free-form, open-ended natural language question about the image, and generates a natural language answer as output. In short, VQA is question answering for a given image. [0003] In the traditional VQA system, it is necessary to take image and text questions as input, combine these two parts of information, and generate an answer as output. In a specific implementation, it summariz...

AI Technical Summary

Problems solved by technology

[0004] However, the learning method of the existing VQA system is unable to capture the relationship between the above-mentioned multiple text problems due to the independent learning of visual content and text problems, so that it cannot effectively understand the connection between natural language text and visual content, and thus cannot Perform efficient reasoning to obtain valid VQA answers

Images