Method, device and medium for visual question answering

Abstract

The invention discloses a visual question answering method, device and medium, which are applied in the field of deep learning to obtain target text and target images, and convert the target text and target images into text data and image data respectively; the text data and image data are respectively Input to the Transformer-based model to extract text features and image features, and then input the text features and image features to the Transformer model to obtain fusion features, and finally input the fusion features to the classifier to obtain the answer to the visual question answering task. In terms of feature extraction, the Transformer model is completely used to speed up extraction, save computing costs, reduce the amount of operational parameters, aggregate text features and image features, and achieve better interaction between image features and text features, improving feature representation capabilities. , improve the running speed of feature extraction, and improve the answer accuracy of visual question answering.

Description

technical field [0001] The invention relates to the field of deep learning, in particular to a method, device and medium for visual question answering. Background technique [0002] With the rapid development of artificial intelligence, machine learning is usually given to output the results people want. However, in the field of deep learning, computer vision (Computer Vision, CV) and natural language processing (Natural language processing, NLP) have entered a bottleneck state, and multi-modal deep learning has gradually become a research hotspot. Combining CV and NLP The visual question answering (Visual Question Answering, VQA) of VQA is based on a picture and natural language questions about the picture as input, which is given to the machine to understand and fuse the image, and the information contained in the language modality will output the answer. [0003] The feature extraction of the existing VQA model includes three modules, text feature extraction, image featu...

AI Technical Summary

Problems solved by technology

[0003] The feature extraction of the existing VQA model includes three modules, text feature extraction, image feature extraction and feature fusion module. For the text feature extraction module, the commonly used recurrent neural network (Recurrent Neural Network, RNN), long short-term memory network ( Long Short Term Memory networks, LSTM), gated recurrent unit (Gated Recurrent Unit, GRU), etc., commonly used for image feature extraction modules are convolutional neural networks (Convolutional Neural Networks, CNN), multi-layer perceptron (Multi-layer Perceptron, MLP), etc., the feature extraction accuracy of different feature extractors is also different, there are high and low, and the information from image features and text features does not achieve better interaction when performing feature fusion, which greatly reduces the feature representation ability and The running speed of feature extraction has led to a decrease in the accuracy of the results of visual question answering

Images