Construction method of deep visual question answering system for the visually impaired

Abstract

The invention discloses a method for constructing a deep visual question answering system for visually impaired persons. In the training stage of the method, firstly collected pictures and corresponding question and answer texts constitute a training set; for pictures, convolutional neural networks are used to extract features of pictures ;For the question text, use the word vector technology to convert the question into a word vector list, and use it as the input of LSTM to extract the question features; finally, perform the element dot product of the picture and the question feature and then perform a classification operation to obtain the answer prediction value, which is consistent with the training The concentrated answer labels are compared to calculate the loss, and then the model is optimized with the backpropagation algorithm. In the running phase: the client obtains the photos and question texts taken by the user and uploads them to the server. The server inputs the uploaded photos and question texts into the trained model, extracts features in the same way, and outputs the corresponding classifiers. The answer prediction value is fed back to the client; the client feeds back to the user in the form of voice output.

Description

technical field [0001] The present invention relates to the field of visual question answering (Visual Question Answering), a cross field involving both Natural Language Processing (NLP) and Computer Vision (CV), in particular to a deep How to build a visual speech system. Background technique [0002] The visually impaired population accounts for a huge proportion of the world. In addition, according to the National Bureau of Statistics, in 2014 there were about 6-7 million blind people in China, and another 12 million patients with low vision in both eyes. These people will encounter many problems in their daily life. Take the blind people traveling as an example. Although there are blind roads, many blind people dare not walk on the blind roads at all, because the blind roads are often occupied, not to mention the danger of talking on the road. . So they really need some auxiliary equipment to help them "restore the light". The current voice assistants generally only ...

AI Technical Summary

Problems solved by technology

There is also a product called Third Eye, which is mainly able to take pictures to identify objects, and then voice input, which obviously skips the stage of interacting with users, and has very large limitations

Images