A Bidirectional Multimodal Recurrent Network Image Description Method

Abstract

The invention provides an image description method for a bidirectional multimodal recursive network, comprising: downloading images as a training set, acquiring images in the training set and their corresponding description sentences; extracting words appearing in the sentences in the training set and constructing a vocabulary ; Use the pre-trained convolutional neural network to extract the features of the images in the data set; build a bidirectional multimodal recursive network model, and fuse the extracted image features with the corresponding text features; train the bidirectional multimodal recurrent network model ; Input a picture into the pre-trained model to get the corresponding description sentence.

Description

technical field [0001] The invention designs an image processing and pattern recognition technology, especially an image description method of a bidirectional multimodal recursive network. Background technique [0002] With the rapid development of computer vision and natural language processing, people pay more and more attention to the acquisition of visual information. How to use natural language to describe the content of an image is a focus of current research, and it is also a research content in the field of pattern recognition. In recent years, relying on convolutional neural networks to extract image features, combined with the advantages of recurrent neural networks in natural language processing, deep neural network models have become the mainstream method for image description. Its principle is based on image features, each word is sequentially generated at each moment in the running process through the recurrent neural network, and these words are combined to fo...

AI Technical Summary

Problems solved by technology

[0003] These currently existing models are used directly after extracting image features. The first is to input image features only at the beginning of the model, which will cause the loss of visual information in the model at subsequent moments. The second is to input image features at each model step. The image features are input at all times, so that although the visual information is guaranteed, the image features remain unchanged at different times, and the words generated by the model at each time change.

In addition, the existing model only considers historical text information and ignores future text information when generating words at each moment, that is, each word in the generated sentence is obtained from its previous words

Images