Image description method of bidirectional multi-mode recursive network

Abstract

The invention provides an image description method of a bidirectional multi-mode recursive network. The image description method comprises the steps that downward images serve as a training set, and images in the training set and description sentences corresponding to the images are obtained; words emerging in the sentences in the training set are extracted, and a vocabulary is established; a pre-trained convolutional neural network is utilized to extract characteristics in the images in the data set; a bidirectional multi-mode recursive model is established, and the extracted image characteristics are fused with corresponding text characteristics; the bidirectional multi-mode recursive model is trained; a picture is input into the pre-trained model to obtain a corresponding description sentence.

Description

technical field [0001] The invention designs an image processing and pattern recognition technology, especially an image description method of a bidirectional multimodal recursive network. Background technique [0002] With the rapid development of computer vision and natural language processing, people pay more and more attention to the acquisition of visual information. How to use natural language to describe the content of an image is a focus of current research, and it is also a research content in the field of pattern recognition. In recent years, relying on convolutional neural networks to extract image features, combined with the advantages of recurrent neural networks in natural language processing, deep neural network models have become the mainstream method for image description. Its principle is based on image features, each word is sequentially generated at each moment in the running process through the recurrent neural network, and these words are combined to fo...

AI Technical Summary

Problems solved by technology

[0003] These currently existing models are used directly after extracting image features. The first is to input image features only at the beginning of the model, which will cause the loss of visual information in the model at subsequent moments. The second is to input image features at each model step. The image features are input at all times, so that although the visual information is guaranteed, the image features remain unchanged at different times, and the words generated by the model at each time change.

In addition, the existing model only considers historical text information and ignores future text information when generating words at each moment, that is, each word in the generated sentence is obtained from its previous words

Images