Multimode recurrent neural network picture description method based on FCN feature extraction

Abstract

The invention provides a multimode recurrent neural network picture description method based on FCN feature extraction. A multimode model composed of three parts, namely, a recurrent neural network (RNN), a fully convolutional neural network (FCN) and a multimode layer is obtained by training massive images labeled with text description, and automatic generation of text description of any input test image is achieved. By means of the method, image features can be effectively extracted, more detail information of the images can be retained, and the relation between words in the text description and the images can be better established. The method has significant advantages on semantics-based description between image salient targets or scenes.

Description

technical field [0001] The present invention relates to the field of artificial intelligence, and more specifically, relates to a multimodal recurrent neural network image description method based on FCN feature extraction. Background technique [0002] In recent years, the cyclic neural network RNN ​​and convolutional neural network CNN have achieved success in natural language processing and image classification processing respectively, which has led to the emergence of a combination of cyclic neural network and convolutional neural network for automatic image description in the field of machine learning. method. Automatic generation of image description is an important branch of artificial intelligence, which can be widely used in image retrieval, blind navigation and so on. Therefore, it has attracted the attention of more and more researchers. In 2011, Mikolov et al. proposed a recurrent neural network model for natural language processing, which achieved the best res...

AI Technical Summary

Problems solved by technology

[0003] Although M-RNN can achieve good results under various test standards, the model can only generate descriptions for large-area targets in the image.

For some areas that occupy a small area in the image, their information has been lost when the convolutional neural network extracts image features, so image descriptions of these lost areas cannot be generated

Therefore, the model ignores more detailed information in the image and cannot generate a more complete image description

Images