An image description method based on distributed word vector cnn-rnn network

Abstract

The invention discloses an image description method based on a distributed word vector CNN-RNN network, which is characterized in that it comprises the following steps: 1) generation of distribution representation word vector; 2) generation of distribution representation label; 3) distribution representation semantic label 4) network design; 5) generating descriptive sentences for images. This method is introduced into the original CNN-RNN network model so that it can generate more accurate results, enable the CNN subnetwork to provide richer semantic content to the RNN subnetwork, and enable the entire CNN-RNN network model to remain structured Advantages, the low-dimensional dense distribution representation in this method can easily embed a large number of words to form a complete semantic space, the visual content can be better mapped to the semantic space, and the supervisory signal designed based on the distribution representation word vector can be more accurately summarized Visual content and fuller use of vector space supervised CNN optimization direction.

Description

technical field [0001] The invention relates to the technical field of intelligent image processing, in particular to an image description method based on a distributed word vector CNN-RNN network. Background technique [0002] In the field of computer vision, breakthroughs continue to be made in basic vision tasks such as image classification, object detection, and semantic segmentation. People's interest gradually turns to image description, a more complex and advanced visual task. The specific task of image description is to generate descriptive sentences of semantic information in the image. Therefore, it is not only necessary to identify and understand (refer to action) the relevant content in the image, but also to describe it in the form of natural language. In practical applications such as assistive systems for the blind, image retrieval, and intelligent interactive systems, the ability to use images to generate corresponding natural language descriptions is crucia...

AI Technical Summary

Problems solved by technology

For example, the network proposed by Mao et al. has a parallel structure, and the image embedding and word embedding are fused together to complete the sentence construction through the idea of ​​feature fusion; The initial state of the hidden layer of the LSTM unit h 0 and c 0 , the prediction of the sentence starts at t=1; the method proposed by You et al. directly uses the image embedding as the input of the initial state of the LSTM unit; in the work of Liu et al., the semantic specification layer is proposed to realize the structured training strategy. The two subnetworks solve the problems of difficult training, long training time, and noise interference to CNN during training. At the same time, the concept of display semantics is introduced to make the tasks of the two subnetworks clear in the network. However, the display semantics using one-hot representation has obvious problems. lack of

The number of words involved in image description is tens of thousands, and the semantic space formed by one-hot representation is very limited. Therefore, the semantic space formed by one-hot representation ignores a large number of semantics and cannot meet the needs of image description tasks.

Images