Image description method based on distribution word vector CNN-RNN network

Abstract

The invention discloses an image description method based on a distributed word vector CNN-RNN network, and the method is characterized by comprising the following steps of 1), generating a distributed representation word vector; 2) generating a distribution representation label; 3) generating the distributed representation semantic tags; 4) carrying out the network design; and 5) generating the descriptive statements of the image. The method is introduced into an original CNN-RNN model, so that a more accurate result can be generated, and the CNN subnet provides richer semantic content for the RNN subnet, so that the whole CNN-RNN network model can still maintain the advantage of structuralization. According to the method, the low-dimensional dense distribution representation can be easily embedded into a large number of words to form a complete semantic space, the visual content can be better mapped into the semantic space, and the visual content can be more accurately summarized andthe vector space can be more fully utilized to supervise the CNN optimization direction based on the supervision signal of the distribution representation word vector design.

Description

technical field [0001] The invention relates to the technical field of intelligent image processing, in particular to an image description method based on a distributed word vector CNN-RNN network. Background technique [0002] In the field of computer vision, breakthroughs continue to be made in basic vision tasks such as image classification, object detection, and semantic segmentation. People's interest gradually turns to image description, a more complex and advanced visual task. The specific task of image description is to generate descriptive sentences of semantic information in the image. Therefore, it is not only necessary to identify and understand (refer to action) the relevant content in the image, but also to describe it in the form of natural language. In practical applications such as assistive systems for the blind, image retrieval, and intelligent interactive systems, the ability to use images to generate corresponding natural language descriptions is crucia...

AI Technical Summary

Problems solved by technology

For example, the network proposed by Mao et al. has a parallel structure, and the image embedding and word embedding are fused together to complete the sentence construction through the idea of ​​feature fusion; The initial state of the hidden layer of the LSTM unit h 0 and c 0 , the prediction of the sentence starts at t=1; the method proposed by You et al. directly uses the image embedding as the input of the initial state of the LSTM unit; in the work of Liu et al., the semantic specification layer is proposed to realize the structured training strategy. The two subnetworks solve the problems of difficult training, long training time, and noise interference to CNN during training. At the same time, the concept of display semantics is introduced to make the tasks of the two subnetworks clear in the network. However, the display semantics using one-hot representation has obvious problems. lack of

The number of words involved in image description is tens of thousands, and the semantic space formed by one-hot representation is very limited. Therefore, the semantic space formed by one-hot representation ignores a large number of semantics and cannot meet the needs of image description tasks.

Images