Chinese image semantic description method combined with multilayer GRU based on residual error connection Inception network

Abstract

The invention discloses a Chinese image semantic description method combined with multilayer GRU based on a residual error connection Inception network, and belongs to the field of computer vision andnatural language processing. The method comprises the steps: carrying out the preprocessing of an AI Challenger image Chinese description training set and an estimation set through an open source tensorflow to generate a file at the tfrecord format for training; pre-training an ImageNet data set through an Inception_ResNet_v2 network to obtain a convolution network pre-training model; loading a pre-training parameter to the Inception_ResNet_v2 network, and carrying out the extraction of an image feature descriptor of the AI Challenger image set; building a single-hidden-layer neural network model and mapping the image feature descriptor to a word embedding space; taking a word embedding characteristic matrix and the image feature descriptor after secondary characteristic mapping as the input of a double-layer GRU network; inputting an original image into a description model to generate a Chinese description sentence; employing an evaluation data set for estimation through employing the trained model and taking a Perplexity index as an evaluation standard. The method achieves the solving of a technical problem of describing an image in Chinese, and improves the continuity and readability of sentences.

Description

technical field [0001] The invention belongs to the field of computer vision and natural language processing, and in particular relates to a Chinese image semantic description method based on a residual connection-based Inception network combined with a multi-layer GRU network. Background technique [0002] Chinese text description of images is a technology that combines computer vision and Chinese natural language processing (NLP). , how to more efficiently search for the image information required by users in a large number of images, only relying on the traditional keyword retrieval method not only has the problem of slow search speed but also has the problem of inaccurate image positioning. For scientific research, in In the era of big data, if you rely on manpower to label a large number of pictures, generating labels is particularly unrealistic. Accurate labeling of images can not only greatly improve people's image retrieval efficiency, but also provide accurate data...

AI Technical Summary

Problems solved by technology

[0003] The machine learning classification algorithm that relies on the combination of traditional manual features can also realize the semantic description of images, but the generated Chinese sentences are not very coherent and readable from both objective and subjective perspectives.

The application of deep learning network has improved this problem, but there are still shortcomings such as low recognition rate.

Images