Image description confrontation generation method based on reinforcement learning

Abstract

The invention discloses an image description confrontation generation method based on reinforcement learning. The method comprises the following steps: S1, retrieving similar images and text description thereof according to a to-be-described image; s2, constructing an image description generation network based on an attention mechanism, introducing the attention mechanism and a long and short-term memory network when a text is generated from an image to be described, and obtaining a generated text description by combining the output of the long and short-term memory network with the extracted image features through the attention mechanism and loss calculation; s3, constructing an image description similarity pairing discrimination network; s4, performing pairing discrimination through a discrimination network by using the to-be-described image, the text description of the annotation pairing of the to-be-described image, the text description generated by the generative network and the text description of the similar image of the to-be-described image, and performing joint iterative optimization on the generative network and the discrimination network according to a reward value output by the discrimination network; and S5, inputting an image of which description is to be generated into the trained generative network to generate text description.

Description

technical field [0001] The present invention relates to the intersecting technical field of computer vision and NLP, in particular to an image description confrontation generation method based on reinforcement learning. Background technique [0002] Image captioning is an emerging research task. Over the past two decades, the fields of natural language processing (NLP) and computer vision (CV) have made tremendous progress in analyzing and generating text and understanding images. Although both fields share a set of approaches based on machine learning and artificial intelligence, they have been studied separately in the past and have not interacted much in the scientific community. However, in recent years, with the progress in the field of artificial intelligence and the development of deep learning models, scholars have become more and more interested in the problem of combining language and visual information. At the same time, there is a large amount of data combining...

AI Technical Summary

Problems solved by technology

The advancement of image retrieval has greatly helped the research on the uniqueness of image descriptions, but the uniqueness of description generation in image descriptions still needs to be improved

Images