Text generation image method based on cross-modal similarity and generative adversarial network

Abstract

The invention relates to a text generation image method based on cross-modal similarity and a generative adversarial network. The method comprises the steps that S1, training a global consistency model, a local consistency model and a relation consistency model by using matched and unmatched data, wherein three models are used for obtaining global representation, local representation and relationrepresentation of a text and an image respectively; S2, obtaining global representation, local representation and relation representation of the to-be-processed text by utilizing the trained global consistency model, local consistency model and relation consistency model; S3, connecting the global representation, the local representation and the relation representation of the to-be-processed textin series to obtain text representation of the to-be-processed text; S4, converting the text representation of the to-be-processed text into a condition vector by utilizing an Fca condition enhancement module; and S5, inputting the condition vector into a generator to obtain a generated image. Compared with the prior art, the method has the advantages of considering local and relation informationand the like.

Description

technical field [0001] The invention relates to image retrieval and matching technology, in particular to a text generation image method based on cross-modal similarity and generation confrontation network. Background technique [0002] In recent years, deep neural networks (DNNs) have achieved great success, especially neural network models trained for discriminative tasks. For example, Convolutional Neural Networks (CNNs) show great promise in computer vision. But the discriminative model focuses on representation learning and cannot capture the data distribution. Learning generative models that can explain complex data distributions is a long-standing problem in the field of deep learning. As a sub-problem of it, image generation from text based on Generative Adversarial Networks (GANs) has made a series of progress. [0003] The text is fed into the generator and the discriminator as conditions, and these deep learning models based on GANs can produce rich and colorfu...

AI Technical Summary

Problems solved by technology

However, these conditional generative adversarial networks are only based on the overall text expression and lack detailed local information, so they cannot generate clear high-quality images.

Therefore, existing GAN-based successes are limited to small sample corpora, and it remains a challenge for complex image generation with many objects.

Images