Bidirectional text image generation method and system based on semantic consistency

Abstract

The invention provides a bidirectional text image generation method and system based on semantic consistency, and the method comprises the steps: obtaining a natural language, inputting the natural language into a text encoder, and extracting a word vector and a sentence vector; inputting sentence vectors into an image generation network, inputting word vectors into a word-level space attention mechanism module, connecting image features generated in each stage in the generation network with the word vectors adjusted by an attention mechanism in series to serve as input of the next stage of the generation network, and performing multi-stage refinement, and finally generating an image; inputting the generated image into an image encoder, and extracting image features; inputting the image features into a long short-term memory network with a sentinel mechanism, and outputting a re-described text; and after the image and the re-description text are optimized by adopting two confrontation loss functions, introducing semantic text reconstruction loss based on cross entropy to further optimize the image until the re-description text of the image is consistent with a natural language, and outputting the image.

Description

technical field [0001] The invention belongs to the technical field of cross-modal image generation, and in particular relates to a method and system for generating images based on semantic consistency of two-way text. Background technique [0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art. [0003] Text-based image generation is to generate high-resolution and visually realistic images that match a given natural language description. It has broad application prospects in the fields of virtual reality, entertainment, electronic sports games, and computer-aided design. In recent years, generative adversarial networks (GAN) have made great progress in generating realistic images. Using the framework of generative adversarial networks, text generation images have proposed many methods to generate high-quality images. Significant progress has been made in text-to-image genera...

AI Technical Summary

Problems solved by technology

In addition, the attention mechanism has been widely used in the field of text generation images, but the traditional attention mechanism only pays attention to the global vector of the whole sentence, but ignores the attention to the word-level local information

[0005] From the above analysis, it can be concluded that the focus of current text-generated images is to ensure the visual authenticity of the generated images, but the semantic inconsistency between the generated images and the text has not been resolved.

Images