A Text-to-Image Method Based on Category-aware Consistency

Abstract

The present invention discloses a method for generating images from text based on category-aware consistency, comprising the following steps: obtaining the features of each image by combining metric learning and an image classification network; and then aggregating features of multiple images under each category of objects to obtain category-aware common features. The category-aware features acquired in advance are used as supervisory information in the model training process to constrain the process of generating images from text, thereby making the image more complete and detailed. Compared with referencing image labels and target positions as supervisory information, the effectiveness of the present invention can effectively constrain the category-corresponding features of the generated image without introducing a large amount of computational complexity; the generated image is more complete and detailed.

Description

Technical Field

[0001] The present invention relates to a technology for generating images from texts, and in particular to a method for generating images from texts based on category-aware consistency. Background Art

[0002] With the rapid development of deep learning, natural language and visual images, two important modalities, have made great progress. Cross-modal tasks have gradually become a hot research direction, among which text-to-image (T2I) has a wide range of applications, such as image editing, machine-assisted drawing, etc. Ordinary image generation tasks only focus on the quality of generated images. However, the T2I task requires modal fusion, modal alignment and other operations while generating corresponding real images based on text information, that is, generating real images that are closely related to the text, which is more challenging.

[0003] The T2I task not only needs to generate high-quality images, but also needs to consider the correspondence between text and images, which requires good interactive alignment between different modalities. The current T2I task often adopts a multi-stage structure to gradually generate high-resolution images from low resolution; the other is a single-stage structure to generate high-resolution images end-to-end. Due to the instability of cross-modal data flows during training, researchers often introduce additional supervisory information, such as image category information, target location, etc. However, these methods cannot limit the specific parts of some category perception in image generation, resulting in the generated image may lose the main features of the object, the overall framework offset or violate the real picture structure. The text-generated image method based on category-aware consistency can effectively solve such problems. Summary of the invention

[0004] The purpose of the present invention is to propose a method for generating images from text based on category-aware consistency in response to the deficiencies of the prior art. The method obtains the features of each image by combining metric learning and image classification network, and then classifies and aggregates them to obtain category-aware features. The obtained category-aware features are used as supervision information in the training process, and the data flow type in the image generation process can perceive the corresponding features of the category by designing a loss function to constrain the image generation process, thereby generating a more complete and detailed image.

[0005] The specific technical solution for achieving the purpose of the present invention is:

[0006] A method for generating images from text based on category-aware consistency includes the following steps:

[0007] Step 1: Get the dataset

[0008] Obtain pictures of different categories of a certain object from the Internet or a proprietary gallery through a crawler, classify them according to category information, and then manually describe each picture in text to obtain a data set of pictures and corresponding text information; the data set contains pictures of several categories of the object, each category has dozens of pictures, and each picture corresponds to several corresponding description texts. The data set is divided into 80% training set and 20% test set for subsequent model training; wherein the training set and the test set have no intersection of categories;

[0009] Step 2: Obtain category-aware features

[0010] Combined with metric learning, an image encoder based on a CNN network is built to encode the training set images to obtain image features; in the image encoding network, a cross entropy loss function is applied, and triplet loss is cited to increase the inter-class distance of image features and reduce the intra-class distance, with the weights of the two losses being 0.6 and 0.4 respectively; the backbone of the encoding network is composed of a general inception-v3 network, which remains fixed during the training process; all images are encoded through the encoder to obtain image features corresponding to all images, and then all image features are classified, and then multiple image features under each type of object are averaged and clustered to obtain a 512-dimensional feature as the category perception feature of each category of object; finally, the obtained category perception feature is stored in a file and used as prior knowledge in training;

[0011] Step 3: Category-aware features are used for training to generate real images corresponding to the text

[0012] Load the text corresponding to the image in the dataset, encode it through a general text encoder, obtain the global text representation feature, concatenate it with 100-dimensional Gaussian noise, and then input it into the generator of the Generative Adversarial Network (GAN) to obtain a 256x256 generated image I G ; The generator network consists of an upsampling layer, a convolutional layer and a LeakyReLU layer; the image I G The feature representation of the image is obtained by inputting it into the image encoder, and then the category-aware features corresponding to the category are found from the file. Finally, a loss function based on cosine similarity distance is used as an additional constraint. By adding this constraint to the generator loss function, the data stream can perceive the category correspondence information as the network back propagates continuously, generating a more complete and detailed image. After the test set text is encoded, it is spliced ​​with random noise and input into the trained generator model to generate the image corresponding to the text.

[0013] The beneficial effects of the present invention are: The present invention is easy to implement, and can obtain category-aware features that completely rely on a data set of prior knowledge, without the need to obtain additional data. At the same time, a powerful pre-training model can make it easier to extract image features. The effectiveness of the present invention is that compared with referencing image labels and target positions as supervisory information, the present invention can effectively constrain the category-related features of the generated graph without introducing a lot of computational complexity. The present invention is easy to expand, and the category-aware features acquired in advance are used as supervisory information. These features can be used as prior knowledge and can be easily added to different network structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the present invention;

[0015] Figure 2 This is a flow chart of step 2 of the present invention;

[0016] Figure 3 This is a flow chart of step 3 of the present invention;

[0017] Figure 4 This is a flow chart of the embodiment of the present invention.

[0018] Specific implementation method

[0019] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0020] See also Figure 1 The specific operation of the present invention is as follows:

[0021] Step 1: Get the dataset

[0022] Images of different categories of a certain object are obtained from the Internet or a proprietary gallery through a crawler, classified according to the category information, and then each image is manually described in text to obtain a data set of images and corresponding text information; the data set contains object images of several categories of the object, each category has dozens of images, and each image corresponds to several corresponding description texts. The data set is divided into 80% training set and 20% test set for subsequent model training; wherein, the training set and the test set have no category intersection.

[0023] Step 2: Obtain category-aware features

[0024] See also Figure 2, combined with metric learning, an image encoder based on a CNN network is built to encode the training set images to obtain image features; in the image encoding network, a cross entropy loss function is applied, and triplet loss is cited to increase the inter-class distance of image features and reduce the intra-class distance, with the weights of the two losses being 0.6 and 0.4 respectively; the backbone of the encoding network is composed of a general inception-v3 network, which remains fixed during the training process; all images are encoded through the encoder to obtain image features corresponding to all images, and then all image features are classified, and then multiple image features under each type of object are averaged and clustered to obtain a 512-dimensional feature as the category perception feature of each category of object; finally, the obtained category perception feature is stored in a file and used as prior knowledge in training;

[0025] Step 3: Category-aware features are used for training to generate real images corresponding to the text

[0026] See also Figure 3 , load the text corresponding to the image in the dataset, encode it through a general text encoder, obtain the global text representation feature, concatenate it with 100-dimensional Gaussian noise, and then input it into the generator of the Generative Adversarial Network (GAN) to obtain a 256x256 generated image I G ; The generator network consists of an upsampling layer, a convolutional layer and a LeakyReLU layer; the image I G The feature representation of the image is obtained by inputting it into the image encoder, and then the category-aware features corresponding to the category are found from the file. Finally, a loss function based on cosine similarity distance is used as an additional constraint. By adding this constraint to the generator loss function, the data stream can perceive the category correspondence information as the network back propagates continuously, generating a more complete and detailed image. After the test set text is encoded, it is spliced ​​with random noise and input into the trained generator model to generate the image corresponding to the text. Example

[0027] See also Figure 4 This embodiment takes a bird as an example, and the specific process is as follows:

[0028] Step 1: Get the dataset

[0029] The CUB dataset was obtained from the Internet, and 200 categories of bird pictures were collected, with about 40 images in each category and 10 related description texts for each picture. Different birds were classified separately to form a dataset; the dataset was divided into 80% training set and 20% test set for model training; wherein the training set and the test set have no category intersection;

[0030] Step 2: Obtain category-aware features

[0031] First, an image encoder combined with metric learning is trained for the training set, and then the classified pictures are encoded. Then, the category-aware features are obtained through averaging and clustering algorithms, and then the features are stored.

[0032] Step 3: Category-aware features are used for training to generate real images corresponding to the text

[0033] After that, the GAN network is trained. First, the text is encoded to obtain sentence features. Then, it is spliced ​​with Gaussian noise and input into the generator to obtain the generated graph corresponding to the text. The generated graph is input into the trained image encoder to obtain the generated graph features, and then a loss is made with the category-aware features of the corresponding category. This loss is added to the generator loss for back propagation. After reaching a certain number of iterations, the trained generator and discriminator models are obtained.

[0034] Load the test set data, input the text into the text encoder, and then splice the random noise input into the trained generator model to obtain the generated graph of the test text.

Claims

1. A method for generating images from text based on category-aware consistency, characterized in that: The method comprises the following steps: Step 1: Get the dataset Obtain pictures of different categories of a certain object from the Internet or a gallery through a crawler, classify them according to the category information, and then manually describe each picture in text to obtain a data set of pictures and corresponding text information; the data set contains pictures of several categories of the object, each category has dozens of pictures, and each picture corresponds to several corresponding description texts. The data set is divided into 80% training set and 20% test set for subsequent model training; wherein the training set and the test set have no intersection of categories; Step 2: Obtain category-aware features Combined with metric learning, an image encoder based on a CNN network is built to encode the training set images to obtain image features; in the image encoding network, a cross entropy loss function is applied, and a triplet loss triplet loss is cited to increase the inter-class distance of image features and reduce the intra-class distance, with the weights of the two losses being 0.6 and 0.4 respectively; the backbone of the encoding network is composed of a general perception network inception-v3 network, which remains fixed during the training process; all images are encoded through the encoder to obtain image features corresponding to all images, and then all image features are classified, and then multiple image features under each type of object are averaged and clustered to obtain a 512-dimensional feature as the category perception feature of each category of object; finally, the obtained category perception feature is stored in a file and used as prior knowledge in training; Step 3: Category-aware features are used for training to generate real images corresponding to the text Load the text corresponding to the image in the dataset, encode it through a general text encoder, obtain the global text representation feature, concatenate it with 100-dimensional Gaussian noise, and then input it into the generator of the Generative Adversarial Network (GAN) to obtain a 256x256 generated image I G ; The generator network consists of an upsampling layer, a convolutional layer and a LeakyReLU layer; the image I G The feature representation of the image is obtained by inputting it into the image encoder, and then the category-aware features corresponding to the category are found from the file, and finally a loss function based on cosine similarity distance is adopted as an additional constraint. By adding this constraint to the generator loss function, the data stream can perceive the category correspondence information and generate a more complete and detailed image as the network backpropagates. After the test set text is encoded, it is spliced ​​with random noise and input into the trained generator model to generate the image corresponding to the text.

Images