A method for generating images based on attribute-driven GANs
By constructing an attribute memory module and a conditional generation network with cross-modal feature alignment, the problem of insufficient description by a single sentence is solved, achieving accurate and stable text-to-image generation and producing realistic image samples.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-21
- Publication Date
- 2026-03-03
AI Technical Summary
Existing technologies for text-to-image generation suffer from the problem that a single sentence description is insufficient to accurately cover the image description, and redundant sentence expressions lead to the neglect of attribute information, resulting in difficulties in training cross-modal generative networks and mode collapse.
An attribute memory module is constructed, attribute sets are labeled through a multimodal dataset, sentence and attribute features are extracted by a pre-trained text encoder, a conditional generation network framework jointly regulated by sentences and attributes is designed, and the training network is optimized by cross-modal feature alignment and loss function.
It achieves more accurate image description, stable cross-modal GAN model training, generates realistic and highly semantically matched images, and improves the generation effect.
Smart Images
Figure CN115690245B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, and relates to cross-modal network optimization and text-to-image generation, particularly a method for text-to-image generation based on attribute-driven GAN. Background Technology
[0002] Generative Adversarial Networks (GANs) aim to transform random distributions into image distributions through adversarial mechanisms, thereby enabling the sampling of realistic image samples from random distributions. This technique has been extensively studied in recent years and has become dominant in the field of image generation. There are various variants of GAN models, among which Conditional GAN (cGAN) is the most widely studied. It generates images that conform to given conditions. This model uses different modalities of data as conditions, leading to a variety of different research tasks, such as image translation, speech-to-image generation, and text-to-image generation.
[0003] Text-to-image generation aims to generate realistic and semantically matching images based on given text descriptions, with wide applications such as design assistance, automatic illustration, and interactive entertainment. However, the modal gap between the input text and the output image makes optimizing cross-modal generative networks particularly difficult, easily leading to modality collapse or even training failure. Existing methods typically use a pre-trained text encoder to extract text features from a single sentence, using these features as conditions to control image generation in a conditional generative adversarial network. However, using only a single sentence has two limitations: 1) a single sentence's textual expression is insufficient to accurately cover the entire image description; 2) sentence expressions often contain redundant formats such as articles and conjunctions, which are irrelevant. This makes the text feature representation extracted by the text encoder prone to overlooking some key information describing the image sample, namely attribute descriptions, as attributes are the most decisive factors in describing an image. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention aims to research and design a method for attribute-driven memory enhancement in text-to-image generation. This method achieves attribute-driven text-to-image generation by constructing an attribute memory module and proposing a novel cross-modal training approach, achieving state-of-the-art results. The technical solution of this invention consists of four parts: construction of the attribute set, feature transformation between sentence description and attribute description, construction of a conditional generation network jointly regulated by sentences and attributes, and optimization of the conditional generation network framework.
[0005] The specific technical solution adopted in this invention is as follows:
[0006] A method for generating images from text based on attribute-driven GANs includes the following steps:
[0007] S1. For the multimodal dataset, construct an attribute set by describing all the attributes present in the entire dataset; and each image sample in the multimodal dataset is labeled with sentence-level sentence descriptions and attribute-level multi-attribute labels;
[0008] S2. Based on the pre-trained text encoder, the sentence description of each image sample in the multimodal dataset is directly processed by the text encoder to obtain the sentence feature representation. At the same time, the text encoder is used to extract the text features of each attribute description in the attribute set, and the corresponding sentence features and attribute feature sets are obtained. The attribute feature set is used as an attribute memory module to provide assistance for network training. Then, according to the multi-attribute labels of each image sample, the corresponding attribute features are extracted from the attribute memory module and integrated into a common attribute feature representation.
[0009] S3. Set sentence feature representation and attribute feature representation as conditions for generating images, and construct a conditional generation network framework jointly controlled by sentences and attributes as a text image generator; the conditional generation network framework is divided into two adversarially trained networks: a generation model and a discriminative model.
[0010] The generative model is constructed from a series of upsampled ResBlocks, with Gaussian noise sampled from a Gaussian distribution as input. Each upsampled ResBlock is subject to feature modulation using either attribute feature representation or sentence feature representation, where attribute feature representation serves as a low-level feature condition and sentence feature representation serves as a high-level feature condition. Each upsampled ResBlock double-upsamples the input feature map, then modifies the feature map using the feature representation corresponding to the current ResBlock based on Adaptive Instance Normalization, and inputs the feature map after feature modulation into the next upsampled ResBlock. After serial processing of multiple upsampled ResBlocks, the mapping from low-dimensional noise to a high-dimensional image is finally achieved.
[0011] The discriminative model is constructed from a series of downsampled ResBlocks, and its input is a high-dimensional image generated by the generative model. Each downsampled ResBlock is responsible for performing convolution operations on the input feature map and downsampling it by a factor of two. After serial processing by multiple downsampled ResBlocks, the high-dimensional image is transformed into a low-dimensional feature embedding. Finally, the low-dimensional feature embedding is passed through a classifier for multi-attribute prediction and through a convolutional layer for true / false prediction.
[0012] S4. Using the multimodal dataset as training data, the conditional generation network framework is trained and optimized through attribute memory module updates and cross-modal feature alignment, and the resulting generation model is used as a text image generator.
[0013] The attribute memory module update includes a reading process and a writing process. The reading process involves extracting the attribute representation of the sample based on multiple attribute labels, while the writing process involves real-time updates through backpropagation of the network. Furthermore, the attribute module has two different update strategies: a single-sample update mechanism and a joint-sample update mechanism.
[0014] The cross-modal feature alignment is achieved by aligning image feature representations with corresponding sentence feature representations and attribute feature representations through contrastive learning.
[0015] The total loss function used when training and optimizing the conditional generative adversarial network framework includes the discriminative loss of the conditional generative adversarial network, the multi-attribute prediction loss, and the cross-modal feature alignment loss;
[0016] S5. After the conditional generation network framework is trained, it generates corresponding images based on the optimized attribute memory module and text image generator, based on the specified sentence description and multi-attribute labels.
[0017] Preferably, the multimodal dataset mentioned in S1 is:
[0018]
[0019] In the formula: (x i ,s i ,y i (This is a multimodal dataset) One of the samples, x i For the i-th image, s i Let y be the sentence description of the i-th image. i Let N be the multi-attribute label of the i-th image; N is the multimodal dataset. Total number of image samples;
[0020] The set of attributes mentioned is:
[0021]
[0022] In the formula: For multimodal datasets The set of attributes, a1, a2, ..., a n These are descriptions of all n attributes contained in the multimodal dataset.
[0023] Preferably, in S2, for any sentence description s, the sentence feature expression is extracted in the following way:
[0024] e s =T(s)
[0025] In the formula: e s T(·) represents the sentence features, and T(·) is a pre-trained text encoder.
[0026] The extraction method for the attribute feature representation is as follows:
[0027] e a =y·M a
[0028] In the formula: e a For attribute feature expression; M a For the attribute memory module, the pre-trained text encoder T(·) processes the attribute set. The results obtained from feature extraction are as follows:
[0029] As a preferred embodiment, the expression for the conditional generation network framework described in S3 is as follows:
[0030]
[0031] In the formula: Generate a network framework for the aforementioned conditions. In Gaussian noise space, For sentence description space, For attribute tag space, For image space; this expression represents the network. Noise space exist and Transformed into image space under common conditions
[0032] As a preferred embodiment, the expressions for reading and writing the attribute memory module in S4 are as follows:
[0033] M read =Forward(y,M) a )
[0034]
[0035] Where: M read and M write These represent the reading and writing processes of the attribute memory module, respectively; Forward(·) and Backward(·) represent the inference phase and gradient backpropagation phase of the network, respectively; during the reading process, the attribute representation is changed from multiple attribute labels y in the attribute memory module M. aFeature extraction is performed during the writing process; the loss function of the generative model in the conditional generation network framework is used during the writing process. Memory is updated during gradient backpropagation.
[0036] The single-sample update mechanism of the attribute memory module is expressed as follows:
[0037] e a =y·M a
[0038] In the formula: e a The attribute feature representation obtained by the single-sample update mechanism consists of the label y of a single sample in the attribute memory module M. a Feature extraction is performed during M; during gradient backpropagation in network training, M... a The update is only constrained by a single sample;
[0039] The joint sample update mechanism expression of the attribute memory module is as follows:
[0040] e' a =y·H L
[0041] H l+1 =LeakyRELU(C·H l ·W l )
[0042]
[0043]
[0044] In the formula: e' a The attribute feature representation obtained by the joint sample update mechanism is composed of the attribute memory module M. a Feature propagation is performed through an L-layer GCN model, and finally, the feature is extracted from the multi-attribute label y; H L The feature, dimension, and attribute memory module M is the output of the last layer of the GCN model. a Same; H l+1 and H l C and C' are the features output at layers l+1 and l of the GCN model, respectively; C and C' are the adjacency matrices of the GCN model, and C' is used for both the construction of the GCN model and feature propagation; C ij With C i ' j Let be the elements in the i-th row and j-th column of the adjacency matrices C and C', respectively; p and τ are both hyperparameter thresholds; P ij Let be the probability that the i-th attribute label and the j-th attribute label appear simultaneously in the dataset.
[0045] As a preferred embodiment, the cross-modal feature alignment process described in S4 is as follows: First, a pre-trained text encoder is used to extract sentence feature representations and attribute feature representations. Then, the backbone network of the discriminative model is used to extract image feature representations. Finally, feature alignment between images and sentences, images and attributes, and real images and generated images is achieved in the latent space.
[0046] Furthermore, in the cross-modal feature alignment process, for any two different modal feature representations u and v, the feature alignment loss function is defined as follows:
[0047]
[0048]
[0049] In the formula: cos(u i ,v j ) represents the distance metric between two different feature representations of the i-th image sample, u i and v i Let u and v represent the i-th image sample, respectively; m is the sample size of a training batch; and η is a hyperparameter.
[0050] Preferably, the discriminative loss of the conditional generative adversarial network described in S4 is defined as follows:
[0051]
[0052]
[0053] In the formula: To determine the adversarial loss of the model, This represents the adversarial loss for the generative model; max(·,·) denotes the maximum value function.
[0054] Preferably, the multi-attribute prediction loss of the conditional generative adversarial network is defined as follows:
[0055]
[0056]
[0057]
[0058] In the formula: To determine the multi-attribute prediction loss of the model, For multi-attribute discrimination loss in generative models; Cross-entropy loss for multi-attribute prediction; r It is the attribute classification prediction of a real image through a discriminative model, y r It is the attribute classification labeling of real images; fIt is the attribute classification prediction of the generated image through a discriminative model, y f It is the attribute classification labeling of the generated image; y i For the predicted value of the i-th attribute, l i Label the i-th attribute with a value;
[0059] Preferably, the cross-modal feature alignment loss of the conditional generative adversarial network is defined as follows:
[0060]
[0061]
[0062] In the formula: To determine the cross-modal feature alignment loss of the model, The cross-modal feature alignment loss for the generative model; where the loss term is... The definition is as follows:
[0063]
[0064]
[0065]
[0066]
[0067]
[0068] In the formula: and These are feature alignments between the real image and the attributes, and between the generated image and the attributes, respectively. and Feature alignment is performed between real images and sentences, and between generated images and sentences, respectively. Align features between real and generated images; Let x represent the feature alignment loss function; x represents the real image, x f Indicates the generated image; D img (·) indicates the operation of mapping an image into an image feature representation by adding a fully connected layer to the backbone network of the discriminant model.
[0069] As a preferred embodiment, the total loss function used for training the conditional generative adversarial network framework described in S4 is defined as follows:
[0070]
[0071]
[0072] In the formula: To determine the overall loss of the model, where To match the perceptual gradient penalty regularization term. λ1 represents the overall loss of the generative model; λ1, λ2, λ3, λ4, and λ5 are all weight terms.
[0073] Compared to traditional text-to-image generation methods, this invention provides an attribute-driven GAN-based text-to-image generation method that leverages the expressive power of supplementary sentences to more accurately describe an image sample. It allows the network to be jointly controlled by sentences and attributes during training, achieving refined image generation. Furthermore, it enables multi-modal transformations and stable training of cross-modal GAN models. The invention achieves advanced levels in both subjective qualitative performance and objective quantitative performance on existing datasets. Attached Figure Description
[0074] Figure 1 This is a flowchart of the method for generating images from text based on attribute-driven GAN according to the present invention;
[0075] Figure 2 This is a schematic diagram illustrating the sentence and attribute joint control of image generation proposed in this invention;
[0076] Figure 3 A schematic diagram of the conditional generation network framework of the present invention;
[0077] Figure 4 This is a schematic diagram illustrating the construction of the attribute memory module proposed in this invention;
[0078] Figure 5 This is a demonstration of the experimental dataset for this invention;
[0079] Figure 6 The image generation experiment results are shown in this invention. Detailed Implementation
[0080] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0081] This invention proposes a novel text representation, called an attribute memory module, which serves as a supplementary representation to enrich the textual information of a sentence. Thus, the text-to-image generation task is transformed into an attribute-driven conditional generation problem. The key aspects of this invention in solving the problem lie in two areas: 1) how to construct the attribute memory module; and 2) how to combine attributes and sentences for cross-modal model training.
[0082] For constructing the attribute memory module, this invention first collects all possible attribute descriptions in the entire dataset and combines them into an attribute set, then transforms it into an attribute memory module using a text encoder. Next, this invention uses attribute labels to extract sample-related attributes from the attribute memory module and fuses them to form a common attribute feature representation. To obtain an attribute memory module with higher generalization ability, this invention designs two different update mechanisms to optimize the attribute memory module: a single-sample update mechanism and a joint-sample update mechanism. Specifically, the single-sample update mechanism obtains the corresponding attribute feature representation through the attribute label of a single sample, so that the network updates the memory module only related to the current sample during the update process. The joint-sample update mechanism, on the other hand, pre-constructs the adjacency matrix of the entire dataset using GCN, which contains the co-occurrence relationships between attributes, and updates the attribute memory module through GCN propagation, so that the features extracted from the attribute labels can also update other attributes related to the current attribute during backpropagation.
[0083] After constructing the attribute memory module, this invention designs an optimization strategy for joint learning of attributes and sentences to achieve transformations between multiple modalities, namely sentences, attributes, and images. Specifically, within the framework of this invention, an image sample should strictly correspond to its sentence description and attribute description. Therefore, this invention proposes to utilize contrastive learning to achieve feature alignment between multiple modal expressions, bringing modalities belonging to the same sample closer together and pushing modalities belonging to different samples further apart. This strategy can effectively alleviate the difficulty of cross-modal network training, thereby generating realistic image samples with high semantic matching.
[0084] The basic workflow of text-to-image generation based on attribute-driven GAN in this invention is as follows: Start → Import cross-modal dataset → Obtain attribute labels and sentence descriptions for each sample → Attribute set construction → Import pre-trained text encoder → Extract text features to form sentence feature representations and attribute memory modules → Extract attribute feature representations from attribute memory modules based on attribute labels → Set sentence feature representations and attribute feature representations as joint conditions for conditional image generation → Training begins → Attribute memory module update and cross-modal model training optimization → Training complete → Given a sample's sentence description and attribute labels, generate an image → Text-to-image generation complete. The specific implementation of this workflow is described in detail below.
[0085] like Figure 1 As shown, in a preferred embodiment of the present invention, a method for generating images from text based on attribute-driven GANs is provided, the method comprising the following steps:
[0086] S1. Construct the attribute set of the dataset.
[0087] Given a multimodal dataset, predefine the attribute set of the dataset, that is, collect all possible attribute descriptions in the entire dataset. Generally, an image can contain multiple different sentences and attribute descriptions. Label each image sample with a sentence description and its corresponding attribute label. Thus, an image has two levels of labeling: sentence-level descriptions and attribute-level multi-attribute labels.
[0088] In this step, the multimodal dataset is used as the target domain image dataset, and its definition is as follows:
[0089]
[0090] In the formula: (x i ,s i ,y i (a multimodal dataset) One of the samples, x i For the i-th image, s i Let y be the sentence description of the i-th image. i Let be the multi-attribute labels for the i-th image. Multimodal dataset. There are a total of N image samples.
[0091] In this step, the attribute set is defined as follows:
[0092]
[0093] In the formula: For multimodal datasets The set of attributes, a1, a2, ..., a n These are descriptions of all n attributes contained in the multimodal dataset.
[0094] S2, Feature Transformation of Sentence Description and Attribute Description
[0095] Given a pre-trained text encoder for extracting text features, after obtaining the sentence description and attribute set in step S1, each attribute description in each sentence description and attribute set is processed by the text encoder to extract text features, resulting in the corresponding sentence features and attribute feature sets. This invention refers to the attribute feature set as the attribute memory module, which provides assistance for subsequent network training. Given an image sample in a multimodal dataset and its sentence description and attribute labels, this invention obtains the corresponding feature representations through different strategies: 1) The sentence description of the image sample is directly processed by the text encoder to obtain the sentence feature representation; 2) Based on the attribute labels of the image sample, the corresponding attribute features are extracted from the attribute memory module and integrated into a common attribute feature representation.
[0096] In this step, for any sentence description s, the method for extracting sentence features is defined as follows:
[0097] e s =T(s)
[0098] In the formula: e s T represents the sentence features, and T(·) is a pre-trained text encoder.
[0099] In this step, the method for extracting attribute feature representations described in S2 is defined as follows:
[0100] e a =y·M a
[0101]
[0102] In the formula: e a For attribute feature representation, M a For the attribute memory module, the text encoder T(·) processes the attribute set. The result is obtained through feature extraction.
[0103] S3. Construction of a Conditional Generation Network Framework for Joint Regulation of Sentence and Attributes
[0104] Based on the sentence feature representation and attribute feature representation obtained in step S2, these are set as conditions for generating images, and a corresponding conditional generation network framework is designed to obtain a text image generator that achieves joint regulation of sentence and attribute to generate images.
[0105] like Figure 2 As shown, in this invention, the task of generating images from text is achieved by coupling an attribute memory module and a text image generator, wherein the attribute memory module has been constructed in step S2. Figure 3 As shown, this conditional generative network framework consists of a generative model and a discriminative model, also known as a generator and a discriminator, which are a pair of networks trained adversarially.
[0106] The generative model described above is constructed from a series of upsampled ResBlocks, with Gaussian noise sampled from a Gaussian distribution as input. Each upsampled ResBlock is modulated using either attribute feature representation or sentence feature representation, where attribute feature representation serves as a low-level feature condition and sentence feature representation serves as a high-level feature condition. In this embodiment, the generative model is constructed from six upsampled ResBlocks (i.e., the upsampled fusion blocks in the figure). The Gaussian noise sampled from the Gaussian distribution is passed through a fully connected layer to form a feature map, which is then input into the first upsampled ResBlock. The first three ResBlocks are modulated using attribute feature representation, while the last three ResBlocks are modulated using sentence feature representation. Each upsampled ResBlock doubles the input feature map, then modulates the feature map using the feature representation corresponding to the current ResBlock based on Adaptive Instance Normalization, and inputs the modulated feature map into the next upsampled ResBlock. After serial processing by multiple upsampled ResBlocks, the mapping from low-dimensional noise to high-dimensional images is finally achieved.
[0107] Furthermore, the aforementioned discriminative model is constructed from a series of downsampled ResBlocks, with the high-dimensional image generated by the generative model as its input. Each downsampled ResBlock is responsible for performing a convolution operation on the input feature map and then downsampling it by a factor of two. After serial processing by multiple downsampled ResBlocks, the high-dimensional image is transformed into a low-dimensional feature embedding. In this embodiment, the discriminative model is constructed from six downsampled ResBlocks (i.e., the downsampled blocks in the figure), and these six downsampled blocks constitute the backbone network of the discriminative model. The features output by the backbone network of the discriminative model... Figure 1 On the one hand, a low-dimensional image feature representation is obtained after passing through a fully connected layer, which is used for cross-model alignment during the training phase. On the other hand, it is concatenated with the sentence feature representation formed by replication to form a low-dimensional feature embedding. Finally, the low-dimensional feature embedding is passed through a classifier for multi-attribute prediction and through a convolutional layer for true / false prediction.
[0108] Based on this, the conditional generative network can sample a Gaussian noise from a Gaussian distribution and transform it into an image through a series of upsampling modules. During this process, the image features are regulated by different feature expressions at different network layers, thereby realizing the text-to-image generation task that is jointly regulated by sentences and attributes.
[0109] In this step, the expression definition of the conditional generation network framework is as follows:
[0110]
[0111] In the formula: Generate a network framework for the aforementioned conditions. In Gaussian noise space, For sentence description space, For attribute tag space, For image space. This expression represents the network. Noise space exist and Transformed into image space under common conditions
[0112] S4. Optimization of Conditional Generation Network Framework: Attribute Memory Module Update and Cross-Modal Feature Alignment
[0113] Using the multimodal dataset obtained in step S1 as training data, the conditional generative network framework is trained and optimized by updating the attribute memory module and aligning cross-modal features. The resulting generative model serves as a text image generator.
[0114] The training of this conditional generation network framework mainly includes two core parts: attribute memory module updates and cross-modal feature alignment.
[0115] 1) Attribute memory module update:
[0116] The reading process of the attribute memory module involves extracting the attribute representation of the sample based on the multi-attribute labels, while the writing process involves real-time updates via backpropagation of the network. This setup enables the network to obtain a more generalizable attribute memory module during training, thereby providing the network with better attribute representations. This invention designs two different strategies for attribute module updates: a single-sample update mechanism and a joint-sample update mechanism.
[0117] 2) Cross-modal feature alignment:
[0118] The training of conditional generative networks under the task of this invention requires achieving modal alignment between text and images. This invention aligns image feature representations with corresponding sentence feature representations and attribute feature representations through contrastive learning, thereby achieving accurate cross-modal correspondence and alleviating the difficulties of cross-modal network training.
[0119] In this step, the expressions for reading and writing the attribute memory module are defined as follows:
[0120] M read =Forward(y,M) a )
[0121]
[0122] Where: M read and Mwrite These represent the reading and writing processes of the attribute memory module, respectively. Forward(·) and Backward(·) represent the inference phase and gradient backpropagation phase of the network, respectively. During the reading process, the attribute representation is stored in the attribute memory module M by the attribute label y. a Feature extraction is performed during the writing process; the loss function of the generative model in the conditional generation network framework is used during the writing process. Memory is updated during gradient backpropagation.
[0123] In this step, the single-sample update mechanism process expression of the attribute memory module is defined as follows:
[0124] e a =y·M a
[0125] In the formula: e a The attribute feature representation obtained by the single-sample update mechanism consists of the label y of a single sample in the attribute memory module M. a The features are extracted during this process. Therefore, when performing gradient backpropagation during network training, M... a The update is only constrained by a single sample.
[0126] In this step, the joint sample update mechanism of the attribute memory module is defined as follows:
[0127] e' a =y·H L
[0128] H l+1 =LeakyRELU(C·H l ·W l )
[0129]
[0130]
[0131] In the formula: e' a The attribute feature representation obtained by the joint sample update mechanism is composed of the attribute memory module M. a Feature propagation is performed through an L-layer GCN model, and finally, the feature is extracted from the multi-attribute label y; H L The feature, dimension, and attribute memory module M is the output of the last layer of the GCN model. a Same; H l+1 and H l C and C' are the features output at layers l+1 and l of the GCN model, respectively; C and C' are both adjacency matrices of the GCN model, and C' is used for both the construction of the GCN model and feature propagation in this invention; C ij With C i 'j are the elements in the i-th row and j-th column of the adjacency matrices C and C', respectively; p and τ are both hyperparameter thresholds (in this embodiment, they can be 0.4 and 0.25, respectively); P ij Let be the probability that the i-th attribute label and the j-th attribute label appear simultaneously in the dataset.
[0132] The above-mentioned attribute memory module construction and update process is as follows: Figure 4 As shown.
[0133] In this step, the cross-modal feature alignment process is as follows: First, a pre-trained text encoder is used to extract sentence feature representations and attribute feature representations. Then, the backbone network of the discriminant model is used to extract image feature representations. Finally, feature alignment between images and sentences, images and attributes, and real images and generated images is achieved in the latent space.
[0134] In this step, and during the cross-modal feature alignment process, for any two different modal feature representations u and v in image-sentence, image-attribute, or real image-generated image, the feature alignment loss function is defined as follows:
[0135]
[0136]
[0137] In the formula: cos(u i ,v j ) represents the distance metric between two different feature representations of the i-th image sample, u i and v i Let u and v represent the i-th image sample, respectively; m is the sample size of a training batch. η is a hyperparameter, set to 0.1 in this embodiment.
[0138] In this step, the optimized total loss function during the training of the conditional generative adversarial network framework includes: the discriminative loss of the conditional generative adversarial network, the multi-attribute prediction loss of the conditional generative adversarial network, and the cross-modal feature alignment loss of the conditional generative adversarial network.
[0139] In this step, when training the conditional generative adversarial network framework, the discriminative loss of the generative adversarial network is defined as follows:
[0140]
[0141]
[0142] In the formula: To determine the adversarial loss of the model, This is the adversarial loss for the generative model. max(·,·) is the maximum value function.
[0143] In this step, when training the conditional generative adversarial network framework, the multi-attribute prediction loss of the generative adversarial network is defined as follows:
[0144]
[0145]
[0146]
[0147] In the formula: To determine the multi-attribute prediction loss of the model, This is used for the multi-attribute discrimination loss in the generative model. Cross-entropy loss for multi-attribute prediction. r It is the attribute classification prediction of a real image through a discriminative model, y r It is the attribute classification labeling of real images; f It is the attribute classification prediction of the generated image through a discriminative model, y f It is the attribute classification labeling of the generated image; y i For the predicted value of the i-th attribute, l i The value is labeled for the i-th attribute.
[0148] In this step, the alignment process of the multiple modal features corresponds to the loss function, i.e., the loss term. It is expressed as follows:
[0149]
[0150]
[0151]
[0152]
[0153]
[0154] In the formula: and These are feature alignments between the real image and the attributes, and between the generated image and the attributes, respectively. and Feature alignment is performed between real images and sentences, and between generated images and sentences, respectively. Align features between real and generated images; Let x represent the feature alignment loss function; x represents the real image, x f Indicates the generated image; D img (·) indicates the operation of mapping an image into an image feature representation by adding a fully connected layer to the backbone network of the discriminant model.
[0155] In this step, when training the conditional generative adversarial network framework, the cross-modal feature alignment loss of the generative adversarial network is defined as follows:
[0156]
[0157]
[0158] In the formula: To determine the cross-modal feature alignment loss of the model, This is the cross-modal feature alignment loss for the generative model.
[0159] In this step, the overall optimization loss of the conditional generative adversarial network framework training is defined as follows:
[0160]
[0161]
[0162] In the formula: To determine the overall loss of the model, where To match the perceptual gradient penalty regularization term. The overall loss of the generative model is represented by λ1, λ2, λ3, λ4, and λ5, which are weight terms. The above-mentioned matching-aware gradient penalty regularization term... It is an existing, publicly known technology.
[0163] Finally, once the above-mentioned conditional generation network framework has been trained, it can be used based on the optimized attribute memory module and text image generator to... Figure 2 The framework shown inputs the specified multi-attribute labels into the attribute memory module, and inputs the sentence description and the output of the attribute memory module into the text image generator to generate the corresponding image.
[0164] The above methods are applied to specific embodiments below so that those skilled in the art can better understand the effects of the present invention.
[0165] Example
[0166] The simulation experiment is conducted based on the above method. The implementation method of this embodiment is as described above, and the specific steps will not be described in detail. The results will only be shown below.
[0167] This embodiment uses the method of the present invention to train and test performance on the CUB and COCO datasets. Partial examples of the two datasets are shown below. Figure 5 As shown.
[0168] Finally, the implementation results of the simulation experiment are shown in Tables 1 and 2. Figure 6As shown in Table 1, this invention was tested on the CUB and COCO datasets, and the corresponding FID and IS scores are presented. On the CUB dataset, the invention achieved the best performance in both FID and IS metrics. This indicates that the invention performs optimally on the CUB dataset in terms of both generated quality and image diversity. On the COCO dataset, the FID score also achieved the best results, only the IS score did not outperform other methods. This is because the COCO dataset is a multi-object dataset, while the IS score evaluates the performance of a single object; therefore, the IS score cannot accurately reflect the image quality on COCO. In comparison, the FID score is more objective, and the FID scores of this invention are significantly higher than other methods.
[0169] As shown in Table 2, the ablation experiment results of the three core schemes proposed in this invention demonstrate that single-sample update, joint sample update, and cross-modal alignment all improve the performance of the baseline scheme, i.e., the FID score decreases. The table shows that the joint sample update strategy is more effective than the single-sample update strategy because it considers the attribute distribution of the entire dataset and the global symbiotic relationships between attributes. Furthermore, the joint sample update strategy combined with cross-modal alignment achieves the best results.
[0170] Figure 6 This document provides a visualization of the generation performance of this invention compared to the baseline scheme DF-GAN. It is evident that, on both the CUB and COCO datasets, the images generated by this invention are more realistic and have a higher degree of matching with the input text. Particularly on COCO, this invention tends to generate more reasonable images. While training on multi-object datasets is particularly difficult, and the model still cannot achieve the effect of realistic images due to insufficient data, its comparative performance is clearly superior.
[0171] Table 1. Experimental results of the method of the present invention on the FID / IS datasets of CUB and COCO datasets.
[0172]
[0173] Table 2 shows the ablation experiments of the proposed scheme in this invention, and the FID experimental results on the CUB dataset.
[0174] plan Experiment 1 Experiment 2 Experiment 3 Experiment 4 Experiment 5 Experiment 6 Baseline √ √ √ √ √ √ Single Sample Update √ √ Joint Sample Update √ √ Cross-modal alignment √ √ √ CUB-FID↓ 14.81 9.31 9.04 12.43 8.92 8.57
[0175] In summary, the embodiments of the present invention, unlike existing technologies, achieve superior quantitative and qualitative results in text-to-image generation scenarios. They enable stable training of cross-modal GAN models and joint generation of images from attribute sentences.
[0176] It should be understood that this embodiment is for illustrative purposes only and is not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.
Claims
1. A method for generating images from text based on attribute-driven GANs, characterized in that, Includes the following steps: S1. For the multimodal dataset, construct an attribute set by describing all the attributes present in the entire dataset; and each image sample in the multimodal dataset is labeled with sentence-level sentence descriptions and attribute-level multi-attribute labels; S2. Based on the pre-trained text encoder, the sentence description of each image sample in the multimodal dataset is directly processed by the text encoder to obtain the sentence feature representation. At the same time, the text encoder is used to extract the text features of each attribute description in the attribute set, and the corresponding sentence features and attribute feature sets are obtained. The attribute feature set is used as an attribute memory module to provide assistance for network training. Then, according to the multi-attribute labels of each image sample, the corresponding attribute features are extracted from the attribute memory module and integrated into a common attribute feature representation. S3. Set sentence feature representation and attribute feature representation as conditions for generating images, and construct a conditional generation network framework jointly controlled by sentences and attributes as a text image generator; the conditional generation network framework is divided into two adversarially trained networks: a generation model and a discriminative model. The generative model is constructed from a series of upsampled ResBlocks, whose input is Gaussian noise sampled from a Gaussian distribution. Each upsampled ResBlock is subject to feature modulation using either attribute feature representation or sentence feature representation, where attribute feature representation serves as a low-level feature condition and sentence feature representation serves as a high-level feature condition. Each upsampled ResBlock double-upsamples the input feature map, then modifies the feature map using the feature representation corresponding to the current ResBlock based on Adaptive Instance Normalization, and inputs the feature map after feature modulation into the next upsampled ResBlock. After serial processing of multiple upsampled ResBlocks, the mapping from low-dimensional noise to a high-dimensional image is finally achieved. The discriminative model is constructed from a series of downsampled ResBlocks, and its input is a high-dimensional image generated by the generative model. Each downsampled ResBlock is responsible for performing convolution operations on the input feature map and downsampling it by a factor of two. After serial processing by multiple downsampled ResBlocks, the high-dimensional image is transformed into a low-dimensional feature embedding. Finally, the low-dimensional feature embedding is passed through a classifier for multi-attribute prediction and through a convolutional layer for true / false prediction. S4. Using the multimodal dataset as training data, the conditional generation network framework is trained and optimized through attribute memory module updates and cross-modal feature alignment, and the resulting generation model is used as a text image generator. The attribute memory module update includes a reading process and a writing process. The reading process involves extracting the attribute representation of the sample based on multiple attribute labels, while the writing process involves real-time updates through backpropagation of the network. Furthermore, the attribute module has two different update strategies: a single-sample update mechanism and a joint-sample update mechanism. The cross-modal feature alignment is achieved by aligning image feature representations with corresponding sentence feature representations and attribute feature representations through contrastive learning. The total loss function used when training and optimizing the conditional generative adversarial network framework includes the discriminative loss of the conditional generative adversarial network, the multi-attribute prediction loss, and the cross-modal feature alignment loss; S5. After the conditional generation network framework is trained, it generates corresponding images based on the optimized attribute memory module and text image generator, based on the specified sentence description and multi-attribute labels.
2. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: The multimodal dataset mentioned in S1 is: In the formula: (x i ,s i ,y i (This is a multimodal dataset) One of the samples, x i For the i-th image, s i Let y be the sentence description of the i-th image. i Let N be the multi-attribute label of the i-th image; N is the multimodal dataset. Total number of image samples; The set of attributes mentioned is: In the formula: For multimodal datasets The set of attributes, a1, a2, ..., a n These are descriptions of all n attributes contained in the multimodal dataset.
3. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: In S2, for any sentence description s, the sentence feature representation is extracted as follows: e s =T(s) In the formula: e s T(·) represents the sentence features, and T(·) is a pre-trained text encoder. The extraction method for the attribute feature representation is as follows: e a =y·M a In the formula: e a It is a representation of attribute features; M a For the attribute memory module, the pre-trained text encoder T(·) processes the attribute set. The results obtained from feature extraction are as follows:
4. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: The expression for the conditional generative network framework described in S3 is as follows: In the formula: Generate a network framework for the aforementioned conditions. In Gaussian noise space, For sentence description space, For attribute tag space, For image space; this expression represents the network. Noise space exist and Transformed into image space under common conditions 5. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: The expressions for reading and writing attributes in S4 are as follows: M read =Forward(y,M a ) Where: M read and M write These represent the reading and writing processes of the attribute memory module, respectively; Forward(·) and Backward(·) represent the inference phase and gradient backpropagation phase of the network, respectively; during the reading process, the attribute representation is changed from multiple attribute labels y in the attribute memory module M. a Feature extraction is performed during the writing process; the loss function of the generative model in the conditional generation network framework is used during the writing process. Memory is updated during gradient backpropagation; The single-sample update mechanism of the attribute memory module is expressed as follows: e a =y·M a In the formula: e a The attribute feature representation obtained by the single-sample update mechanism consists of the label y of a single sample in the attribute memory module M. a Feature extraction is performed to obtain the results. During gradient backpropagation in network training, M a The update is only constrained by a single sample; The joint sample update mechanism expression of the attribute memory module is as follows: e' a =y·H L H l+1 =LeakyRELU(C H l ·IN l ) In the formula: e' a The attribute feature representation obtained by the joint sample update mechanism is composed of the attribute memory module M. a Feature propagation is performed through an L-layer GCN model, and finally, the feature is extracted from the multi-attribute label y; H L The feature, dimension, and attribute memory module M is the output of the last layer of the GCN model. a Same; H l+1 and H l C and C' are the features output at layers l+1 and l of the GCN model, respectively; C and C' are the adjacency matrices of the GCN model, and C' is used for both the construction of the GCN model and feature propagation; C ij With C i ' j Let be the elements in the i-th row and j-th column of the adjacency matrices C and C', respectively; p and τ are both hyperparameter thresholds; P ij Let be the probability that the i-th attribute label and the j-th attribute label appear simultaneously in the dataset.
6. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: The process of cross-modal feature alignment described in S4 is as follows: First, a pre-trained text encoder is used to extract sentence feature representations and attribute feature representations. Then, the backbone network of the discriminative model is used to extract image feature representations. Finally, feature alignment between images and sentences, images and attributes, and real images and generated images is achieved in the latent space. Furthermore, in the cross-modal feature alignment process, for any two different modal feature representations u and v, the feature alignment loss function is defined as follows: In the formula: cos(u i ,v j ) represents the distance metric between two different feature representations of the i-th image sample, u i and v i Let u and v represent the i-th image sample, respectively; m is the sample size of a training batch; and η is a hyperparameter.
7. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: The discriminative loss of the conditional generative adversarial network described in S4 is defined as follows: In the formula: To determine the adversarial loss of the model, This represents the adversarial loss for the generative model; max(·,·) denotes the maximum value function.
8. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: The multi-attribute prediction loss of the conditional generative adversarial network is defined as follows: In the formula: To determine the multi-attribute prediction loss of the model, For multi-attribute discrimination loss in generative models; Cross-entropy loss for multi-attribute prediction; r It is the attribute classification prediction of a real image through a discriminative model, y r It is the attribute classification labeling of real images; f It is the attribute classification prediction of the generated image through a discriminative model, y f It is the attribute classification labeling of the generated image; y i For the predicted value of the i-th attribute, l i The value is labeled for the i-th attribute.
9. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: The cross-modal feature alignment loss of the conditional generative adversarial network is defined as follows: In the formula: To determine the cross-modal feature alignment loss of the model, The cross-modal feature alignment loss for the generative model; where the loss term is... The definition is as follows: In the formula: and These are feature alignments between the real image and the attributes, and between the generated image and the attributes, respectively. and Feature alignment is performed between real images and sentences, and between generated images and sentences, respectively. Align features between real and generated images; Let x represent the feature alignment loss function; x represents the real image, x f Indicates the generated image; D img (·) indicates the operation of mapping an image into an image feature representation by adding a fully connected layer to the backbone network of the discriminant model.
10. The method for generating images from text based on attribute-driven GAN according to claim 1, characterized in that: The total loss function used for training the conditional generative adversarial network framework described in S4 is defined as follows: In the formula: To determine the overall loss of the model, where To match the perceptual gradient penalty regularization term, λ1 represents the overall loss of the generative model; λ2, λ3, λ4, and λ5 are all weight terms.