Iterative text-to-image generation method based on long-term and short-term cross-modal consistency reasoning

Abstract

The present invention discloses an iterative text image generation method based on long-term and short-term cross-modal consistency reasoning. First, a feature vector representation of the text instruction intent is obtained; then the long-term visual feature information and short-term visual feature information of the source visual image are extracted; then a target image generator is constructed, and the target image is output through the target image generator; the text instruction information, the long-term and short-term source visual features and the generated target image are input into the long-term and short-term cross-modal discriminator, and the long-term cross-modal consistency score and the short-term cross-modal consistency score are output respectively. Through long-term and short-term consistency reasoning, the present invention can better align the visual attributes of objects and related text instructions in the semantic space, and better construct the relative position relationship between objects, further ensuring the cross-modal consistency between the user text and the image, and improving the generation quality.

Description

Technical Field

[0001] The present invention relates to the field of text image generation, and in particular to an iterative text image generation method based on long-term and short-term cross-modal consistency reasoning. Background Art

[0002] In recent years, with the rapid development of deep convolutional neural networks and recurrent neural networks, the cross-modal interaction design of text and images has become one of the important research topics. The text-image generation task not only needs to generate a realistic picture but also needs to meet the instructions put forward by the user. It plays an important role in practical applications such as animation creation and art design.

[0003] Early text images mainly focused on generating a single image from available text instructions. However, some complex creations require multiple steps and prior knowledge to be successfully applied. El-Nouby et al. tried to iteratively edit the target image with continuous language as input, which brought more natural and more effective AI interaction.

[0004] In order to iteratively generate the target image, CoDraw gradually places a single visual object in the correct position through a reinforcement learning algorithm; GeNeVA proposes a recurrent image generation model that considers the image output of the current step and all past generation instructions in the pixel space to iteratively construct the visual scene; SSCR proposes a counterfactual reasoning framework that allows the model to consider out-of-distribution instructions paired with previous images to overcome the data scarcity in the iterative process. IR-GAN uses adversarial learning to design an incremental reasoning mechanism, using the generator to generate the target image and the discriminator to judge the quality of the generated image.

[0005] However, the above methods mainly focus on generating high-quality image features, and lack the ability to solve the problem of cross-modal semantic information inconsistency between vision and text during the iteration process. This will cause the text description in the instruction to be unable to align with the visual elements in the source image, further resulting in the position of objects in the target image or its own attribute information being inconsistent with the text instructions, and the required image cannot be generated according to the user's intention. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to propose an iterative text image generation method based on long-term and short-term cross-modal consistency reasoning, which effectively solves the problem of inconsistency between text and image cross-modal information during the iteration process, can more clearly understand the user's instruction intention, and ensure the good logic of the target image.

[0007] The present invention solves the above technical problems through the following technical solutions.

[0008] The iterative text image generation method based on long-term and short-term cross-modal consistency reasoning includes the following steps:

[0009] Step 1: Get the text instructions and corresponding real images in the iteration sequence. The text instructions are preprocessed using the Glove model, and the real images are used for subsequent adversarial training.

[0010] Step 2: Input the preprocessed text instruction into the instruction encoder to obtain the feature vector representation of the text instruction intent in the iterative process.

[0011] Step 3: Extract long-term visual feature information and short-term visual feature information of the source visual image from the iterative visual scene.

[0012] Step 4: Build a target image generator and output the target image through the target image generator.

[0013] The target image generator performs cross-modal fusion of long-term and short-term visual features with text instructions, and then uses an image decoder to decode the multi-modal synthetic representation and generate a 128x128 target image.

[0014] Step 5: The text instruction information extracted in step 2, the long-term and short-term source visual features obtained in step 3, and the target image generated in step 4 are input into the long-term and short-term cross-modal discriminator, and the long-term cross-modal consistency score and the short-term cross-modal consistency score are output respectively.

[0015] Step 6: Construct long-term and short-term consistency against loss.

[0016] The long-term and short-term consistency adversarial loss is divided into the loss function of the target image generator and the loss function of the long-term and short-term cross-modal discriminator. By minimizing the loss function of the target image generator, the quality and good logic of the generated target image are guaranteed; by minimizing the loss function of the long-term and short-term cross-modal discriminator, it is ensured that the model can discriminate the cross-modal consistency between long-term and short-term visual changes and user intentions.

[0017] Step 7: By alternately training the model parameters in the target image generator and the long-term cross-modal discriminator, when the long-term cross-modal consistency adversarial loss function reaches the optimal solution, the trained target image generator will output the desired target image.

[0018] Preferably, in step 2, the instruction encoder has a hierarchical text information extraction structure, including a word-level text encoder and a sentence-level text encoder;

[0019] The word-level text encoder encodes the current text instruction through a bidirectional GRU;

[0020]

[0021] Among them, d t represents the last hidden state of the bidirectional GRU and also represents the semantic representation of the current instruction; n represents the number of words in each sentence after preprocessing; Represents the feature embedding vector for each word in the entire sentence.

[0022] The sentence-level text encoder encodes the current instruction and the historical text instructions through a unidirectional GRU;

[0023] h t =GRU(d t ,h t-1 ) (2)

[0024] Among them, h t Represents the current user instruction intention; h t-1 Represents historical command information.

[0025] Preferably, the long-term visual feature information and the short-term visual feature information are acquired from the source visual image using a shallow convolutional neural network (CNN).

[0026]

[0027]

[0028] Among them, I t-1 Represents the source visual image at time t-1; I t-m Represents the source visual image at time tm. m is a configurable hyperparameter. The larger the m, the longer the long-term time span.

[0029] Furthermore, the specific method of step 4 is as follows:

[0030] In the target image generator, the long-term visual feature information and the short-term visual feature information obtained in step 3 are fused with the text instruction respectively, and then the fused synthetic features are decoded into the target image using the image decoder:

[0031] Step 4.1: Fusion of short-term visual features and user text instructions. First, the semantic representation h t Through multi-layer perception (MLP), it is projected into the visual space, and then the elements are added according to the spatial dimension in the semantic space to obtain the fused cross-modal features:

[0032]

[0033] in, Represents synthetic features that fuse semantic information and short-term visual representation.

[0034] Step 4.2: Similarly, long-term visual features are fused with text instructions for multimodal features:

[0035]

[0036] in, Represents synthetic features that fuse semantic information and long-term visual representation.

[0037] Step 4.3: Get and The input is fed into the image decoder to generate the desired target image. The image decoder consists of stacked ResNet residual upsampling layers and conditional batch normalization layers.

[0038] Furthermore, in step 5, the long-term and short-term cross-modal discriminator calculates the long-term cross-modal consistency score and the short-term cross-modal consistency score according to the current text instruction extracted in step 2, the long-term and short-term source visual features extracted in step 3, and the target image generated in step 4. The process includes the following steps:

[0039] Step 5.1: Input the target image generated in step 4.3 into a convolutional neural network to extract the visual features of the target image.

[0040] Step 5.2: Perform element-wise subtraction on the short-term visual feature information obtained in step 3 and the target visual feature obtained in step 5.1 to extract the difference, so as to obtain the short-term visual feature change; then the user instructions and short-term visual changes extracted in step 2 are input into the short-term cross-modal projection module SCP. SCP follows the design of the projection discriminator of CGANs. First, the visual features are projected to the same spatial dimension as the text instructions through a fully connected layer, and then a dot product operation is performed with the text instructions to finally calculate the short-term consistency score.

[0041]

[0042]

[0043] in, Indicates short-term visual feature changes; h t Indicates the current text instruction characteristics; f a1 (.) and f a2 (.) are two multi-layer perceptrons responsible for projecting visual features onto the t On the same feature dimension; Score for short-term cross-modal consistency.

[0044] Step 5.3: Similar to step 5.2, the long-term consistency change and long-term cross-modal consistency scores are obtained by element-wise subtraction and long-term cross-modal projection model LCP. LCP also follows the design of the projection discriminator of CGANs, and obtains the consistency score of long-term visual feature information and text instructions through projection and dot product operations.

[0045]

[0046]

[0047] in, Indicates long-term visual feature changes; h t Indicates the current text instruction characteristics; f a1 (.) and f a2 (.) are two multi-layer perceptrons respectively; Score for long-term cross-modal consistency.

[0048] Furthermore, the specific method of step 6 is as follows:

[0049] Step 6.1: Construct the loss function of the long-term cross-modal discriminator. In the case of multimodal conditions, the discriminator may focus too much on visual details and ignore the global impact of text instructions. Therefore, in addition to the loss of real and fake long-term and short-term, an asymmetric loss is introduced.

[0050]

[0051] Among them, γ and δ represent the loss weight of long-term consistency and the loss weight of inconsistency respectively; Represents real losses over the long term; Represents real losses in the short term; represents asymmetric loss; represents long-term false losses; Represents short-term false losses.

[0052] By minimizing L D To ensure that the generated image is consistent with the corresponding instructions.

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] Step 6.2: Construct the loss function of the target image generator.

[0059] By training the loss function of the target image generator, the target image generator is encouraged to better fuse the text representation learned from the instruction encoder with the source visual image to deceive the long-term and short-term cross-modal discriminator, thereby ensuring the quality and good logic of the generated target image.

[0060]

[0061] in, represents the long-term generator loss; Represents the short-term generator loss.

[0062]

[0063]

[0064] The beneficial effects of the present invention are as follows:

[0065] Through long-term and short-term consistency reasoning, the model can better align the visual attributes of objects and related text instructions in the semantic space, and better construct the relative position relationship between objects, further ensuring the cross-modal consistency between user text and the image, and improving the generation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A network structure diagram of a method according to an embodiment of the present invention;

[0067] Figure 2 It is a schematic diagram of the result of iterative image generation on the CoDraw data set based on the present invention;

[0068] Figure 3 It is a schematic diagram of the result of iterative image generation on the i-CLEVR dataset based on the present invention. DETAILED DESCRIPTION

[0069] The present invention is described in detail below in conjunction with specific implementation modes.

[0070] Figure 1 A network structure diagram of a method according to an embodiment of the present invention;

[0071] The iterative text image generation method based on long-term and short-term cross-modal consistency reasoning has the following specific steps:

[0072] Step 1: Obtain the text instructions and corresponding real images in the iterative sequence from the dataset and perform preprocessing;

[0073] Reference datasets (i-CLEVR, CoDraw), i-CLEVR is a computer-rendered dataset where each image consists of several objects with different shapes, colors, and sizes. i-CLEVR starts with an empty canvas and generates a series of (image, instruction) pairs, each of which describes a new object to be added with its shape and color. The i-CLEVR dataset contains 10,000 sequences, each of which contains 5 language instructions. CoDraw is a dataset for collaborative drawing of cartoon pictures, mainly consisting of visual scenes of children playing in the park. Each sample in the CoDraw dataset contains 58 types of different objects, such as trees, people, and balls, etc., each of which has different sizes, orientations, and positions. The CoDraw dataset consists of a total of 9993 samples, each of which consists of conversations of different lengths. For the i-CLEVR dataset, there are 6000, 2000, and 2000 examples for training, validation, and testing, respectively. For the CoDraw dataset, there are 7988, 1002, and 1002 images used for training, validation, and testing, respectively. The real image size used from the dataset is 128x128.

[0074] Step 2: Construct an instruction encoder, and extract the user's contextual instruction intention by passing the preprocessed text vector through a word-level instruction encoder and a sentence-level instruction encoder. For the embedding code of each word, the present invention uses a bidirectional GRU to encode the current instruction and calculates the last hidden state as the word-level representation of the current instruction. The dimension of the bidirectional GRU is 1024. In order to extract the contextual textual connection, a unidirectional GRU is responsible for learning the historical instruction representation, and the dimension of the sentence level is 256.

[0075] Step 3: Perceive long-term visual feature information and short-term visual feature information from the source visual image. The long-term and short-term of the long-term and short-term cross-modal consistency inference method of the present invention mainly refers to the long-term visual changes and short-term visual changes of the visual scene. Therefore, the image encoder is first used to obtain the initial long-term and short-term visual information from the source visual image. The image encoder is mainly composed of 4 ResNet downsampling blocks, the convolution kernel size is 3×3, and the average stride is 2×2. After feature extraction, the long-term and short-term visual feature maps of size 16x16 are output respectively. and The long-term time span m is set to 2 in the CoDraw dataset and to 3 in the i-CLEVR dataset.

[0076] Step 4: Input the text instructions into the target image generator, fuse the long-term and short-term visual features with the text information, and then decode it into the target image. We use a multi-layer perceptron MLP to map the text information to the same spatial dimension as the visual features, and then fuse the information by element-wise addition. and and The length, width and number of channels of the visual features are 16, 16, and 256 respectively. and The input is sent to the image decoder to generate the target image, and four ResNet upsampling blocks are stacked during the decoding process. After upsampling, conditional batch normalization CBN is used for guidance, and the final output is a generated image of size 128x128.

[0077] Step 5: Input the real image, generated image and user intent information into the long-term cross-modal discriminator to output the long-term consistency score and short-term consistency scores Following cGANs, we use a long-term cross-modal projection module LCP and a short-term cross-modal projection module SCP to measure the consistency of these two multimodal conditions. The consistency score formula of SCP and LCP is as follows:

[0078]

[0079]

[0080] in, Represents the element difference between the currently generated target image features and the long-term visual features Represents the element difference between the currently generated target image features and the short-term visual features Since the error of the generated image may be caused by historical erroneous operations, it is not enough to rely on visual increment alone. Therefore, in addition to the visual element difference, the present invention also uses the source visual image feature map as an auxiliary information.

[0081] Step 6: In order to generate realistic images and ensure semantic consistency between text descriptions and corresponding images, the present invention constructs a long-term and short-term consistency adversarial loss. The long-term and short-term consistency adversarial loss is divided into a loss function of a long-term and short-term cross-modal discriminator and a loss function of a target image generator.

[0082]

[0083]

[0084] Among them, L Drepresents the loss function of the long-term and short-term cross-modal discriminator, L G Represents the loss function of the target image generator. For the CoDraw dataset, the long-term loss weight γ = 0.4, the inconsistent loss weight δ = 0.2; for the i-CLEVR dataset, γ = 0.35, the inconsistent loss weight δ = 0. Further, the present invention uses the hinge loss to construct the long-term consistency loss ( and ), short-term consistency loss ( and ) and inconsistency loss

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] In theory, adversarial learning with hinge loss can effectively converge to the Nash equilibrium between the generator and the discriminator. When training the target image generator, the parameters of the long-term cross-modal discriminator will be frozen, and the model will optimize the loss function of the target generator:

[0091]

[0092]

[0093] in, The generator loss representing the long-term target image; Represents the generator loss for the short-term target image.

[0094] Step 7: During training, the present invention uses the Adam optimizer to optimize our word-level instruction encoder, sentence-level instruction encoder, target image generator, and long-term cross-modal discriminator, and the corresponding learning rates α are set to 0.003, 0.0003, 0.0001, and 0.0004, respectively. In addition, in order to encourage the purification of historical instruction information into the same user intent representation as long-term and short-term visual consistency, the instruction encoder is only optimized by back-propagation of the long-term and short-term cross-modal discriminator. Finally, the present invention uses NVIDIA GeForce GTX 3090Ti for training and implements our model on the PyTorch platform. The experimental test results are shown in the figure. Figure 2 and 3As shown, the results indicate that the method of the present invention can better understand the logic of text description and present a clearer image structure.

[0095] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0096] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. An iterative text image generation method based on long-term and short-term cross-modal consistency reasoning, characterized in that: The steps include: Step 1: Get the text instructions and corresponding real images in the iteration sequence. The text instructions are preprocessed using the Glove model, and the real images are used for subsequent adversarial training. Step 2: Input the preprocessed text instruction into the instruction encoder to obtain the feature vector representation of the text instruction intent in the iterative process; Step 3: extracting long-term visual feature information and short-term visual feature information of the source visual image from the iterative visual scene; Step 4: Construct a target image generator and output the target image through the target image generator; The target image generator performs cross-modal fusion of long-term and short-term visual features with text instructions, and then uses an image decoder to decode the multi-modal synthetic representation and generate a 128x128 target image; Step 5: The text instruction information extracted in step 2, the long-term and short-term source visual features obtained in step 3, and the target image generated in step 4 are input into the long-term and short-term cross-modal discriminator, and the long-term cross-modal consistency score and the short-term cross-modal consistency score are output respectively; Step 6: Construct long-term and short-term consistency against loss; The long-term and short-term consistency adversarial loss is divided into a loss function of a target image generator and a loss function of a long-term and short-term cross-modal discriminator; by minimizing the loss function of the target image generator, the quality and good logic of the generated target image are guaranteed; by minimizing the loss function of the long-term and short-term cross-modal discriminator, it is ensured that the model can discriminate the cross-modal consistency between long-term and short-term visual changes and user intentions; Step 7: By alternately training the model parameters in the target image generator and the long-term cross-modal discriminator, when the long-term cross-modal consistency adversarial loss function reaches the optimal solution, the trained target image generator will output the desired target image; In step 2, the instruction encoder has a hierarchical text information extraction structure, including a word-level text encoder and a sentence-level text encoder; The word-level text encoder encodes the current text instruction through a bidirectional GRU; Among them, d t represents the last hidden state of the bidirectional GRU and also represents the semantic representation of the current instruction; n represents the number of words in each sentence after preprocessing; The feature embedding vector representing each word in the entire sentence; The sentence-level text encoder encodes the current instruction and the historical text instructions through a unidirectional GRU; h t =GRU(d t ,h t-1 ) (2) Among them, h t Represents the current user's instruction intention; h t-1 Represents historical instruction information; The long-term visual feature information and the short-term visual feature information are obtained from the source visual image using a shallow convolutional neural network (CNN); Among them, I t-1 Represents the source visual image at time t-1; I t-m Represents the source visual image at time tm. m is a configurable hyperparameter. The larger the m, the longer the long-term time span.

2. The iterative text image generation method based on long-term and short-term cross-modal consistency reasoning according to claim 1 is characterized in that: Step 4: In the target image generator, the long-term visual feature information and the short-term visual feature information obtained in step 3 are fused with the text instruction respectively, and then the fused synthetic features are decoded into the target image using the image decoder: Step 4.1: Fusion of short-term visual features and user text instructions; First, the semantic representation h t Through multi-layer perception (MLP), it is projected into the visual space, and then the elements are added according to the spatial dimension in the semantic space to obtain the fused cross-modal features: in, Represents synthetic features that fuse semantic information and short-term visual representation; Step 4.2: Similarly, long-term visual features are fused with text instructions for multimodal features: in, Represents synthetic features that fuse semantic information and long-term visual representation; Step 4.3: Get and The input is fed into the image decoder to generate the desired target image; the image decoder consists of stacked ResNet residual upsampling layers and conditional batch normalization layers.

3. The iterative text image generation method based on long-term and short-term cross-modal consistency reasoning according to claim 2 is characterized in that: In step 5, the long-term and short-term cross-modal discriminator calculates the long-term cross-modal consistency score and the short-term cross-modal consistency score according to the current text instruction extracted in step 2, the long-term and short-term source visual features extracted in step 3, and the target image generated in step 4. The process includes the following steps: Step 5.1: Input the target image generated in step 4.3 into a convolutional neural network to extract the visual features of the target image; Step 5.2: Perform element-wise subtraction on the short-term visual feature information obtained in step 3 and the target visual feature obtained in step 5.1 to extract the difference, so as to obtain the short-term visual feature change; then input the user instructions and short-term visual changes extracted in step 2 into the short-term cross-modal projection module SCP. SCP follows the design of the projection discriminator of CGANs. First, the visual features are projected to the same spatial dimension as the text instructions through a fully connected layer, and then dot product operations are performed with the text instructions to finally calculate the short-term consistency score; in, Indicates short-term visual feature changes; h t Indicates the current text instruction characteristics; f a1 (.) and f a2 (.) are two multi-layer perceptrons responsible for projecting visual features onto the t On the same feature dimension; To score short-term cross-modal consistency; Step 5.3: Similar to step 5.2, the long-term consistency change and long-term cross-modal consistency scores are obtained respectively by element-wise subtraction method and long-term cross-modal projection model LCP; LCP also follows the design of CGANs projection discriminator, and obtains the consistency score of long-term visual feature information and text instructions through projection and dot product operations; in, Indicates long-term visual feature changes; h t Indicates the current text instruction characteristics; f a1 (.) and f a2 (.) are two multi-layer perceptrons respectively; Score for long-term cross-modal consistency.

4. The iterative text image generation method based on long-term and short-term cross-modal consistency reasoning according to claim 3 is characterized in that: The specific method of step 6 is as follows: Step 6.1: Construct the loss function of the long-term cross-modal discriminator; in the case of multimodal conditions, the discriminator may focus too much on visual details and ignore the global impact of text instructions; therefore, in addition to the real and fake losses about the long-term and short-term, an asymmetric loss is introduced; Among them, γ and δ represent the loss weight of long-term consistency and the loss weight of inconsistency respectively; Represents real losses over the long term; Represents real losses in the short term; represents asymmetric loss; represents long-term false losses; represents a short-term false loss; by minimizing L D To ensure that the generated image is consistent with the corresponding instructions; Step 6.2: Construct the loss function of the target image generator; By training the loss function of the target image generator, the target image generator is forced to better integrate the text representation learned from the instruction encoder with the source visual image to deceive the long-term cross-modal discriminator, thereby ensuring the quality and good logic of the generated target image; in, represents the long-term generator loss; represents the short-term generator loss;

Images