A method for generating a face image based on multi-level feature guided text description

Abstract

The application discloses a method for generating a face image based on a multi-level feature of a text description. By dividing the reverse sampling of the diffusion model into two stages, firstly, the Gaussian distribution random sampling is mapped to the image hidden space under the guidance of the input text description, then the attribute words appearing in the text description are encoded, the attribute word vector and the text description vector are fused, and the content related to the age description in the attribute word vector is extracted. Then, the hidden space representation, the fused features and the features related to the age description are input into the diffusion model to constrain the generated image from different levels, reduce the uncertainty of the generated image, and improve the semantic consistency between the text and the generated image.

Description

Technical Field

[0001] This invention relates to the field of multimedia technology, specifically to a method for generating two-dimensional images based on text descriptions, and more particularly to a method for generating face images guided by text descriptions based on multi-level features. Background Technology

[0002] Text-to-image generation is an emerging research direction in the multimedia field. It requires generating images that are as realistic as possible based on natural language descriptions, while ensuring semantic consistency between the text description and the generated image. Text-to-face generation, as a sub-branch of text-to-image generation, is more challenging than traditional image generation tasks such as generating flowers or birds due to the more numerous and detailed features of facial images. Furthermore, text-to-face generation methods have practical applications in personalized advertising, virtual face design in movies and games, face recognition, and age estimation.

[0003] In existing technologies, Diffusion Transformer (DiT) generates a latent space representation of an image by iterating over the original noisy data in a high-dimensional representation space, and then decodes the latent space representation into a complex and detailed image, significantly reducing the computational complexity of the diffusion model. Simultaneously, Vision Transformer (ViT) is chosen to replace the original Unet as the backbone network of the model, leveraging the powerful modeling capabilities of the Transformer to further improve the quality of the generated image. Although the DiT model has achieved good results in text-guided image generation, due to the complexity of facial elements, a mismatch still exists between the content of the text description and the content of the generated image. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a method for generating face images based on text descriptions guided by multi-level features. This method takes attribute word features and text features as inputs, first generates a latent space representation, and then fuses multi-level features to constrain the generated image from different levels, ultimately outputting a generated face image. This reduces the uncertainty of the generated image and improves the semantic consistency between the text description and the generated image.

[0005] A method for generating face images based on text descriptions guided by multi-level features, specifically including the following steps:

[0006] Step 1: Preparation.

[0007] The real images are encoded and noise is added for subsequent model training.

[0008] Step 2: Generate text vectors.

[0009] The text description is encoded using a pre-trained CLIP model text encoder to obtain text vectors.

[0010] Step 3: Generate attribute word vectors.

[0011] Construct a facial attribute lexicon, compare the text description with the facial attribute lexicon, and perform One-Hot encoding on the attribute words in the lexicon to obtain attribute word vectors.

[0012] Step 4: Latent space representation mapping.

[0013] Random sampling is performed in a Gaussian distribution, and the sampling results are mapped to the latent space of the image by combining the text vector obtained in step two to obtain the latent space representation of the image.

[0014] Step 5: Multi-level feature fusion.

[0015] The text vector and attribute word vectors are dimensionally aligned and summed, then input into UNet for feature fusion. The input and output residuals of UNet are concatenated to obtain the vector fusionV. Age-related vectors are extracted from the attribute word vectors and combined with time-step embeddings to obtain the vector extractedV.

[0016] Step Six: Face Image Generation

[0017] The latent space representation of the image obtained in step four is used as the input of the noisy image to the diffusion model. The fusionV and extractedV vectors obtained in step five are used as conditional inputs. The real noise vector after adding noise in step one is compared with the generated noise vector output by the diffusion model to optimize the model parameters.

[0018] Input text vectors and corresponding attribute word vectors into the trained model, and output a face image.

[0019] The present invention has the following beneficial effects:

[0020] By using attribute word features and text features as input, the generated image is constrained at different levels, reducing the uncertainty of the generated image and improving the semantic consistency between the text and the generated image. Attached Figure Description

[0021] Figure 1 This is a flowchart of a method for generating face images based on text descriptions guided by multi-level features;

[0022] Figure 2 This is a schematic diagram of the implicit space representation mapping in the embodiment;

[0023] Figure 3This is a schematic diagram of multi-level feature fusion in the embodiment;

[0024] Figure 4 This is a schematic diagram of the diffusion model structure in the embodiment. Detailed Implementation

[0025] The present invention will be further explained below with reference to the accompanying drawings;

[0026] like Figure 1 As shown, a method for generating face images based on text descriptions guided by multi-level features specifically includes the following steps:

[0027] Step 1: Preparation.

[0028] s1.1. Use a VAE encoder to encode the real image corresponding to the text description to obtain the image vector Z0.

[0029] s1.2. Add a small amount of Gaussian noise ∈ to the image vector Z0 to obtain the noise vector Z1. Repeat this process T times to obtain the pure noise image. Save the noise vector Z1 obtained after T iterations. t , used for subsequent model training, t is the time step, t=1,2,…T.

[0030] Step 2: Generate text vectors.

[0031] The text encoder of the pre-trained CLIP model is used to encode the input text description to obtain a text vector.

[0032] Step 3: Generate attribute word vectors.

[0033] s3.1. Create a facial attribute dictionary. Use 11 categories of attribute words, including age, gender, face orientation, eyes, nose, mouth, ears, skin, face shape, hair, and makeup, to describe facial images, and put 162 attribute words into the facial attribute dictionary.

[0034] s3.2. Compare the input text description with the attribute word library. For attribute words that appear in the text description, encode them as 1, and for those that do not appear, encode them as 0. Obtain the encoding results of all attribute words in the face attribute word library, and use them as attribute word vectors.

[0035] Step 4: Latent space representation mapping.

[0036] s4.1. Randomly sample a vector Z of size batch_size × 16 × 16 × 4 from a Gaussian distribution. g After serialization, it is converted to a batch size of batch_size × (16 × 16 / batch_size). 2 The vector c0 is obtained by adding position information to a vector of 512 (5×512).

[0037] s4.2 Input the text vector obtained in step 2 into the fully connected layer to obtain vectors scale1, scale2, and bias1 representing scale and bias information.

[0038] s4.3. Take vector c0 as input, perform layer normalization, multiply by scale1 and add bias to obtain feature vector c1.

[0039] s4.4. Input the feature vector c1 into the multi-head self-attention module to obtain vector c2, further enhancing the model's representation ability.

[0040] s4.5 Multiply vector c2 by scale2 and add c0 to obtain vector c3, which helps stabilize model training while maintaining the feature vector range.

[0041] s4.6. Use the result vector obtained in s4.5 as the input of s4.3, and repeat steps s4.3 to s4.5 five times.

[0042] s4.7 Deserialize the vector output from s4.6, that is, map the one-dimensional vector to a two-dimensional space to obtain the predicted noise vector. Ensure consistency between input and output, such as Figure 2 As shown.

[0043] Step 5: Multi-level feature fusion.

[0044] s5.1 Input the text vector obtained in step 2 and the attribute word vector obtained in step 3 into the fully connected network respectively, perform dimension alignment, and output a 512-dimensional vector.

[0045] s5.2. Add the aligned text vector and the attribute word vector together and input them into Unet to further extract features. Then, perform a residual connection between the output of Unet and the input to obtain the feature vector fusionV.

[0046] s5.3 Extract age-related vectors from attribute word vectors and input them into a fully connected layer for fine-tuning.

[0047] s5.4. After adding the fine-tuned age-related vector to the time-step embedding of the diffusion model, the feature vector extractedV is obtained, as shown below. Figure 3 As shown.

[0048] Step Six: Face Image Generation

[0049] s6.1, construct as follows Figure 4The Diffusion Transformer (DiT) model shown inputs the feature vector extractedV into a fully connected layer to obtain scale3, scale4, and bias2 vectors related to scale and bias.

[0050] s6.3, The predicted noise vector obtained in step four As input to the DiT model, after layer normalization, it is multiplied by scale3 and then bias2 to adjust the overall face style. After being input into the multi-head self-attention module, it is multiplied by scale4 and then bias2 is added.

[0051] s6.4. After layer normalization of the result of s6.3, multiply by scale3 and add bias2. Then, input it together with the feature vector fusionV obtained in step 5 into the cross-attention module. Multiply the output of the cross-attention module by scale4 and then perform residual connection with the result of s6.3.

[0052] s6.5. After layer normalization of the result from s6.4, multiply by scale3 and add bias2. Input the result into a fully connected layer and multiply by scale4. Finally, perform a residual connection with the result from s6.4 to obtain the predicted noise vector.

[0053] s6.6, will As input, repeat steps s6.3 to s6.5 until a vector is obtained.

[0054] s6.7, via VAE decoder The image is then converted to pixel space to obtain the predicted generated image.

[0055] s6.8, The predicted noise vector generated in s6.6 The true noise vector Z obtained in step one t The current time step t is used as input to calculate the loss function, which is then used to update the model parameters.

[0056] loss = L MSE +L KL (1)

[0057] Among them, L MSE L represents the mean squared error loss between the actual noise and the predicted noise. KL The KL divergence (Kullback-Leibler divergence) represents the difference between the true and predicted distributions.

[0058]

[0059] in, This represents the model's noise prediction, where θ represents the model parameters, and τ represents the model's noise prediction. θ (y) is used to map external conditions to a vector of a specified size, where y represents the external condition. E ∈(x),∈～N(0,1),t Let P(X) be the mean squared error loss function. Let P(X) represent the true data distribution. Let X represent the true image. θ (∈ θ ,σ θ )) represents the predicted data distribution, X θ (∈ θ ,σ θ ) represents the predicted generated image. ∈ θ With σ θ These represent the predicted noise and variance, respectively. This represents the KL divergence.

[0060] To evaluate the performance of this method, a comparative experiment was conducted with classic image generation methods on the FFHQ-Text and Face2Text datasets. The complete data was randomly divided into training and test sets, and the dataset parameters are shown in Table 1.

[0061] Table 1

[0062]

[0063] The FFHQ-Text dataset contains 760 images, 500 of which were used for training. Each image is accompanied by 9 English text descriptions. The Face2Text dataset contains 400 images, 300 of which were used for training. Each image is accompanied by 1 to 4 English text descriptions, for a total of 1237 text descriptions. Each text description is between 2 and 94 words in length.

[0064] Inception Score (IS), Fréchet Inception Distance (FID), and R-precision (RP) were used as evaluation metrics for quantitative comparative analysis to assess the quality, realism, and matching degree of the generated images with the input text. The results are shown in Table 2.

[0065] Table 2

[0066]

[0067] Table 2 shows that on the FFHQ-Text dataset, compared to the baseline model DiT, our method achieved a 3.02% decrease in FID score, a 10.67% increase in IS score, and a 12.81% improvement in RP (from 40.2 to 45.35). Furthermore, the RP value exhibited the smallest fluctuation range, indicating the best stability of the generated images. Simultaneously, compared to other comparative models, our method achieved the best results across all three evaluation metrics. On the Face2Text dataset, our method achieved an IS score of 3.81 and an RP score of 55.3. Compared to the LDM model, the IS score improved by 0.56, and the RP score improved by 47.46%. These quantitative analyses demonstrate that our method significantly improves the semantic consistency between the input descriptive text and the generated image, and also enhances the realism of the generated image.

Claims

1. A method for generating face images based on text descriptions guided by multi-level features, characterized in that: Specifically, the following steps are included: Step 1: Encode the text description using the pre-trained CLIP model's text encoder to obtain text vectors; Step 2: Add words related to the face description to the face attribute lexicon, compare the text description with the face attribute lexicon, encode the attribute words in the lexicon, and obtain attribute word vectors; Step 3: Perform random sampling within a Gaussian distribution, combine it with the text vector, and map the sampling results into the image latent space to obtain the latent space representation of the image. Step 4: Align the text vector and attribute word vectors in terms of dimensions and add them together. Input the result into the feature extraction model for feature fusion to obtain the vector fusionV. Extract the age-related vector from the attribute word vectors and combine it with the time step embedding to obtain the vector extractedV. Step 5: Use the latent space representation of the image as the input of the noisy image to the diffusion model, use the vectors fusionV and extractedV obtained in Step 4 as the conditional input of the diffusion model, and transform the output vector to the pixel space to obtain the predicted generated image. Step 6: Encode and add noise to the face image corresponding to the text description, compare it with the output of the diffusion model, and optimize the model parameters; input the text vector and the corresponding attribute word vector into the trained model to obtain the face image generated by the text description.

2. The method for generating face images based on text description guided by multi-level features as described in claim 1, characterized in that: The facial attribute lexicon includes 11 categories of attribute words related to describing facial images, such as age, gender, face orientation, eyes, nose, mouth, ears, skin, face shape, hair, and makeup.

3. The method for generating face images based on text description guided by multi-level features as described in claim 1, characterized in that: The method for obtaining the latent space representation of an image is as follows: s3.

1. Obtain vector Z by random sampling from a Gaussian distribution. g Serialize and add position information to obtain vector c0; s3.2 Input the text vector into the fully connected layer to obtain vectors scale1, scale2, and bias1 representing scale and bias information; s3.

3. Take vector c0 as input, perform layer normalization, multiply by scale1 and add bias to obtain feature vector c1; s3.

4. Pass the feature vector c1 into the multi-head self-attention module to obtain vector c2; s3.5 Multiply vector c2 by scale2 and add c0 to obtain vector c3; s3.

6. Use the vector obtained in s3.5 as the input to s3.3, and repeat s3.3 to s3.5 five times; s3.7 Deserialize the vector output from s3.6 to obtain the latent space representation of the image, i.e., the predicted noise vector.

4. The method for generating face images based on text description guided by multi-level features as described in claim 1, characterized in that: The method to obtain the feature vector fusionV is as follows: input the text vector and the attribute word vector into a fully connected network respectively, align the dimensions and add them together, then input them into the Unet network, perform a residual connection between the output and input of the Unet network to obtain the feature vector fusionV.

5. The method for generating face images based on text description guided by multi-level features as described in claim 1, characterized in that: The age-related vectors from the attribute word vectors are input into a fully connected layer for fine-tuning, and then added to the time-step embeddings of the diffusion model. The eigenvector extractedV is obtained.

6. The method for generating face images based on text description guided by multi-level features as described in claim 1, characterized in that: The diffusion model comprises multiple DiT blocks, in which the image is represented in the latent space of the first DiT block. As input to a noisy image, the output is... Each subsequent DiT block uses the output of the previous DiT block as input, until the desired result is obtained. The specific steps are as follows: s5.1 Input the feature vector extractedV into the fully connected layer to obtain the scale and bias-related vectors scale3, scale4, bias2; s5.2, to After layer normalization, multiply by scale3 and add bias2, then input the multi-head self-attention module, multiply by scale4 and add bias2. s5.

3. After layer normalization of the result of s5.2, multiply by scale3 and add bias2, then input it together with the feature vector fusionV into the cross attention module. Multiply the output of the cross attention module by scale4 and then perform residual connection with the result of s5.

2. s5.4 After normalizing the result from s5.3, multiply it by scale3 and add bias2. Input the result into a fully connected layer and multiply it by scale4. Finally, perform a residual connection with the result from s5.3 to obtain the predicted noise vector. s5.5, via VAE decoder The image is then converted to pixel space to obtain the predicted generated image.

7. The method for generating face images based on text description guided by multi-level features as described in claim 6, characterized in that: The method for encoding and adding noise to the face image corresponding to the text description is as follows: Encode the real image corresponding to the text description using a VAE encoder to obtain the image vector Z0; add a small amount of Gaussian noise ∈ to the image vector Z0 to obtain the noise vector Z1; repeat this process T times to obtain a pure noise image, and save the noise vector Z1 obtained after T iterations. t t is the time step, t = 1, 2, ..., T.

8. The method for generating face images based on text description guided by multi-level features as described in claim 7, characterized in that: The method for optimizing model parameters is to use the predicted noise vector output by the DiT block. The true noise vector Z t And using the current time step t as input, calculate the loss function and update the model parameters: loss=L MSE +L KL (1) Among them, L MSE L represents the mean squared error loss between the actual noise and the predicted noise. KL The KL divergence between the true distribution and the predicted distribution is: in, This represents the model's noise prediction, where θ represents the model parameters, and τ represents the model's noise prediction. θ (y) is used to map external conditions to a vector of a specified size, where y represents the external condition; E ∈(x),∈～N(0,1),t Let P(X) be the mean squared error loss function; P(X) represents the true data distribution; X represents the true image; P(X) θ (∈ θ ,σ θ )) represents the predicted data distribution, X θ (∈ θ ,σ θ ) represents the predicted generated image; ∈ θ With σ θ These represent the predicted noise and variance, respectively. This represents the KL divergence.

Images