Method for generating face image under guidance of text description based on multi-level features

Abstract

The invention discloses a method for generating a face image under the guidance of text description based on multi-level features. Reverse sampling of a diffusion model is divided into two stages, firstly, Gaussian distribution is randomly sampled and mapped to an image hidden space under the guidance of input text description, then coding is performed according to attribute words appearing in the text description, feature fusion is performed on attribute word vectors and text description vectors, and finally, feature fusion is performed. And extracting the content related to the age description in the attribute word vector. And then the hidden space representation, the fusion feature and the feature related to the age description are input into a diffusion model together, so that the generated image is constrained from different levels, the uncertainty of the generated image is reduced, and the semantic consistency between the text and the generated image is improved.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technology and to a method for generating a two-dimensional image based on a text description, and in particular to a method for generating a face image guided by a text description based on multi-level features. Background Art

[0002] Text-to-image is an emerging research direction in the multimedia field. It requires generating images that are as realistic as possible based on natural language descriptions and ensuring semantic consistency between the text description and the generated image. Text-to-face is a sub-branch of text-to-image generation. Compared with traditional image generation tasks such as generating flowers or birds, face images have more and more detailed features and are more challenging. In addition, the text-to-face method has practical application value in personalized advertising, virtual face design for movies and games, face recognition, and age estimation.

[0003] In the existing technology, Diffusion Transformer (DiT) generates a latent space representation of an image by iterating the original noise data in a high-dimensional representation space, and then decodes the latent space representation into a complex and detailed image, which significantly reduces the computational complexity of the diffusion model. At the same time, Vision Transformer (ViT) is used to replace the original Unet as the backbone network of the model, and the quality of the generated image is further improved with the help of the powerful modeling ability of Transformer. Although the DiT model has achieved good results in the field of text-guided image generation, due to the complexity of facial elements of the human face, there is still a mismatch between the content of the text description and the content of the generated image. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a method for generating facial images guided by text description based on multi-level features. The method takes attribute word features and text features as input at the same time, first generates a latent space representation, and then integrates the multi-level features to constrain the generated image from different levels. Finally, the generated facial image is output, reducing the uncertainty of the generated image to improve the semantic consistency between the text description and the generated image.

[0005] A method for generating a face image based on a text description guided by multi-level features, specifically comprising the following steps:

[0006] Step 1: Preparation.

[0007] The real image is encoded and denoised for subsequent model training.

[0008] Step 2: Generate text vector.

[0009] The text encoder of the pre-trained CLIP model is used to encode the text description to obtain the text vector.

[0010] Step 3: Generate attribute word vectors.

[0011] Build a face attribute vocabulary, compare the text description content with the face attribute vocabulary, perform One-Hot encoding on the attribute words in the vocabulary, and obtain the attribute word vector.

[0012] Step 4: Latent space representation mapping.

[0013] Random sampling is performed in the Gaussian distribution, and combined with the text vector obtained in step 2, the sampling result is mapped into the image latent space to obtain the latent space representation of the image.

[0014] Step 5: Multi-level feature fusion.

[0015] The text vector and the attribute word vector are dimensionally aligned and added, and input into Unet for feature fusion. The input and output residual of Unet are connected to obtain the vector fusionV. The age vector is extracted from the attribute word vector and combined with the time step embedding to obtain the vector extractedV.

[0016] Step 6: Face image generation

[0017] The latent space representation of the image obtained in step 4 is used as the noise image input of the diffusion model. The vectors fusionV and extractedV fused in step 5 are used as conditional inputs. The real noise vector after encoding and denoising in step 1 is compared with the generated noise vector output by the diffusion model to optimize the model parameters.

[0018] Input the text vector and the corresponding attribute word vector into the trained model, and output the face image.

[0019] The present invention has the following beneficial effects:

[0020] The attribute word features and text features are used as input to constrain the generated images from different levels, reduce the uncertainty of the generated images, and improve the semantic consistency between the text and the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a method for generating face images guided by text description based on multi-level features;

[0022] Figure 2 is a schematic diagram of latent space representation mapping in an embodiment;

[0023] Figure 3is a schematic diagram of multi-level feature fusion in an embodiment;

[0024] Figure 4 Schematic diagram of the diffusion model structure in the embodiment. DETAILED DESCRIPTION

[0025] The present invention will be further explained below with reference to the accompanying drawings;

[0026] like Figure 1 As shown, a method for generating a face image based on a text description guided by multi-level features specifically includes the following steps:

[0027] Step 1: Preparation.

[0028] s1.1. Use the VAE encoder to encode the real image corresponding to the text description to obtain the image vector Z0.

[0029] s1.2. Add a small amount of Gaussian noise ∈ to the image vector Z0 to obtain the noise vector Z1. After repeating T times, a pure noise image is obtained. Save the noise vector Z obtained T times. t , used for subsequent model training, t is the time step, t=1,2,…T.

[0030] Step 2: Generate text vector.

[0031] The text encoder of the pre-trained CLIP model is used to encode the input text description to obtain a text vector.

[0032] Step 3: Generate attribute word vectors.

[0033] s3.1. Create a face attribute word library. Use 11 attribute words such as age, gender, face orientation, eyes, nose, mouth, ears, skin, face shape, hair, and makeup to describe face images, and put 162 attribute words into the face attribute word library.

[0034] s3.2. Compare the input text description content with the attribute word library. For the attribute words that appear in the text description, encode them as 1, and those that do not appear are encoded as 0. The encoding results of the attribute words in all face attribute word libraries are obtained as the attribute word vector.

[0035] Step 4: Latent space representation mapping.

[0036] s4.1. Randomly sample a vector Z of size batch_size×16×16×4 from a Gaussian distribution g , after serialization, it is converted into a batch_size×(16×16 / patch_size 2 )×512 vector and add the position information to obtain vector c0.

[0037] s4.2. Input the text vector obtained in step 2 into the fully connected layer to obtain vectors scale1, scale2, and bias1 representing scale and bias information.

[0038] s4.3. Take vector c0 as input, perform layer normalization, multiply by scale1 and add bias to obtain feature vector c1.

[0039] s4.4. Pass the feature vector c1 into the multi-head self-attention module to obtain vector c2, further enhancing the model representation capability.

[0040] s4.5. Multiply vector c2 by scale2 and add c0 to obtain vector c3, which helps stabilize the model training while maintaining the feature vector range.

[0041] s4.6. Use the result vector obtained in s4.5 as the input of s4.3 and repeat steps s4.3 to s4.5 five times.

[0042] s4.7. Deserialize the vector output by s4.6, that is, map the one-dimensional vector to the two-dimensional space to obtain the predicted noise vector Ensure the consistency of input and output, such as Figure 2 shown.

[0043] Step 5: Multi-level feature fusion.

[0044] s5.1. Input the text vector obtained in step 2 and the attribute word vector obtained in step 3 into the fully connected network respectively, align the dimensions, and output a 512-dimensional vector.

[0045] s5.2. Add the aligned text vector and the attribute word vector and input them into Unet to further extract features. Perform a residual connection between the output and input of Unet to obtain the feature vector fusionV.

[0046] s5.3. Extract age-related vectors from the attribute word vectors and input them into the fully connected layer for fine-tuning.

[0047] s5.4. Add the fine-tuned age-related vector to the time step embedding of the diffusion model to obtain the feature vector extractedV, such as Figure 3 shown.

[0048] Step 6: Face image generation

[0049] s6.1、Build as Figure 4The Diffusion Transformer (DiT) model shown in Figure 1 inputs the feature vector extractedV into the fully connected layer to obtain vectors scale3, scale4, and bias2 related to scale and bias.

[0050] s6.3, the predicted noise vector obtained in step 4 As the input of the DiT model, the layer is normalized, multiplied by scale3 and then bias2 to adjust the overall style of the face. After entering the multi-head self-attention module, it is multiplied by scale4 and then

[0051] s6.4, normalize the result of s6.3, multiply it by scale3 and add bias2, then input it into the cross-attention module together with the feature vector fusionV obtained in step 5, multiply the output of the cross-attention module by scale4, and then perform a residual connection with the result of s6.3.

[0052] s6.5, after layer normalization, the result of s6.4 is multiplied by scale3 and then bias2. The result is input into the fully connected layer and then multiplied by scale4. Finally, a residual connection is performed with the result of s6.4 to obtain the predicted noise vector

[0053] s6.6, will As input, repeat steps s6.3 to s6.5 until you get the vector

[0054] s6.7, through the VAE decoder Convert to pixel space and get the predicted generated image.

[0055] s6.8, the prediction noise vector generated by s6.6 The real noise vector Z obtained in step 1 t And the current time step t is used as input to calculate the loss function loss to update the model parameters:

[0056] loss = L MSE +L KL (1)

[0057] Among them, L MSE represents the mean square error loss between the real noise and the predicted noise, L KL Represents the KL divergence (Kullback-Leibler divergence) between the true distribution and the predicted distribution:

[0058]

[0059] in, represents the model noise prediction, θ represents the model parameters, τ θ (y) is used to map external conditions to a vector of specified size, where y represents the external conditions. ∈(x),∈～N(0,1),t is the mean square error loss function. P(X) represents the true data distribution. X represents the true image. P(X θ (∈ θ ,σ θ )) represents the predicted data distribution, X θ (∈ θ ,σ θ ) represents the predicted generated image. ∈ θ With σ θ denote the noise and variance of the prediction, respectively. represents the KL divergence.

[0060] In order to evaluate the performance of this method, comparative experiments were carried out on the FFHQ-Text dataset and the Face2Text dataset with the classic image generation method. The complete data was randomly divided into training and testing sets. The dataset parameters are shown in Table 1.

[0061] Table 1

[0062]

[0063] The FFHQ-Text dataset consists of 760 images, 500 of which are used for training, and each image is accompanied by 9 English text descriptions. The Face2Text dataset covers 400 images, 300 of which are used for training. Each image is accompanied by 1 to 4 English text descriptions, for a total of 1,237 text descriptions, and the length of each text description ranges from 2 to 94 words.

[0064] Inception Score (IS), Fréchet Inception Distance (FID) and R-precision (RP) are used as evaluation indicators for quantitative comparative analysis to evaluate the quality, authenticity and matching degree of the generated images with the input text. The results are shown in Table 2:

[0065] Table 2

[0066]

[0067] It can be concluded from Table 2 that on the FFHQ-Text dataset, compared with the baseline model DiT, the FID score of this method decreased by 3.02%, the IS score increased by 10.67%, and the RP increased from 40.2 to 45.35, an increase of 12.81%, and the fluctuation range of the RP value was the smallest, indicating that the stability of the generated image by this method is also the best. At the same time, compared with other comparison models, this method achieved the best results in all three evaluation indicators. In the Face2Text dataset, the IS score of this method reached 3.81 and the RP score reached 55.3. Compared with the LDM model, the IS score increased by 0.56 and the RP score increased by 47.46%. The above quantitative analysis shows that this method can significantly improve the semantic consistency between the input description text and the generated image, and the authenticity of the generated image is also improved.

Claims

1. A method for generating a face image based on a text description guided by multi-level features, characterized in that: The specific steps include: Step 1: Encode the text description using the text encoder of the pre-trained CLIP model to obtain a text vector; Step 2: Put the words related to face description into the face attribute word library, compare the text description content with the face attribute word library, encode the attribute words in the word library, and obtain the attribute word vector; Step 3: Perform random sampling in the Gaussian distribution, combine the text vector, map the sampling results into the image latent space, and obtain the latent space representation of the image Step 4: Align the dimensions of the text vector and the attribute word vector and add them together, input them into the feature extraction model for feature fusion, and obtain the vector fusionV; extract the age vector from the attribute word vector, combine it with the time step embedding, and obtain the vector extractedV; Step 5: Use the latent space representation of the image as the noise image input of the diffusion model, use the vectors fusionV and extractedV obtained in step 4 as the conditional input of the diffusion model, transform the output vector into the pixel space, and obtain the predicted generated image; Step 6: Encode and add noise to the facial image corresponding to the text description, compare it with the output of the diffusion model, and optimize the model parameters; input the text vector and the corresponding attribute word vector into the trained model to obtain the text description to guide the generation of facial images.

2. A method for generating a facial image based on a multi-level feature-based text description as claimed in claim 1, characterized in that: The face attribute word library includes 11 types of attribute words related to describing the age, gender, face orientation, eyes, nose, mouth, ears, skin, face shape, hair, and makeup of face images.

3. A method for generating a facial image based on a multi-level feature-based text description as claimed in claim 1, characterized in that: The method to obtain the latent space representation of the image is: s3.

1. Randomly sample the vector Z from the Gaussian distribution g , perform serialization conversion and add position information to obtain vector c0; s3.2, input the text vector into the fully connected layer to obtain vectors scale1, scale2, bias1 representing scale and bias information; s3.3, take vector c0 as input, perform layer normalization, multiply by scale1 and add bias to get feature vector c1; s3.4, pass the feature vector c1 into the multi-head self-attention module to obtain vector c2; s3.

5. Multiply vector c2 by scale2 and add c0 to obtain vector c3. s3.

6. Use the vector obtained in s3.5 as the input of s3.3 and repeat s3.3 to s3.5 five times; s3.

7. Deserialize the vector output by s3.6 to obtain the latent space representation of the image, that is, the predicted noise vector 4. The method for generating a facial image based on a multi-level feature-based text description as claimed in claim 1, characterized in that: The method for obtaining the vector fusionV is as follows: input the text vector and the attribute word vector into the fully connected network respectively, add them after aligning the dimensions, and then input them into the Unet network, perform residual connection on the output and input of the Unet network, and obtain the feature vector fusionV.

5. The method for generating a facial image based on a multi-level feature-based text description as claimed in claim 1, characterized in that: The age-related vector in the attribute word vector is input into the fully connected layer for fine-tuning, and then added to the time step embedding of the diffusion model. Get the feature vector extractedV.

6. The method for generating a facial image based on a multi-level feature-based text description as claimed in claim 1, characterized in that: The diffusion model includes multiple DiT blocks. In the first DiT block, the latent space of the image is represented As a noisy image input, output Each subsequent DiT block takes the output of the previous DiT block as input until The specific steps are as follows: s5.1 Input the feature vector extractedV into the fully connected layer to obtain vectors scale3, scale4, bias2 related to scale and bias; s5.2, yes After layer normalization, multiply by scale3 and add bias2, then input into the multi-head self-attention module, multiply by scale4 and add s5.3, normalize the result of s5.2, multiply it by scale3 and add bias2, then input it into the cross attention module together with the feature vector fusionV, multiply the output of the cross attention module by scale4, and then perform a residual connection with the result of s5.2; s5.4, after layer normalization, the result of s5.3 is multiplied by scale3 and then bias2. The result is input into the fully connected layer and then multiplied by scale4. Finally, a residual connection is performed with the result of s5.3 to obtain the predicted noise vector s5.5, through the VAE decoder Convert to pixel space and get the predicted generated image.

7. A method for generating a facial image based on a multi-level feature-based text description guide as claimed in claim 6, characterized in that: The method for encoding and denoising the face image corresponding to the text description is as follows: use the VAE encoder to encode the real image corresponding to the text description to obtain the image vector Z0; add a small amount of Gaussian noise ∈ to the image vector Z0 to obtain the noise vector Z1, repeat T times to obtain a pure noise image, and save the noise vector Z obtained T times t , t is the time step, t=1,2,…T.

8. A method for generating a facial image based on a multi-level feature-based text description guide as claimed in claim 7, characterized in that: The method to optimize the model parameters is to convert the predicted noise vector output by the DiT block into The true noise vector Z t And the current time step t is used as input to calculate the loss function loss and update the model parameters: loss=L MSE +L KL (1) Among them, L MSE represents the mean square error loss between the real noise and the predicted noise, L KL Represents the KL divergence between the true distribution and the predicted distribution: in, represents the model noise prediction, θ represents the model parameters, τ θ (y) is used to map the external condition to a vector of specified size, where y represents the external condition; E ∈(x),∈～N(0,1),t is the mean square error loss function; P(X) represents the real data distribution; X represents the real image; P(X θ (∈ θ ,σ θ )) represents the predicted data distribution, X θ (∈ θ ,σ θ ) represents the predicted generated image; ∈ θ With σ θ denote the noise and variance of the prediction, respectively. represents the KL divergence.

Images