A text-guided controllable portrait generation method and system based on diffusion model

Abstract

The present invention discloses a method and system for generating a text-guided controllable portrait based on a diffusion model. First, a text prompt T is input into a CLIP model to obtain a text embedding v. p ; Then embed v based on the text p , using the text pose generation model G to extract the pose guidance map specified by the text guidance conditions; then, using the facial fine-tuning model M, the initial estimate of facial key points in the pose guidance map is refined to output the pose guidance map P; finally, given an input image x0, the pose portrait editing model is used to generate an edited portrait based on the pose guidance map P. Experimental results on the DF-PASS dataset validate the effectiveness of the proposed model, demonstrating that the proposed method can effectively improve the flexibility and diversity of generated images, reduce local image blur, and enhance image fidelity.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and relates to a controllable portrait generation method and system, and specifically to a text-guided arbitrary posture portrait generation method and system based on a diffusion model. Background Art

[0002] Image generation technology has made significant progress in recent years. Text-guided portrait generation involves generating a portrait that corresponds to a text description by understanding its semantic meaning. This task combines natural language processing and computer vision techniques, offering innovative possibilities for virtual character design, visual effects production, high-quality virtual dataset enhancement, and personalized user experiences.

[0003] With advances in deep learning technology, particularly the emergence of models like generative adversarial networks (GANs), significant breakthroughs have been achieved in the field of image generation. In recent years, emerging techniques such as diffusion models have emerged. These models gradually transform noisy images into realistic ones, generating high-quality images. These models are relatively stable and have greater controllability in training, gradually replacing GANs as the leading model. However, conventional diffusion models still face challenges in image generation, such as insufficient realism and diversity, distortion of local details, and insufficient controllability. Recent work in the fashion field has explored the use of skeleton graphs for controllable portrait generation, achieving some success. However, skeleton graphs only provide limited pose guidance information, and extracting them is relatively cumbersome. Therefore, improving the flexibility and diversity of the portrait generation process is a pressing research issue.

[0004] Currently, diffusion models have attracted much attention in the field of text-guided portrait generation. They provide a new approach to achieve fine-grained portrait generation. The following are the basic training and testing steps for text-guided portrait generation using diffusion models:

[0005] Data preparation: Prepare training and test datasets containing text descriptions and corresponding portrait images.

[0006] Text-gesture generation model construction: Encode the text, build a text-gesture generation model, train and save the optimal model.

[0007] Diffusion model construction: Construct a diffusion model, which usually consists of an encoder and a decoder. The encoder is used to convert the portrait image into a latent space representation, while the decoder associates the latent space representation with the text description and generates the edited portrait image.

[0008] Diffusion model training: Train the diffusion model using the training dataset. During training, the model learns to associate text descriptions with generated portraits and optimizes the difference between the generated edits and the target edits. Save the best-performing diffusion model for subsequent testing.

[0009] Testing phase: Use the saved optimal model to edit new portrait images and text descriptions. The model generates edited results based on the text descriptions and converts them into real portrait images.

[0010] The key to using diffusion models in text-guided generation is to establish a connection between text descriptions and portraits, and to achieve high-quality image generation through the diffusion process. Diffusion models offer excellent stability and training controllability, and have achieved significant progress in image generation quality and editing control. However, the use of diffusion models in text-guided portrait generation currently faces several challenges, such as the loss of local details. These issues require further research and improvement to enhance the performance and effectiveness of diffusion models in text-guided portrait generation. Therefore, improving the quality of generated images and enhancing the flexibility and diversity of the image generation process remain pressing technical challenges. Summary of the Invention

[0011] In response to the problems existing in the existing technology, the present invention proposes a method and system for text-guided controllable portrait generation based on a diffusion model based on the currently advanced diffusion model fine-tuning method, so as to conveniently realize local editing of images, improve the quality of image generation, and enhance the controllability of the image generation process.

[0012] The technical solution adopted by the method of the present invention is: a method for generating a text-guided controllable portrait based on a diffusion model, comprising the following steps:

[0013] Step 1: Input the text prompt T into the CLIP model to obtain the text embedding v p ;

[0014] Step 2: Embed v based on text p ,Using the text posture generation model G, the posture guidance graph specified by the text guidance condition is extracted;

[0015] Step 3: Use the facial fine-tuning model M to refine the initial estimate of facial key points of the pose guidance map and output the pose guidance map P;

[0016] Step 4: Given an input image x0, based on the posture guidance map P, use the posture portrait editing model to generate the edited portrait.

[0017] Preferably, in step 2, the text gesture generation model G first uses a ReLU activated linear layer to transform v pProjected into a 1024-dimensional latent space φ B In the figure, a 1024-dimensional noise vector η~N(0,I) is then linearly concatenated, where I is a 1024*1024 identity matrix; then through 5 up-convolution blocks, in each up-convolution block, a transposed convolution is performed, followed by batch normalization and ReLU activation; five transposed convolutions produce the output of G by passing the output of the last up-convolution block; finally, the concatenated feature map passes through a point convolution layer with 256 filters (convolution kernels) and LeakyReLU activation, and the feature map is passed through another convolution layer using a single convolution kernel to estimate the final scalar output of G.

[0018] As a preference, in step 3, the facial fine-tuning model M uses a linear fully connected network N R Refine the initial estimate of facial key points through regression; first define five facial key points in, It's the position of the nose. The position of the right eye, The position of the left eye, The position of the right ear, The position of the left ear; then the translated facial key points are normalized so that the scaled key points Located in a square with a span of ±1, the scaled nose is located at the origin (0, 0); the coordinates of the five normalized key points are then flattened into a 10-dimensional vector and passed through three linear fully connected layers, each of which has 512 filters and ReLU activations; the final output layer of the network consists of 10 nodes and LeakyReLu.

[0019] As a preference, in step 4, the posture portrait editing model first performs texture encoding on the input image x0, then uses the posture guidance map P to perform posture guidance on the texture-encoded image, and then inputs several sequentially connected conditional denoising models ∈ θ , generate the edited portrait.

[0020] Preferably, the text gesture generation model G is a trained model; during training, the objective function of G is defined as:

[0021]

[0022] Among them, (x,y i )~p t is a pair of keypoint graphs and text embeddings sampled from the training set; η is the noise added to the training set, η~p η is a noise vector sampled from a Gaussian distribution. If there is a point with a coordinate <0 in the output posture guidance map, the point is considered to be occluded.i represents the true value of the i-th sample, represents the predicted value of the i-th sample, and n represents the number of samples.

[0023] As a preference, the face fine-tuning model M is a trained model; during training, random 2D spatial perturbations are used to increase and predict The original value of ; optimize the parameters by minimizing the mean square error MSE between the actual coordinates and the predicted coordinates; then use N R The prediction results update the coordinates of the facial key points to obtain a refined set of key points.

[0024] As an example, the pose portrait editing model is a trained model; during training, a noisy sample y is first generated by adding Gaussian noise to the input image y0. t ~q(y t |y0), and then train a conditional denoising model ∈ θ (y t ,t,x p ,x s );

[0025] Finally, the loss function is obtained:

[0026]

[0027] Among them, q(y t |y0) represents the image distribution after adding noise, t represents time, x p represents the posture guidance diagram, x s represents the target image, T represents the termination time, q(y0) represents the generated image distribution without noise, ∈ represents the noise term, ∈ θ represents the prediction of the noise term;

[0028] Then, inference is performed, during which a classifier-free bootstrapping technique is used, where the noisy predictions at each step are weighted by a combination of unconditional and conditional predictions; As an unconditional embedding, the noisy prediction at each inference step is calculated as follows:

[0029]

[0030] Among them, τ θ ("") indicates no guide condition, z ′ t represents the image generated at time t, c represents the guidance condition; w represents the guidance ratio, and a higher guidance ratio encourages the generation of images that are closely related to the text prompt P.

[0031] The technical solution adopted by the system of the present invention is: a text-guided controllable portrait generation system based on a diffusion model, comprising:

[0032] one or more processors;

[0033] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the text-guided controllable portrait generation method based on the diffusion model.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) Based on the existing posture-portrait generation model, the present invention proposes a plug-and-play text-guided controllable posture generation method, which improves the controllability of the text-guided portrait generation model. The relevant image indicators are better than the existing indicators, and the image generation effect meets the text requirements and is natural and coordinated.

[0036] (2) The text-guided posture generation module proposed in the present invention includes a facial fine-tuning module. This method effectively improves the accuracy of the facial control effect and makes the generated portrait expression more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The technical solution of the present invention is further illustrated below using embodiments and specific implementation methods. In addition, some drawings are also used in the process of illustrating the technical solution. For those skilled in the art, other drawings and the intention of the present invention can be obtained based on these drawings without making any creative efforts.

[0038] Figure 1 A schematic diagram of a method according to an embodiment of the present invention;

[0039] Figure 2 This is a network structure diagram of a text gesture generation module G according to an embodiment of the present invention;

[0040] Figure 3 This is a network structure diagram of the face fine-tuning module M according to an embodiment of the present invention;

[0041] Figure 4 This is a network structure diagram of a posture portrait editing model according to an embodiment of the present invention;

[0042] Figure 5 The generated effect diagrams corresponding to the input text examples 1 and 2 in the embodiments of the present invention;

[0043] Figure 6 These are the generated effect diagrams corresponding to text input examples 3 and 4 in the embodiments of the present invention. DETAILED DESCRIPTION

[0044] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below with reference to examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0045] Please see Figure 1 This embodiment provides a method for generating a controllable portrait guided by text based on a diffusion model, comprising the following steps:

[0046] Step 1: Input the text prompt T into the CLIP model to obtain the text embedding v p ;

[0047] Step 2: Embed v based on text p ,Using the text posture generation model G, the posture guidance graph specified by the text guidance condition is extracted;

[0048] In one embodiment, see Figure 2 , the text gesture generation model G first uses a ReLU activated linear layer to transform v p Projected into a 1024-dimensional latent space φ B In the figure, a 1024-dimensional noise vector η~N(0,I) is linearly concatenated, where I is a 1024*1024 identity matrix. The network then passes through five upconvolution blocks, in each of which a transposed convolution is performed, followed by batch normalization and ReLU activation. The five transposed convolutions use 1024, 512, 512, 256, and 256 filters (convolution kernels), respectively. The five transposed convolutions produce the output of G by passing the output of the last upconvolution block. Finally, the concatenated feature map is passed through a point-wise convolution layer with 256 filters (convolution kernels) and LeakyReLU activation, and the feature map is passed through another convolution layer using a single convolution kernel to estimate the final scalar output of G.

[0049] In one embodiment, the text gesture generation model G is a trained model; during training, the objective function of G is defined as:

[0050]

[0051] Among them, (x,y i )~p t is a pair of keypoint graphs and text embeddings sampled from the training set; η is the noise added to the training set, η~p η is a noise vector sampled from a Gaussian distribution. If there is a point with a coordinate <0 in the output posture guidance map, the point is considered to be occluded. i represents the true value (target value) of the i-th sample, represents the predicted value of the i-th sample (model output), and n represents the number of samples.

[0052] Step 3: Use the facial fine-tuning model M to refine the initial estimate of facial key points of the pose guidance map and output the pose guidance map P;

[0053] In one embodiment, see Figure 3 , the facial fine-tuning model M, uses a linear fully connected network N R Refine the initial estimate of facial key points through regression; first define five facial key points in, It's the position of the nose. The position of the right eye, The position of the left eye, The position of the right ear, The position of the left ear; then the translated facial key points are normalized so that the scaled key points Located in a square with a span of ±1, the scaled nose is located at the origin (0, 0); the coordinates of the five normalized key points are then flattened into a 10-dimensional vector and passed through three linear fully connected layers, each of which has 512 filters and ReLU activations; the final output layer of the network consists of 10 nodes and LeakyReLu.

[0054] In one embodiment, the face fine-tuning model M is a trained model; during training, random 2D spatial perturbations are used to increase and predict The original value of ; optimize the parameters by minimizing the mean square error MSE between the actual coordinates and the predicted coordinates; then use N R The prediction results update the coordinates of the facial key points to obtain a refined set of key points.

[0055] This embodiment uses MSELoss loss to train the text-to-pose generation model G and the face fine-tuning module M on the DF-PASS dataset.

[0056] Step 4: Given an input image x0, generate an edited portrait based on the pose guidance graph P using the pose portrait editing model;

[0057] In one embodiment, see Figure 4 The pose portrait editing model first performs texture encoding on the input image x0, then uses the pose guidance map P to perform pose guidance on the texture-encoded image, and then inputs several sequentially connected conditional denoising models ∈ θ , generate the edited portrait.

[0058] In one embodiment, the pose portrait editing model is a trained model;

[0059] The conditional denoising model is based on the diffusion model. The general idea is to design a diffusion process to gradually add noise to the data sampled from the target distribution y0~q(y0), while the backward denoising process attempts to learn the reverse mapping. The denoising diffusion process eventually converts the isotropic Gaussian noise into The data is converted to the target distribution in T steps. During training, a noisy sample y is first generated by adding Gaussian noise to the input image y0 t ~q(y t |y0), and then train a conditional denoising model ∈ θ (y t ,t,x p ,x s );

[0060] Finally, the loss function is obtained:

[0061]

[0062] Among them, q(y t |y0) represents the image distribution after adding noise, t represents time, x p represents the posture guidance diagram, x s represents the target image, T represents the termination time, q(y0) represents the generated image distribution without noise, ∈ represents the noise term, ∈ θ represents the prediction of the noise term;

[0063] Then, inference is performed, during which a classifier-free bootstrapping technique is used, where the noisy predictions at each step are weighted by a combination of unconditional and conditional predictions; As an unconditional embedding, the noisy prediction at each inference step is calculated as follows:

[0064]

[0065] Among them, τ θ ("") indicates no guide condition, z ′ t represents the image generated at time t, c represents the guidance condition; w represents the guidance ratio, and a higher guidance ratio encourages the generation of images that are closely related to the text prompt P.

[0066] In order to keep the identity of the person in the image and the rest of the image except the edited area unchanged, the decoder D generates the edited image x after the inference process. e Combined with the original image x0.

[0067] This embodiment also provides a text-guided controllable portrait generation system based on a diffusion model, including:

[0068] one or more processors;

[0069] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the text-guided controllable portrait generation method based on the diffusion model.

[0070] The present invention is further described below through specific experiments.

[0071] This experiment is conducted on the DF-PASS dataset, using 37,344 samples for training and 3,144 samples out of 40,488 annotated samples as a validation set for testing.

[0072] In the experiment, all images are downsampled to 256×256 resolution. This experiment trains the text-pose generation model G, the face fine-tuning model M, and the image generation main network on DF-PASS for 100 epochs with a batch size of 64. The Adam optimizer is used and the learning rate is set to 1×10 -4 In training, we use the AdamW optimizer to fine-tune it on the DF-PASS dataset for 140k steps and set the learning rate to 1×10 -5 The experiment compares the performance of the portrait editing main network and the portrait editing model with the text-pose generation model added on three test indicators: FID, LPIPS, and SSIM.

[0073] In the experiment, the training images are input into the model for training; first, the CLIP text encoder is trained to encode the text description into a vector representation. Secondly, the editing area positioning module is trained to accurately generate the corresponding posture guidance map based on the text description. Finally, the diffusion model is trained to learn how to generate high-quality images based on conditional inputs. Throughout the training process, parameter tuning, model evaluation, and verification are required to ensure that the method can achieve good results in posture editing tasks. This experiment fine-tuned the model using the AdamW optimizer [Loshchilov, I.; and Hutter, F. 2018. Decoupled Weight Decay Regularization. In ICLR.] on the DF-PASS dataset with a fine-tuning step size of 140k and a learning rate of 1×10 -5To save memory, this experiment adopts the mixed precision (Micikevicius et al., 2018) [Micikevicius, P.; Narang, S.; Alben, J.; Diamos, G.; Elsen, E.; Garcia, D.; Ginsburg, B.; Houston, M.; Kuchaiev, O.; Venkatesh, G.; et al. 2018. Mixed Precision Training. In ICLR] and gradient accumulation strategy, where the step size of gradient accumulation is set to 4 and the batch size is set to 1. In the inference phase, this experiment uses the PNDM scheduler with 50 iteration steps [Liu, L.; Ren, Y.; Lin, Z.; and Zhao, Z. 2021. Pseudo Numerical Methods for Diffusion Models on Manifolds. In ICLR] and sets the classifier-free bootstrap ω to 7.5.

[0074] Then, this experiment performs network optimization and parameter update; the update includes two parts: forward propagation and back propagation. Forward propagation calculates the output of the network and the loss function L mse The size of the network is then updated by backpropagating the gradient of the loss through the stochastic gradient descent optimization strategy.

[0075] Finally, this experiment tests the image generation network based on the diffusion model. During the test phase, the network is not trained or parameter updated, and the trained model is used to generate images. This experiment uses Fr′echet Inception Distance (FID), Learned Perceptual Image Patch Similarity (LPIPS), and SSIM to quantitatively evaluate the sample fidelity of the generated fashion images.

[0076] The test results are shown in Table 1. The experiment shows that after adding the text-pose generation model, the model generation effect does not decrease and performs better in FID and LPIPS.

[0077] Table 1

[0078] Test indicators FID(↓) LPIPS(↓) PIDM(pose_guided) 20.547 0.4920 Ours(text_guided) 20.524 0.4749

[0079] Please see Figure 5 , which is the generated effect diagram corresponding to input text 1 and 2 in this experiment, please see Figure 5 , which is the generated effect diagram corresponding to input text 3 and 4 in this experiment.

[0080] Input text example 1: A woman is standing with her body facing towards front. Her head is facing front and she is keeping her face straight. Both of her hands are straight. She is keeping her left wrist near left hip and her right wrist near right hip. Both of her legs are straight.

[0081] Input text example 2: A woman is standing with her body facing towards front. Her head is facing front and she is keeping her face tilted right. Both of her hands are straight. She is keeping her left wrist near left hip and her right wrist near right hip. Both of her legs are occluded.

[0082] Input text example 3: A woman is standing with her body facing towards front. Her head is facing front and she is keeping her face straight. Her left hand is folded but her right hand is straight. She is keeping her left wrist near right elbow and her right wrist near right hip. Both of her legs are occluded.

[0083] Input text example 4: A woman is standing with her body facing towardsfront.Her head is facing front and she is keeping her face straight.Both of her hands are straight.She is keeping her left wrist near left hip and herright wrist near right hip.Her left leg is folded but her right leg isstraight.

[0084] The example results show that the text posture generation module proposed in this method has excellent performance in understanding text and generating corresponding postures, and the effect is natural (all facial information mentioned in this patent is specially processed, and the model can generate naturally restored faces).

[0085] It should be understood that the embodiments described above are only some of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention may be arbitrarily combined with each other to form a feasible technical solution. Such combination is not restricted by the order of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually inconsistent or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0086] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A text-guided controllable portrait generation method based on a diffusion model, characterized in that: The following steps are involved: Step 1: Input the text prompt T into the CLIP model to obtain the text embedding ; Step 2: Text Embedding ,Using the text posture generation model G, the posture guidance graph specified by the text guidance condition is extracted; The text gesture generation model G first uses a ReLU activated linear layer to transform Projected into a 1024-dimensional latent space Then linearly concatenate a 1024-dimensional noise vector , where I is a 1024*1024 identity matrix; then it passes through 5 up-convolution blocks, in each up-convolution block, a transposed convolution is performed, followed by batch normalization and ReLU activation; five transposed convolutions produce the output of G by passing the output of the last up-convolution block; finally, the concatenated feature map passes through a point convolution layer with 256 convolution kernels and LeakyReLU activation, and the feature map is passed through another convolution layer using a single convolution kernel to estimate the final scalar output of G; Step 3: Use the facial fine-tuning model M to refine the initial estimate of facial key points of the pose guidance map and output the pose guidance map P; Step 4; Given an input image ,Based on the posture guidance graph P, the posture portrait editing model is used to generate the edited portrait.

2. The method for generating a controllable human portrait guided by text based on a diffusion model according to claim 1, characterized in that: In step 3, the facial fine-tuning model M is constructed using a linear fully connected network Refine the initial estimate of facial key points through regression; first define five facial key points ,in, It's the position of the nose. The position of the right eye, The position of the left eye, The position of the right ear, The position of the left ear; then the translated facial key points are normalized so that the scaled key points The scaled nose is located in a square with a span of ±1, and is located at the origin (0, 0); the coordinates of the five normalized key points are then flattened into a 10-dimensional vector and passed through three linear fully connected layers, each of which has 512 convolution kernels and ReLU activation; the final output layer of the network consists of 10 nodes and LeakyReLu.

3. The method for generating a controllable portrait guided by text based on a diffusion model according to claim 1, characterized in that: In step 4, the pose portrait editing model first processes the input image Perform texture encoding, then use the posture guidance map P to guide the posture of the texture-encoded image, and then input several sequentially connected conditional denoising models , generate the edited portrait.

4. The method for generating a controllable human portrait guided by text based on a diffusion model according to any one of claims 1 to 3, characterized in that: The text gesture generation model G is a trained model; during training, the objective function of G is defined as: in, are keypoint image and text embedding pairs sampled from the training set; is the noise added to the training set, is a noise vector sampled from a Gaussian distribution. If there is a point with a coordinate < 0 in the output posture guidance map, the point is considered to be occluded. represents the true value of the i-th sample, represents the predicted value of the i-th sample, n Indicates the sample size.

5. The method for generating a controllable human portrait guided by text based on a diffusion model according to any one of claims 1 to 3, characterized in that: The face fine-tuning model M is a trained model; during training, random 2D spatial perturbations are used to increase , and predict The original value of ; optimize the parameters by minimizing the mean square error MSE between the actual coordinates and the predicted coordinates; then use the non-regular processing and re-translation of the predicted facial key points The prediction results update the coordinates of the facial key points to obtain a refined set of key points.

6. The method for generating a controllable human portrait guided by text based on a diffusion model according to any one of claims 1 to 3, characterized in that: The pose portrait editing model is a trained model. During training, the first step is to input the image Add Gaussian noise to generate a noisy sample , and then train a conditional denoising model ; Finally, the loss function is obtained: ； in, represents the image distribution after adding noise, t represents time, Indicates the posture guidance diagram, represents the target image, T represents the termination time, represents the generated image distribution without noise, represents the noise term, represents the prediction of the noise term; Then, inference is performed, during which a classifier-free bootstrapping technique is used, where the noisy predictions at each step are weighted by a combination of unconditional and conditional predictions; As an unconditional embedding, the noisy prediction at each inference step is calculated as follows: in, Indicates no boot condition. represents the image generated at time t, and c represents the guiding condition; represents the guidance ratio, and a higher guidance ratio encourages the generation of images that are closely related to the text prompt P.

7. A text-guided controllable portrait generation system based on a diffusion model, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the text-guided controllable portrait generation method based on a diffusion model as described in any one of claims 1 to 6.

Images