Text-guided controllable portrait generation method and system based on diffusion model

Abstract

The invention discloses a text-guided controllable portrait generation method and system based on a diffusion model, and the method comprises the steps: firstly inputting a text prompt T into a CLIP model, and obtaining a text embedding vp; then based on the text embedding vp, a model G is generated by utilizing a text posture, and a posture guiding graph specified by a text guiding condition is extracted; then, refining initial estimation of face key points of the posture guide map by using a face fine tuning model M, and outputting a posture guide map P; and finally, giving an input picture x0, and generating an edited portrait by utilizing a posture portrait editing model based on the posture guide graph P. The experimental result of the model on a DF-PASS data set verifies the effectiveness of the model, the flexibility and diversity of the generated image can be effectively improved, the local blurring of the image is reduced, and the fidelity of the image is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and relates to a controllable portrait generation method and system, and specifically to a text-guided arbitrary posture portrait generation method and system based on a diffusion model. Background Art

[0002] In recent years, image generation technology has made significant progress. Text-guided portrait generation is the task of generating portraits corresponding to text descriptions by understanding the semantics in text descriptions. This task combines natural language processing and computer vision techniques, providing innovative possibilities in the fields of virtual character design, visual effects production, high-quality virtual dataset enhancement, and personalized user experience.

[0003] With the advancement of deep learning technology, especially the emergence of models such as Generative Adversarial Networks (GANs), the field of image generation has made significant breakthroughs. In recent years, emerging technologies such as diffusion models have emerged. They gradually transform noisy images into real images to generate high-quality images. They are relatively stable and have stronger training controllability. They have gradually replaced generative adversarial networks and become the leading models. However, conventional diffusion models still have some problems in image generation, such as the lack of authenticity and diversity of generated images, local detail distortion and lack of controllability. Recently, some work in the fashion field has explored the use of skeletons for controllable portrait generation and achieved certain results. However, skeletons can only provide limited posture guidance information, and extracting skeletons is relatively cumbersome. Therefore, how to improve the flexibility and diversity of the portrait generation process is also an issue that needs to be considered in current research.

[0004] Currently, diffusion models have attracted much attention in the field of text-guided portrait generation, providing a new way to achieve fine-grained portrait generation. The following are the basic training and testing steps for text-guided portrait generation using diffusion models:

[0005] Data preparation: Prepare training and testing datasets containing text descriptions and corresponding portrait images.

[0006] Text-gesture generation model construction: Encode the text, build a text-gesture generation model, train and save the optimal model.

[0007] Diffusion model construction: Construct a diffusion model, which usually consists of an encoder and a decoder. The encoder is used to convert the portrait image into a latent space representation, while the decoder associates the latent space representation with the text description and generates the edited portrait image.

[0008] Diffusion model training: Use the training dataset to train the diffusion model. During the training process, the model learns to associate text descriptions with portrait generation and optimizes the difference between the generated edits and the target edits. Save the best performing diffusion model for subsequent testing.

[0009] Testing phase: Use the saved optimal model to edit new portrait images and text descriptions. The model generates editing results based on the text description and converts it into a real portrait image.

[0010] In text-guided generation, the key to using diffusion models is to establish the connection between text descriptions and portraits, and to achieve high-quality image generation through the diffusion process. Diffusion models have good stability and training controllability, and have made significant progress in image generation quality and editing control. However, there are still some challenges in using diffusion models in text-guided portrait generation, such as local detail collapse. These issues require further research and improvement to improve the performance and effect of diffusion models in text-guided portrait generation. Therefore, how to improve the image quality of generated images and enhance the flexibility and diversity in the image generation process is a technical difficulty that needs to be solved urgently. Summary of the invention

[0011] In view of the problems existing in the prior art, the present invention proposes a method and system for text-guided controllable portrait generation based on a diffusion model based on the currently advanced diffusion model fine-tuning method, so as to conveniently realize local editing of images, improve the quality of image generation, and enhance the controllability of the image generation process.

[0012] The technical solution adopted by the method of the present invention is: a method for generating a controllable portrait guided by text based on a diffusion model, comprising the following steps:

[0013] Step 1: Input the text prompt T into the CLIP model to obtain the text embedding v p ;

[0014] Step 2: Embed v based on text p , using the text gesture generation model G, extract the gesture guidance graph specified by the text guidance condition;

[0015] Step 3: Use the facial fine-tuning model M to refine the initial estimate of the facial key points of the pose guidance map and output the pose guidance map P;

[0016] Step 4: Given an input image x0, based on the posture guidance map P, use the posture portrait editing model to generate an edited portrait.

[0017] Preferably, in step 2, the text gesture generation model G first uses a ReLU activated linear layer to transform v pProjected into a 1024-dimensional latent space φ B In the figure, a 1024-dimensional noise vector η~N(0,I) is then linearly cascaded, where I is a 1024*1024 unit matrix; then through 5 up-convolution blocks, in each up-convolution block, a transposed convolution is performed, followed by batch normalization and ReLU activation; five transposed convolutions produce the output of G by passing the output of the last up-convolution block; finally, the concatenated feature map passes through a point convolution layer with 256 filters (convolution kernels) and LeakyReLU activation, and the feature map is passed through another convolution layer using a single convolution kernel to estimate the final scalar output of G.

[0018] Preferably, in step 3, the facial fine-tuning model M uses a linear fully connected network N R Refine the initial estimate of facial key points through regression; first define five facial key points in, It's the position of the nose. The position of the right eye, The position of the left eye, The position of the right ear, The position of the left ear; then the translated facial key points are normalized so that the scaled key points Located in a square with a span of ±1, the scaled nose is located at the origin (0, 0); the coordinates of the five normalized key points are then flattened into a 10-dimensional vector and passed through three linear fully connected layers, each of which has 512 filters and ReLU activations; the final output layer of the network consists of 10 nodes and LeakyReLu.

[0019] Preferably, in step 4, the posture portrait editing model first performs texture encoding on the input image x0, then uses the posture guidance map P to perform posture guidance on the texture-encoded image, and then inputs several sequentially connected conditional denoising models ∈ θ , generate the edited portrait.

[0020] Preferably, the text gesture generation model G is a trained model; during training, the objective function of G is defined as:

[0021]

[0022] Among them, (x,y i )~p t is a pair of key point graphs and text embeddings sampled from the training set; η is the noise added to the training set, η~p η is a noise vector sampled from a Gaussian distribution. If there is a point with a coordinate <0 in the output posture guidance graph, the point is considered to be occluded;i represents the true value of the i-th sample, represents the predicted value of the i-th sample, and n represents the number of samples.

[0023] Preferably, the face fine-tuning model M is a trained model; during training, random 2D spatial perturbations are used to increase And predict The original value of ; optimize the parameters by minimizing the mean square error MSE between the actual coordinates and the predicted coordinates; then use N R The prediction results update the coordinates of the facial key points to obtain a refined set of key points.

[0024] Preferably, the pose portrait editing model is a trained model; during training, a noisy sample y is first generated by adding Gaussian noise to the input image y0. t ~q(y t |y0), and then train a conditional denoising model ∈ θ (y t ,t,x p ,x s );

[0025] Finally, the loss function is obtained:

[0026]

[0027] Among them, q(y t |y0) represents the image distribution after adding noise, t represents time, x p represents the posture guidance diagram, x s represents the target image, T represents the termination time, q(y0) represents the generated image distribution without noise, ∈ represents the noise term, ∈ θ represents the prediction of the noise term;

[0028] Then, inference is performed, during which a classifier-free bootstrapping technique is used, in which the noisy predictions at each step are weighted by a combination of unconditional and conditional predictions; As an unconditional embedding, the noisy prediction at each inference step is calculated as:

[0029]

[0030] Among them, τ θ ("") indicates no guide condition, z ′ t represents the image generated at time t, c represents the guidance condition; w represents the guidance ratio, and a higher guidance ratio encourages the generation of images that are closely related to the text prompt P.

[0031] The technical solution adopted by the system of the present invention is: a text-guided controllable portrait generation system based on a diffusion model, comprising:

[0032] one or more processors;

[0033] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the text-guided controllable portrait generation method based on the diffusion model.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] (1) Based on the existing posture-portrait generation model, the present invention proposes a plug-and-play text-guided controllable posture generation method, which improves the controllability of the text-guided portrait generation model. The relevant image indicators are better than the existing indicators, and the image generation effect meets the text requirements and is natural and coordinated.

[0036] (2) The text-guided posture generation module proposed in the present invention includes a facial fine-tuning module. This method effectively improves the accuracy of the facial control effect and makes the generated portrait expression more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic diagram of a method according to an embodiment of the present invention;

[0039] Figure 2 This is a network structure diagram of a text gesture generation module G according to an embodiment of the present invention;

[0040] Figure 3 This is a network structure diagram of a face fine-tuning module M according to an embodiment of the present invention;

[0041] Figure 4 A network structure diagram of a posture portrait editing model according to an embodiment of the present invention;

[0042] Figure 5 The generated effect diagrams corresponding to the input text examples 1 and 2 in the embodiments of the present invention;

[0043] Figure 6 These are the generation effect diagrams corresponding to the input text examples 3 and 4 in the embodiments of the present invention. DETAILED DESCRIPTION

[0044] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below with reference to examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0045] Please see Figure 1 The present embodiment provides a method for generating a controllable portrait guided by text based on a diffusion model, comprising the following steps:

[0046] Step 1: Input the text prompt T into the CLIP model to obtain the text embedding v p ;

[0047] Step 2: Embed v based on text p , using the text gesture generation model G, extract the gesture guidance graph specified by the text guidance condition;

[0048] In one embodiment, see Figure 2 The text gesture generation model G first uses a ReLU activated linear layer to transform v p Projected into a 1024-dimensional latent space φ B In the figure, a 1024-dimensional noise vector η~N(0,I) is then linearly cascaded, where I is a 1024*1024 unit matrix; then through 5 up-convolution blocks, in each up-convolution block, a transposed convolution is performed, followed by batch normalization and ReLU activation; the five transposed convolutions use 1024, 512, 512, 256, and 256 filters (convolution kernels) respectively. The five transposed convolutions produce the output of G by passing the output of the last up-convolution block; finally, the concatenated feature map is passed through a point convolution layer with 256 filters (convolution kernels) and LeakyReLU activation, and the feature map is passed through another convolution layer using a single convolution kernel to estimate the final scalar output of G.

[0049] In one implementation, the text gesture generation model G is a trained model; during training, the objective function of G is defined as:

[0050]

[0051] Among them, (x,y i )~p t is a pair of key point graphs and text embeddings sampled from the training set; η is the noise added to the training set, η~p η is a noise vector sampled from a Gaussian distribution. If there is a point with a coordinate <0 in the output posture guidance graph, the point is considered to be occluded; i represents the true value (target value) of the i-th sample, represents the predicted value (model output) of the i-th sample, and n represents the number of samples.

[0052] Step 3: Use the facial fine-tuning model M to refine the initial estimate of the facial key points of the pose guidance map and output the pose guidance map P;

[0053] In one embodiment, see Figure 3 , the facial fine-tuning model M, uses a linear fully connected network N R Refine the initial estimate of facial key points through regression; first define five facial key points in, It's the position of the nose. The position of the right eye, The position of the left eye, The position of the right ear, The position of the left ear; then the translated facial key points are normalized so that the scaled key points Located in a square with a span of ±1, the scaled nose is located at the origin (0, 0); the coordinates of the five normalized key points are then flattened into a 10-dimensional vector and passed through three linear fully connected layers, each of which has 512 filters and ReLU activations; the final output layer of the network consists of 10 nodes and LeakyReLu.

[0054] In one embodiment, the face fine-tuning model M is a trained model; during training, random 2D spatial perturbations are used to increase And predict The original value of ; optimize the parameters by minimizing the mean square error MSE between the actual coordinates and the predicted coordinates; then use N R The prediction results update the coordinates of the facial key points to obtain a refined set of key points.

[0055] This embodiment uses MSELoss loss to train the text-pose generation model G and the face fine-tuning module M on the DF-PASS dataset.

[0056] Step 4: Given an input image x0, based on the posture guidance graph P, the posture portrait editing model is used to generate an edited portrait;

[0057] In one embodiment, see Figure 4 , the pose portrait editing model first performs texture encoding on the input image x0, then uses the pose guidance map P to perform pose guidance on the texture-encoded image, and then inputs several sequentially connected conditional denoising models ∈ θ , generate the edited portrait.

[0058] In one embodiment, the pose portrait editing model is a trained model;

[0059] The conditional denoising model is based on the diffusion model. The overall idea is to design a diffusion process to gradually add noise to the data sampled from the target distribution y0~q(y0), while the backward denoising process attempts to learn the reverse mapping. The denoising diffusion process eventually converts the isotropic Gaussian noise In T steps, it is converted to the target data distribution. During training, a noisy sample y is first generated by adding Gaussian noise to the input image y0 t ~q(y t |y0), and then train a conditional denoising model ∈ θ (y t ,t,x p ,x s );

[0060] Finally, the loss function is obtained:

[0061]

[0062] Among them, q(y t |y0) represents the image distribution after adding noise, t represents time, x p represents the posture guidance diagram, x s represents the target image, T represents the termination time, q(y0) represents the generated image distribution without noise, ∈ represents the noise term, ∈ θ represents the prediction of the noise term;

[0063] Then, inference is performed, during which a classifier-free bootstrapping technique is used, in which the noisy predictions at each step are weighted by a combination of unconditional and conditional predictions; As an unconditional embedding, the noisy prediction at each inference step is calculated as:

[0064]

[0065] Among them, τ θ ("") indicates no guide condition, z ′ t represents the image generated at time t, c represents the guidance condition; w represents the guidance ratio, and a higher guidance ratio encourages the generation of images that are closely related to the text prompt P.

[0066] In order to keep the identity of the person in the image and the rest of the image except the edited area unchanged, the decoder D generates the edited image x after the inference process. e Combined with the original image x0.

[0067] This embodiment also provides a text-guided controllable portrait generation system based on a diffusion model, including:

[0068] one or more processors;

[0069] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the text-guided controllable portrait generation method based on the diffusion model.

[0070] The present invention is further described below through specific experiments.

[0071] This experiment is conducted on the DF-PASS dataset, using 37,344 samples for training and 3,144 samples out of 40,488 annotated samples as a validation set for testing.

[0072] In the experiment, all images are downsampled to 256×256 resolution. This experiment trains the text-pose generation model G, the face fine-tuning model M and the image generation main network on DF-PASS for 100 epochs with a batch size of 64. The Adam optimizer is used and the learning rate is set to 1×10 -4 In training, we use the AdamW optimizer to fine-tune it on the DF-PASS dataset with 140k steps and set the learning rate to 1×10 -5 The experiment compares the performance of the portrait editing main network and the portrait editing model with the text-pose generation model added on the three test indicators FID, LPIPS, and SSIM.

[0073] In the experiment, the training images are input into the model for training; first, the CLIP text encoder is trained to encode the text description into a vector representation. Secondly, the editing area positioning module is trained to accurately generate the corresponding posture guidance map based on the text description. Finally, the diffusion model is trained to learn how to generate high-quality images based on conditional inputs. Throughout the training process, parameter tuning, model evaluation, and verification are required to ensure that the method can achieve good results in posture editing tasks. This experiment fine-tuned the model using the AdamW optimizer [Loshchilov, I.; and Hutter, F. 2018. Decoupled Weight Decay Regularization. In ICLR.] on the DF-PASS dataset, with a fine-tuning step size of 140k and a learning rate set to 1×10 -5. In order to save memory, this experiment adopts the mixed precision (Micikevicius et al., 2018) [Micikevicius, P.; Narang, S.; Alben, J.; Diamos, G.; Elsen, E.; Garcia, D.; Ginsburg, B.; Houston, M.; Kuchaiev, O.; Venkatesh, G.; et al. 2018. Mixed Precision Training. In ICLR] and gradient accumulation strategy, where the step size of gradient accumulation is set to 4 and the batch size is set to 1. In the inference stage, this experiment uses the PNDM scheduler with 50 iteration steps [Liu, L.; Ren, Y.; Lin, Z.; and Zhao, Z. 2021. Pseudo Numerical Methods for Diffusion Models on Manifolds. In ICLR], and sets the classifier-free bootstrap ω to 7.5.

[0074] Then, this experiment performs network optimization and parameter update; the update includes two parts: forward propagation and back propagation. Forward propagation calculates the output of the network and the loss function L mse The size of the network is updated by back-propagating the gradient of the loss through the optimization strategy of stochastic gradient descent.

[0075] Finally, this experiment tests the image generation network based on the diffusion model; in the test phase, the network is not trained and the parameters are not updated, and the trained model is used to generate images. This experiment uses Fr′echet InceptionDistance (FID), Learned Perceptual Image Patch Similarity (LPIPS) and SSIM to quantitatively evaluate the sample fidelity of the generated fashion images.

[0076] The test results are shown in Table 1. The experiment shows that after adding the text-pose generation model, the model generation effect does not decrease and performs better in FID and LPIPS.

[0077] Table 1

[0078] Test indicators FID(↓) LPIPS(↓) PIDM(pose_guided) 20.547 0.4920 Ours(text_guided) 20.524 0.4749

[0079] Please see Figure 5 , which is the generated effect diagram corresponding to input text 1 and 2 in this experiment, please see Figure 5 , which is the generated effect diagram corresponding to input text 3 and 4 in this experiment.

[0080] Input text example 1: A woman is standing with her body facing towards front. Her head is facing front and she is keeping her face straight. Both of her hands are straight. She is keeping her left wrist near left hip and her right wrist near right hip. Both of her legs are straight.

[0081] Input text example 2: A woman is standing with her body facing towards front. Her head is facing front and she is keeping her face tilted right. Both of her hands are straight. She is keeping her left wrist near left hip and her right wrist near right hip. Both of her legs are occluded.

[0082] Input text example 3: A woman is standing with her body facing towards front. Her head is facing front and she is keeping her face straight. Her left hand is folded but her right hand is straight. She is keeping her left wrist near right elbow and her right wrist near right hip. Both of her legs are occluded.

[0083] Input text example 4: A woman is standing with her body facing towardsfront.Her head is facing front and she is keeping her face straight.Both of her hands are straight.She is keeping her left wrist near left hip and herright wrist near right hip.Her left leg is folded but her right leg isstraight.

[0084] The example results show that the text posture generation module proposed in this method has excellent performance in understanding text and generating corresponding postures, and the effect is natural (all facial information mentioned in this patent is specially processed, and the model can generate naturally restored faces).

[0085] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0086] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A text-guided controllable portrait generation method based on a diffusion model, characterized in that: The following steps are involved: Step 1: Input the text prompt T into the CLIP model to obtain the text embedding v p ; Step 2: Embed v based on text p , using the text gesture generation model G, extract the gesture guidance graph specified by the text guidance condition; Step 3: Use the facial fine-tuning model M to refine the initial estimate of the facial key points of the pose guidance map and output the pose guidance map P; Step 4; Given an input image x0, based on the pose guidance graph P, the pose portrait editing model is used to generate the edited portrait.

2. The method for generating a controllable portrait guided by text based on a diffusion model according to claim 1, characterized in that: In step 2, the text gesture generation model G first uses a ReLU activated linear layer to transform v p Projected into a 1024-dimensional latent space φ B In the figure, a 1024-dimensional noise vector η~N(0,I) is then linearly cascaded, where I is a 1024*1024 unit matrix; then through 5 up-convolution blocks, in each up-convolution block, a transposed convolution is performed, followed by batch normalization and ReLU activation; five transposed convolutions produce the output of G by passing the output of the last up-convolution block; finally, the concatenated feature map passes through a point convolution layer with 256 convolution kernels and LeakyReLU activation, and the feature map is passed through another convolution layer using a single convolution kernel to estimate the final scalar output of G.

3. The method for generating a controllable portrait guided by text based on a diffusion model according to claim 1, characterized in that: In step 3, the facial fine-tuning model M uses a linear fully connected network N R Refine the initial estimate of facial key points through regression; first define five facial key points in, It's the position of the nose. The position of the right eye, The position of the left eye, The position of the right ear, The position of the left ear; then the translated facial key points are normalized so that the scaled key points Located in a square with a span of ±1, the scaled nose is located at the origin (0, 0); the coordinates of the five normalized key points are then flattened into a 10-dimensional vector and passed through three linear fully connected layers, each of which has 512 convolution kernels and ReLU activation; the final output layer of the network consists of 10 nodes and LeakyReLu.

4. The method for generating a controllable portrait guided by text based on a diffusion model according to claim 1, characterized in that: In step 4, the pose portrait editing model first performs texture encoding on the input image x0, then uses the pose guidance map P to perform pose guidance on the texture-encoded image, and then inputs several sequentially connected conditional denoising models ∈ θ , generate the edited portrait.

5. The method for generating a controllable portrait guided by text based on a diffusion model according to any one of claims 1 to 4, characterized in that: The text gesture generation model G is a trained model; during training, the objective function of G is defined as: Among them, (x,y i )~p t is a pair of key point graphs and text embeddings sampled from the training set; η is the noise added to the training set, η~p η is a noise vector sampled from a Gaussian distribution. If there is a point with a coordinate <0 in the output posture guidance graph, the point is considered to be occluded; i represents the true value of the i-th sample, represents the predicted value of the i-th sample, and n represents the number of samples.

6. The method for generating a controllable portrait guided by text based on a diffusion model according to any one of claims 1 to 4, characterized in that: The face fine-tuning model M is a trained model; during training, random 2D spatial perturbations are used to increase And predict The original value of ; optimize the parameters by minimizing the mean square error MSE between the actual coordinates and the predicted coordinates; then use N R The prediction results update the coordinates of the facial key points to obtain a refined set of key points.

7. The method for generating a controllable portrait guided by text based on a diffusion model according to any one of claims 1 to 4, characterized in that: The pose portrait editing model is a trained model. During training, a noisy sample y is first generated by adding Gaussian noise to the input image y0. t ~q(y t |y0), and then train a conditional denoising model ∈ θ (y t ,t,x p ,x s ); Finally, the loss function is obtained: Among them, q(y t |y0) represents the image distribution after adding noise, t represents time, x p represents the posture guidance diagram, x s represents the target image, T represents the termination time, q(y0) represents the generated image distribution without noise, ∈ represents the noise term, ∈ θ represents the prediction of the noise term; Then, inference is performed, during which a classifier-free bootstrapping technique is used, in which the noisy predictions at each step are weighted by a combination of unconditional and conditional predictions; As an unconditional embedding, the noisy prediction at each inference step is calculated as: Among them, τ θ ("") indicates no guide condition, z ′ t represents the image generated at time t, c represents the guidance condition; w represents the guidance ratio, and a higher guidance ratio encourages the generation of images that are closely related to the text prompt P.

8. A text-guided controllable portrait generation system based on a diffusion model, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method for generating a controllable portrait guided by text based on a diffusion model as described in any one of claims 1 to 7.

Images