Face Generation Method Based on Keypoint-Guided Graph Diffusion Model

Abstract

The present invention discloses a face generation method based on a key point guided text graph diffusion model. First, data information is collected, and training data pairs are formed from face images, text description information and face key points. The text description information is encoded and converted into an embedding vector using a text encoder, and the face key point image information is mapped to an implicit space using a latent space encoder. The implicit space tensor of a Gaussian noise map and the implicit space tensor of the face key point image information are spliced. An implicit diffusion model is established and trained, and the spliced ​​implicit space tensor is continuously iteratively denoised to obtain face images corresponding to the key points and text description information. A denoising loss module guided by a heat map is used to allocate additional training attention to the key point conditions. The denoising results in the implicit space are mapped to the pixel space using a latent space decoder, and finally a face image with high generation quality and high consistency between the image and the text description information is reconstructed.

Description

Technical Field

[0001] The present invention belongs to the technical field of facial image generation, and in particular relates to a facial generation method based on a key point guided Vincent graph diffusion model. Background Art

[0002] In recent years, with the rapid development of multimodal and artificial intelligence technologies, text-to-image generation has become an emerging research direction in the multimedia field. This field requires generating images that are as realistic as possible from text descriptions while ensuring semantic consistency. With the rise of diffusion models, image generation technology has made significant progress. This allows ordinary users, even those without professional painting knowledge or systematic training, to produce diverse, clear, and realistic images using simple conditions. Facial image generation, a subtask of general image generation, involves generating corresponding facial images from text descriptions. This technology can be used for artistic generation, portrait simulation, generating suspect portraits from eyewitness testimony and confessions, and comparing faces when image information is unavailable. It can also provide additional clues and guidance in criminal investigations and criminal investigations, supplement missing image information, expand the scope of investigations, and provide detailed and accurate facial features, helping police and investigators better understand suspects' appearances.

[0003] At present, due to the more subtle and complex features of facial images and the lack of effective algorithms and large-scale datasets, models usually find it difficult to generate high-quality and qualified faces based on limited conditions. Therefore, the face generation task is more challenging and is an important but rarely solved problem. How to improve the authenticity and controllability of facial image generation has gradually attracted people's attention.

[0004] Among existing methods, those based on generative adversarial networks can generate realistic images. However, due to their difficulty in training and poor multimodal fusion and alignment capabilities, they have yet to gain mainstream acceptance. In recent years, the rise of diffusion models has provided a new approach to image generation, leading to unprecedented improvements in text-to-image generation. However, textual information primarily provides features such as hair and beards, which are subjective, uncertain, and subject to data scarcity. Providing precise positional control is difficult, resulting in insufficient detail and low accuracy in generated images. To further improve the quality of generated faces, pixel-level prior knowledge (such as facial feature segmentation maps) is often required to provide greater control, but such complex conditions are difficult to obtain. To strike a balance between minimizing the need for prior knowledge and providing sufficient facial spatial conditions to control face generation, some work has attempted to guide text-to-face generation through keypoints. We believe that using keypoints (point-level prior knowledge) as conditions is more user-friendly and more responsive to the specific characteristics of the face domain, where hair varies significantly while facial features vary less.

[0005] Existing keypoint-guided text-to-face generation models introduce an additional learnable branch scheme on top of a frozen Stable Diffusion (SD) model to achieve control over the generation process, which greatly improves the controllability of the SD model. However, the essence of this scheme is image editing and is subject to certain limitations. The face image generation task requires the model to generate a face consistent with the given conditions. However, given the text conditions, the SD branch may produce a face image that is inconsistent with the keypoints. For example, the face may appear at different angles and with different facial shapes in different locations in the image. Therefore, the trainable branch must not only learn how to generate a face based on the given keypoints, but also learn how to suppress the various inconsistencies generated by the SD branch, making model training extremely challenging. Generally speaking, the greater the gap between the face generated by the SD and the keypoint guidance, the greater the gap between the generated image and the expected image. Summary of the Invention

[0006] The present invention is aimed at the problems existing in the prior art and provides a face generation method based on a key point guided text graph diffusion model. First, data information is collected, and the corresponding face images, text description information and face key points are constructed into training data pairs. Then, the text description information is encoded and converted into an embedding vector using a text encoder, and the face key point image information is mapped to the implicit space using a latent space encoder. Then, a Gaussian noise map is initialized and mapped to the implicit space through a latent space encoder. The implicit space tensor of the Gaussian noise map and the implicit space tensor of the face key point image information are spliced. An implicit diffusion model is established and the diffusion model is trained to iteratively denoise the spliced ​​implicit space tensor to obtain face images corresponding to the key points and text description information. A denoising loss module guided by a heat map is used to allocate additional training attention to the key point conditions to achieve high generation quality and high image-condition consistency at the same time. A latent space decoder is used to map the denoising result in the implicit space to the pixel space, and finally a face image with high generation quality and high image-condition consistency is reconstructed. The method of the present invention uses a diffusion model to generate a face image by guiding text description information through key points, and ensures that the face image is consistent with the provided face key points and text description information.

[0007] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a face generation method based on a key point guided Wensheng graph diffusion model, comprising at least the following steps:

[0008] S1. Data collection and processing: Collect text description information and facial key points of different facial images, integrate the corresponding facial images, text description information and facial key points, and construct training data pairs;

[0009] S2. Text description information processing: Use a text encoder to encode the text description information corresponding to the face image and convert it into an embedding vector, which is used as the text condition input in the generation process;

[0010] S3. Face key point image processing: Use the latent space encoder to map the face key point image information to the implicit space, obtain the implicit space tensor of the face key point image information, and enable the diffusion model to use the face key points in the implicit space for calculation;

[0011] S4, implicit space tensor splicing: Initialize the Gaussian noise map, map it to the implicit space to obtain the implicit space tensor of the Gaussian noise map, splice the implicit space tensor of the Gaussian noise map with the implicit space tensor of the facial key point image information obtained in step S3, and feed it into the diffusion model;

[0012] S5. Establish an implicit diffusion model: The implicit diffusion model uses the pre-trained Stable Diffusion 2.0, which is a deep neural network with a UNet structure. Through a multi-step inverse diffusion process, samples that conform to the distribution of training data are obtained;

[0013] S6. Model training: The implicit diffusion model established in step S5 is trained, and the spliced ​​implicit space tensor is iteratively denoised. The predicted noise and the real noise are subtracted, and the difference is decoded by the latent space decoder to obtain a difference map. Key point detection is performed on the difference map to obtain an attention heat map. The attention is allocated to the key point conditions to achieve high generation quality and high consistency between the image and the conditions.

[0014] S7. Result output: Initialize the Gaussian noise map, use the denoising network trained in step S6, and gradually predict the noise added in the current step based on the facial key points and text description information. Use the implicit sampling algorithm to iteratively denoise the noise map, and use the latent space decoder to map the implicit space denoising result to the pixel space, and finally reconstruct the face image consistent with the facial key points and text description information.

[0015] As an improvement of the present invention, different facial images are collected in step S1, and the collected facial images are cropped, aligned, and super-resolution operations are performed on them. Corresponding text descriptions and facial key point information are collected based on the facial images, and the facial images are used as corresponding label images.

[0016] As an improvement of the present invention, the text encoder in step S2 is a CLIP model, and the text conditions are converted into high-dimensional embedding vectors after encoding, where the descriptive text information corresponding to the face includes gender, age, eyebrow shape, hairstyle, etc. The high-dimensional embedding vectors are used for text condition input in the generation process.

[0017] As another improvement of the present invention, the latent space encoder ε in step S3 adopts a pre-trained VAE encoder model to transform the facial key point image c composed of the facial key point coordinates into L Mapped to the implicit space, and the implicit space tensor z is formed L =ε(c L ); The implicit space tensor is a compact representation of the original key point image to reduce the computational complexity of the subsequent diffusion model; the corresponding VAE decoder serves as the latent space decoder The latent space decoder is used to reconstruct the latent space tensor into a pixel space image, namely:

[0018]

[0019] As another improvement of the present invention, in step S4, the implicit space tensor of the Gaussian noise map is obtained by the same implicit space encoder as step S3, and the implicit space tensor of the Gaussian noise map is spliced ​​with the implicit space tensor of the facial key point image information and fed into the diffusion model; the splicing is specifically as follows:

[0020] z=concat(z noise ,z L ; dim = 1)

[0021] Among them, z noise =ε(x noise ) is the implicit spatial tensor of the Gaussian noise map, x noise ~N(0,1) is the initial Gaussian noise map, and ε is the latent space encoder.

[0022] As another improvement to the present invention, the implicit diffusion model in step S5 includes an encoder module, an intermediate layer module, and a decoder module, each of which is composed of a residual block, a spatial attention module, an upsampling module, a downsampling module, and a skip connection. The residual block receives the time step encoding and the output of the previous upsampling module, and the spatial attention module receives the text encoding and the output of the previous residual block. Conditioned by the text encoding, the module learns the matching relationship between text information and facial images through cross-attention. The diffusion model is initialized using the Stable Diffusion 2.0 network parameters, and the encoder input form is modified to accommodate the concatenated latent space tensor.

[0023] As a further improvement of the present invention, the loss function of the implicit diffusion model in step S5 is:

[0024]

[0025] Where t is the time step, z t The tensor z0 is obtained after t steps of noise addition. ∈ is the real noise, ∈ θ is the noise predicted by the model, and c is the text condition c T and key point condition c L A collection of .

[0026] As a further improvement of the present invention, in step S6, additional training attention is allocated to the key point conditions of the difference image through a denoising loss module guided by a heat map. The denoising loss module is a face key point detector pre-trained by STAR Loss, which generates a corresponding attention heat map according to the difference image and allocates additional training attention to the key point conditions.

[0027] Compared with the prior art, the present invention has the following technical advantages and effects:

[0028] (1) In step S5 of the present invention, an innovative diffusion model is proposed, which guides the face generation method of the text image through key points. The powerful generation ability of the implicit diffusion model is used to finely control facial features, ensuring that the generated face image is consistent with the facial key points and text conditions.

[0029] (2) The method of the present invention proposes a denoising loss module guided by a heat map, which, through differentiated learning, strengthens the control of facial features while avoiding excessive damage to the generalization performance of the model.

[0030] (3) The present invention eliminates the conflicts inherent in the dual-branch architecture when processing various conditions, and can accurately control the generation of facial features, such as the shape and position of the facial features. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is an example diagram of the face image-text description information-face key point dataset in step S1 of the method of the present invention;

[0032] Figure 2 This is a schematic diagram of the face generation method based on key point guided Wensheng map of the present invention, wherein:

[0033] (a) is the basic structure diagram of the implicit diffusion model;

[0034] (b) is the structure diagram of the denoising loss module guided by the heat map;

[0035] Figure 3 Schematic diagram of the difference between the predicted facial noise and the actual noise in step S6 of embodiment 2 of the present invention;

[0036] Figure 4 This is a comparison chart of face generation results using different combinations of key points and text conditions using the method of the present invention;

[0037] Figure 5 is a comparison chart of face generation results of different models in the test example of the present invention, where:

[0038] (a) is a schematic diagram of the results of the method of the present invention;

[0039] (b) is a schematic diagram of the ControlNet results;

[0040] (c) is a schematic diagram of the results of T2I-Adapter;

[0041] (d) is a schematic diagram of the results of Stable Diffusion 1.5;

[0042] (e) is a schematic diagram of the results of Stable Diffusion 2.1;

[0043] Figure 6 This is a comparison of facial images generated under relatively uncommon text conditions and facial key point conditions selected by different models in the test example of the method of the present invention, where:

[0044] (a) is a schematic diagram of the results of the method of the present invention;

[0045] (b) is a schematic diagram of the ControlNet results;

[0046] (c) is a schematic diagram of the results of T2I-Adapter;

[0047] Figure 7 : is a schematic diagram of the visualization results of the comparison of the face generation and facial key point conditions of different models in the test example of the present invention, wherein:

[0048] (a) is a schematic diagram of the results of the method of the present invention;

[0049] (b) is a schematic diagram of the ControlNet results;

[0050] (c) is a schematic diagram of the results of T2I-Adapter;

[0051] Figure 8 It is a flowchart of the steps of the face generation method based on key point guided Wensheng map of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0053] Example 1

[0054] The face generation method based on the key point guided Wensheng graph diffusion model specifically includes the following steps:

[0055] Step S1: Collect text description information and facial key points of different facial images, integrate the corresponding facial images, text description information and facial key points, and construct training data pairs;

[0056] Collect different facial images, perform cropping, alignment, and super-resolution operations on the collected facial images, and collect corresponding text descriptions and facial key point information based on the facial images, and use the facial images as corresponding label images.

[0057] Step S2: Use a text encoder to encode the description information corresponding to the face and convert it into an embedding vector, which is used as the text condition input in the generation process;

[0058] The CLIP model is selected as the text encoder to conditionally encode the text and convert it into a high-dimensional embedding vector. The descriptive text information corresponding to the face includes gender, age, eyebrow shape, hairstyle, etc. The high-dimensional embedding vector is used as the text condition input in the generation process.

[0059] Step S3: Mapping the facial key point image information to the implicit space using a latent space encoder, so that the diffusion model can perform calculations using the facial key points in the implicit space;

[0060] The latent space encoder uses a pre-trained VAE encoder model to map the facial keypoint image composed of facial keypoint coordinates into the implicit space, forming an implicit space tensor. The diffusion model uses the facial keypoint information in the implicit space for calculation. This implicit space tensor is a compact representation of the original keypoint image, reducing the computational complexity of the subsequent diffusion model.

[0061] Step S4: Initialize the Gaussian noise map, map it to the implicit space through the latent space encoder, concatenate the implicit space tensor of the Gaussian noise map and the implicit space tensor of the facial key point image information in step S3, and feed them into the diffusion model;

[0062] In this step, the same VAE encoder model as in step S3 is used to map the Gaussian noise map to the implicit space and form a latent space tensor.

[0063] Step S5: Establish an implicit diffusion model. The diffusion model uses the pre-trained Stable Diffusion 2.0, a deep neural network with a UNet structure. It can be guided by text conditions and gradually "denoise" to generate new data. Given random noise, the model transforms it into clear samples that conform to the training data distribution through a multi-step inverse diffusion process. Compared with other generative models (such as GANs and VAEs), the implicit diffusion model can more effectively handle complex and high-dimensional data distributions, and has greater advantages in image generation quality and diversity. The diffusion model consists of an encoder, intermediate layers, and a decoder. These modules are composed of residual blocks, spatial attention modules, upsampling modules, downsampling modules, and skip connections. The diffusion model is initialized using the Stable Diffusion 2.0 network parameters, and the encoder input form is modified to adapt to the spliced ​​latent space tensor.

[0064] The residual block in the diffusion model receives the time step encoding and the output of the previous module, and the spatial attention module receives the text encoding and the output of the previous module. It learns the matching relationship between text information and face images through cross attention based on the text encoding.

[0065] Step S6: Globally train the constructed diffusion model, iteratively denoise the spliced ​​implicit spatial tensor, and subtract the predicted noise from the actual noise. The difference is decoded by the VAE decoder to obtain a blurry difference map, which is then fed into the denoising loss module guided by the heat map. This module is a face key point detector pre-trained with STAR Loss. The detector generates a corresponding attention heat map based on the difference map and allocates additional training attention to the key point conditions to achieve both high generation quality and high image-condition consistency.

[0066] Step S7: Use the latent space decoder to map the denoising result in the latent space to the pixel space, and finally reconstruct a face image with high generation quality and high image-condition consistency.

[0067] Example 2

[0068] In this embodiment, the implicit diffusion model takes as input a textual description of a face and its key points. The textual description consists of 40 facial features, including arched eyebrow, attractive, and bald, constructed using a simple syntax. The key point map contains 98 coordinate points representing facial features, including eyebrows, eyes, nose, mouth, and facial contours. While the textual description and key points provide some information about a face, they cannot fully represent all facial features.

[0069] This embodiment is similar to Figure 1The image distribution shown in is trained using 27,000 pairs of face images, text description information, and key points of face information, and tested using 3,000 pairs of face images, text description information, and key points of face information. Figure 2 As shown in the figure, the face generation is assisted by the text diffusion model in a key point guided manner. The facial key point map and Gaussian noise map are first mapped to the implicit space and spliced, and then sent to the diffusion model. The text encoding information is injected into each module of the diffusion model. The powerful generation ability of the implicit diffusion model is used to generate facial features, and finally a high-fidelity face image consistent with the facial key points and text conditions is reconstructed.

[0070] Face generation method based on key point guided graph diffusion model, such as Figure 8 As shown, the specific steps include:

[0071] Step S1: Collect 30,000 different facial images, and perform operations such as cropping, alignment, and super-resolution on the images to unify the facial images into 512×512 images. According to the characteristics of the facial images, 40 labels are marked for each image, including Arched Eyebrows, Attractive, Bald, etc., and then the facial feature description labels are connected into text description sentences through simple syntax. The key points of the facial images are marked, and the facial feature information located by 98 coordinate points includes eyebrows, eyes, nose, mouth, and facial contour areas. In this way, a facial image-text description information-facial key point information data pair is constructed, such as Figure 1 As shown in Figure 2, the first 27,000 data pairs are used as training sets, and the last 3,000 data pairs are used as test sets.

[0072] Step S2: Select the CLIP model as the text encoder, input the text description sentence into the CLIP model, and obtain a high-dimensional vector converted from the text description. The high-dimensional vector is a 77-word embedding vector, and the feature dimension of each vector is 768. These high-dimensional embedding vectors are subsequently fed into the diffusion model as the text condition input c in the generation process. T .

[0073] Step S3: The latent space encoder ε uses the pre-trained VAE encoder model to transform the facial key point image c composed of 98 facial key point coordinates L Mapped to the implicit space, and the implicit space tensor z is formed L =ε(c L ), the diffusion model uses facial keypoint information for calculation in the implicit space. This implicit space tensor is a compact representation of the original keypoint image, which reduces the computational complexity of the subsequent diffusion model.

[0074] The corresponding VAE decoder serves as the latent space decoder The latent space decoder is used to reconstruct the latent space tensor into a pixel space image, i.e.

[0075] Step S4: Initialize Gaussian noise map x noise ~N(0,1), the same VAE encoder model as in step S3 is used as the latent space encoder, and it is mapped to the implicit space through the latent space encoder ε to obtain the implicit space tensor z of the Gaussian noise map noise =ε(x noise ), concatenate the implicit space tensor of the Gaussian noise map and the implicit space tensor of the facial key point image information in step S3 z = concat (z noise , z L ,; dim=1) and feed it into the diffusion model.

[0076] Step S5: Establish an implicit diffusion model; the diffusion model uses the pre-trained Stable Diffusion 2.0. This diffusion model includes an encoder, intermediate layers, and a decoder. These modules consist of residual blocks, spatial attention modules, upsampling modules, downsampling modules, and skip connections. Initialize the diffusion model using the Stable Diffusion 2.0 network parameters, and modify the encoder input format to accommodate the concatenated latent space tensor. The diffusion model is trained using the L2 distance as the loss function:

[0077]

[0078] Where t is the time step, z t The tensor z0 is obtained after t steps of noise addition. ∈ is the real noise, ∈ θ is the noise predicted by the model, and c is the text condition c T and key point condition c L A collection of .

[0079] The residual block in the diffusion model receives the time step encoding and the output of the previous module, and the spatial attention module receives the text encoding and the output of the previous module. It learns the matching relationship between text information and face images through cross attention based on the text encoding.

[0080] Step S6: Figure 2 As shown in (a), the training diffusion model continuously iteratively denoises the spliced ​​implicit spatial tensor to obtain the face image corresponding to the key points and text. The denoising loss module guided by the heat map is used to allocate additional training attention to the key point conditions to achieve high generation quality and high consistency between the image and the conditions. The specific process is as follows:

[0081] The constructed diffusion model is trained globally, and the prediction noise ∈ θ , the real noise ∈ and the predicted noise ∈ θ The difference is passed through the decoder After decoding, a relatively fuzzy difference image is obtained, such as Figure 3 shown

[0082]

[0083] Then the difference map M is sent to the denoising loss module guided by the heat map, as Figure 2 As shown in (b), this module is a face key point detector pre-trained by STAR Loss, which generates the corresponding attention heat map based on the difference map

[0084] H=F(M)

[0085] Here, H∈h×w×k, h and w represent the height and width of the attention heatmap, k is the number of key points, and F is the attention heatmap estimator in the denoising loss module. To simplify the calculation, the attention heatmaps are summed according to the dimension k to generate a single attention heatmap. Figure 3 As shown, the difference between the predicted noise and the real noise ∈-∈ θ The larger , the more obvious the face image in the difference map M is, and the corresponding attention heat map H value is also larger. The attention heat map H is sent to the latent space encoder ε to obtain the heat map encoding:

[0086]

[0087] Finally, the objective function of the optimized diffusion model can be expressed as:

[0088]

[0089]

[0090] Here W a =ω·H E , ω is set to 0.1 by default.

[0091] Step S6: During inference, use the trained denoising network to extract the noise image z t Estimate the noise-free face image z0 in [1]. Load the trained diffusion model parameters and perform iterative denoising in the latent space. Calculate the noise added in the previous step based on the current step noise. Continuously iterate to finally obtain the clean image z0. The sampling formula is as follows:

[0092]

[0093] Among them, z t is noise sampled from a standard normal distribution, is a time-varying hyperparameter, t is the time step, ∈ θ Represents the diffusion model, σ is the variance, and the default value is 0. The sampling formula is used for 50 iterations to obtain z0, and finally z0 is mapped to the pixel space through the latent space decoder to obtain a high-fidelity face image consistent with the facial key points and text conditions.

[0094] Figure 4 This is a comparison chart of face results generated by the method of the present invention after selecting different key points and text condition combinations. Figure 4 As can be seen in the figure, 7 groups of key points and 5 groups of text conditions are combined to form 35 key point and text condition pairs. Finally, 35 face images are obtained through the method of the present invention. Figure 4 As can be seen from the case of , the present invention can generate high-fidelity face images for any combination of key points and text conditions, even for relatively uncommon feature combinations, such as a young man with long hair and a beard (i.e. Figure 4 , line 6), the present invention still successfully generates high-quality facial images.

[0095] Test Case

[0096] The method of the present invention is compared with the results of different face generation models. Figure 5 As shown, Figure 5 The comparison of face generation results of different models is shown. Figure 5 The text above each row of images and the key point image in the first column of the row constitute a pair of text and face key point conditions. The condition pair is fed into the present invention, the dual-branch conditional generation model: ControINet and T2I-Adapter, and the Wensheng graph diffusion model: Stable Diffusion1.5 and Stable Diffusion2.1, and then generates 5 corresponding faces (i.e., the ae column in the figure). The second column in the figure is the real face corresponding to the text and face key point conditions. Figure 5 As shown in Figure 2, under the same conditions, compared with faces generated by other models, the present invention can generate high-fidelity face images that are more consistent with facial key points and text conditions, and are very similar to real images, such as Figure 5In the fourth row, compared to other models, the faces generated by our method are consistent with real-world facial expressions, facial orientation, and eye movement. Although ControlNet and T2I-Aapter can also generate faces based on text and facial key points, they perform less effectively in capturing facial features and hair details. For example, in the fourth row of column (b), the faces generated by ControlNet have different orientations than real-world faces; in the fifth row of column (b), the faces generated by T2I-Aapter do not have the wide-mouthed expression similar to real-world faces. Furthermore, the results of StableDiffusion 1.5 and StableDiffusion 2.1 struggle to maintain consistency with the facial key point conditions, as shown in the fourth rows of columns (c) and (d).

[0097] The face images generated by the method of the present invention and other models are compared under the condition of selecting relatively uncommon texts. Figure 6 shown. Figure 6 The text condition "a young man with long hair and a beard" is combined with 7 sets of facial key points to form 7 pairs of key points and text conditions. The facial images are then generated using the method of the present invention, ControlNet, and T2I-Adapter models respectively. The results generated by the method of the present invention are as follows: Figure 6 As shown in (a), Figure 6 As can be seen in (a), under the conditions of 7 sets of facial key points, the method of the present invention generates facial images consistent with the text conditions. The generation results of ControlNet are as follows Figure 6 As shown in (b), the face images generated by the model have gender inconsistencies, such as Figure 6 (b) The fourth and seventh faces. The results generated by T2I-Adapter are as follows Figure 6 As shown in (c), the face images generated by this model also have gender inconsistency, such as Figure 6 (c) In the fourth picture, there is still inconsistency between the face and hair features, such as Figure 6 (c) The seventh face is ranked. Therefore, under the condition of selecting relatively uncommon texts, the method of the present invention has higher accuracy.

[0098] Figure 7 This is a diagram showing the visualization of the face generated by the method of the present invention and other models compared to the key point conditions. Three key point and text condition pairs were fed into the face images generated by the method of the present invention, ControlNet, and T2I-Adapter models, respectively. Figure 7 The green dots represent the key points of the face used to guide the face generation, while the red dots represent the key points of the face corresponding to the generated face. Figure 7As shown in (a), the face generated by the present invention has a high consistency with the key point conditions of the face. The key points corresponding to the generated face (i.e., the red dots in the figure) almost coincide with the key point conditions of the face (i.e., the green dots in the figure) or are on the extension line of the green dots. The ControlNet generated result is as follows Figure 7 (b) Compared with the present invention, the generation effect of ControlNet is poor, and the position of the red dot is larger than that of the green dot, as shown in Figure 7 In the second row of column (b), the generated result of ControlNet is quite different from the original image. This image not only fails to meet the key point condition of the character opening his mouth wide and closing his left eye, but also fails to meet the requirement of the face turning slightly to the left. Figure 7 (c) shows the generated results of this model. The model is slightly better than ControlNet, but still not as good as the present invention. Figure 7 In the second row of column (c), although the image meets the requirement that the face is slightly turned to the left, it still fails to meet the key point condition that the character has an open mouth and closes the left eye.

[0099] In summary, the method of the present invention trains a diffusion model to iteratively denoise the spliced ​​implicit space tensor, obtains the facial image corresponding to the key points and text, and uses a denoising loss module guided by a heat map to allocate additional training attention to the key point conditions to achieve high generation quality and high image-condition consistency. The denoising results in the implicit space are mapped to the pixel space using a latent space decoder, and finally reconstructs a facial image with high generation quality and high image-condition consistency. The present invention can generate facial images by guiding the text map diffusion model through key points, and generate high-fidelity facial images that are highly consistent with the key points and text descriptions based on prior information.

[0100] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A face generation method based on a key point guided Wensheng graph diffusion model, characterized by , including at least the following steps: S1. Collect and process data: collect image information of different faces, wherein the image information includes at least text description information and facial key points, and construct training data pairs of face image-text description information-face key points from the corresponding face images, text description information and facial key points; S2. Text description information processing: Use a text encoder to encode the text description information corresponding to the face image and convert it into an embedding vector; S3. Face key point image processing: Use the latent space encoder to map the face key point image information to the latent space to obtain the latent space tensor of the face key point image information; S4, implicit space tensor splicing: Initialize the Gaussian noise map, map it to the implicit space to obtain the implicit space tensor of the Gaussian noise map, and splice the implicit space tensor of the Gaussian noise map with the implicit space tensor of the facial key point image information obtained in step S3; S5. Establish an implicit diffusion model: The implicit diffusion model uses the pre-trained Stable Diffusion 2.0, which is a deep neural network with a UNet structure. Through a multi-step inverse diffusion process, samples that conform to the distribution of the training data are obtained; S6. Model training: The implicit diffusion model established in step S5 is trained, and the spliced ​​implicit space tensor is iteratively denoised. The predicted noise and the real noise are subtracted, and the difference is decoded by the latent space decoder to obtain a difference map. The difference map is subjected to key point detection to obtain an attention heat map. The attention is allocated to the key point conditions. S7. Result output: Initialize the Gaussian noise map, use the denoising network trained in step S6, and gradually predict the noise added in the current step based on the facial key points and text description information. Use the implicit sampling algorithm to iteratively denoise the noise map, and use the latent space decoder to map the implicit space denoising result to the pixel space, and finally reconstruct the face image consistent with the facial key points and text description information.

2. The face generation method based on the key point guided Vincent graph diffusion model according to claim 1, characterized in that: In step S1, at least cropping, alignment and super-resolution operations are performed on the face image, and the face image is used as the corresponding label image according to the text description information and facial key points corresponding to the face image.

3. The face generation method based on the key point guided Vincent graph diffusion model according to claim 1, characterized in that: The text encoder in step S2 is a CLIP model, and the text conditions are converted into high-dimensional embedding vectors after encoding.

4. The face generation method based on the key point guided Vincent graph diffusion model according to claim 1, characterized in that: In step S3, the latent space encoder ε adopts a pre-trained VAE encoder model to transform the facial key point image c composed of the facial key point coordinates into L Mapped to the implicit space, and the implicit space tensor z is formed L =ε(c L ); the corresponding VAE decoder serves as the latent space decoder The latent space decoder is used to reconstruct the latent space tensor into a pixel space image, namely:

5. The face generation method based on the key point guided Vincent graph diffusion model according to claim 4, characterized in that: In step S4, the implicit space tensor of the Gaussian noise image is obtained by the same implicit space encoder as in step S3, and the implicit space tensor of the Gaussian noise image is concatenated with the implicit space tensor of the facial key point image information as follows: z=concat(z noise ,from L ;dim=1) Among them, z noise =ε(x noise ) is the implicit spatial tensor of the Gaussian noise map, x noise ~N(0,1) is the initial Gaussian noise map, and ε is the latent space encoder.

6. The face generation method based on the key point guided Vincent graph diffusion model according to claim 1, characterized in that: The implicit diffusion model in step S5 includes an encoder module, an intermediate layer module and a decoder module, each module consisting of a residual block, a spatial attention module, an upsampling module, a downsampling module and a jump connection; wherein the residual block receives the time step encoding and the output of the previous layer upsampling module, the spatial attention module receives the text encoding and the output of the previous layer residual block, and learns the matching relationship between text information and face images through cross attention based on the text encoding.

7. The face generation method based on the key point guided Vincent graph diffusion model according to claim 6, characterized in that: The loss function of the implicit diffusion model in step S5 is: Where t is the time step, z t The tensor z0 is obtained after t steps of noise addition. ∈ is the real noise, ∈ θ is the noise predicted by the model, and c is the text condition c T and key point condition c L A collection of .

8. The face generation method based on the key point guided Vincent graph diffusion model according to claim 1, characterized in that: In step S6, the denoising loss module guided by the heat map is used to allocate additional training attention to the key point conditions of the difference map. The denoising loss module is a face key point detector pre-trained by STAR Loss. It generates a corresponding attention heat map according to the difference map and allocates additional training attention to the key point conditions.

Images