Face generation method based on key point guided text-based graph diffusion model

Abstract

The invention discloses a face generation method based on a key point guide text graph diffusion model, which comprises the following steps of: firstly, collecting data information, forming a training data pair by a face image, text description information and face key points, encoding the text description information by using a text encoder and converting the text description information into an embedded vector, mapping the face key point image information to an implicit space by using an implicit space encoder; splicing the implicit spatial tensor of the Gaussian noise map and the implicit spatial tensor of the face key point image information; establishing and training an implicit diffusion model, continuously iteratively denoising the spliced implicit space tensor to obtain a face image corresponding to the key point and the text description information, and distributing additional training attention for the key point condition by using a denoising loss module under the guidance of a thermodynamic diagram, and a denoising result in the implicit space is mapped to a pixel space by using an implicit space decoder, and finally a face image with high generation quality and high consistency of image and text description information is reconstructed.

Description

Technical Field

[0001] The present invention belongs to the technical field of face image generation, and in particular relates to a face generation method based on a key point guided Vincent graph diffusion model. Background Art

[0002] In recent years, with the rapid development of multimodal technology and artificial intelligence technology, text-image generation has become an emerging research direction in the multimedia field. This direction requires generating images that are as realistic as possible based on text descriptions and ensuring semantic consistency. With the rise of diffusion models, image generation technology has made great progress. This allows ordinary users who lack professional painting knowledge and have not undergone systematic learning to generate diverse, clear, and realistic pictures through simple conditions. Face image generation is a subtask of general image generation, which refers to generating corresponding face images through text descriptions. This technology can be used for art generation, portrait simulation, generating suspect portraits from eyewitness testimony and confessions, and face comparison when there is no image information. It also provides additional clues and guidance in crime investigations and criminal investigations, supplements the lack of image information, expands the scope of investigation, and provides detailed and accurate facial features to help police and investigators better understand the appearance of suspects.

[0003] At present, due to the finer and more complex features of facial images and the lack of effective algorithms and large-scale data sets, it is usually difficult for models to generate high-quality and qualified faces based on limited conditions. Therefore, the face generation task is more challenging. It is an important problem with few solutions. How to improve the authenticity and controllability of face image generation has gradually attracted people's attention.

[0004] Among the existing methods, methods based on generative adversarial networks can generate realistic images, but they have not yet been accepted by the mainstream due to the difficulty of training and poor multimodal fusion and alignment capabilities. In recent years, the rise of diffusion models has provided a new solution for image generation, which has led to unprecedented improvements in text image generation performance. However, text information mainly provides features such as hair and beard, and has the characteristics of subjectivity, uncertainty, and insufficient data. It is difficult to provide precise position control, which makes the generated images often lack details and have low accuracy. In order to further improve the quality of generated faces, it is often necessary to use pixel-level prior knowledge (such as: facial features segmentation map) to provide higher control, but such complex conditions are difficult to obtain. In order to achieve a balance between minimizing the objective demand for prior knowledge and sufficient facial spatial conditions for controlling face generation, some works have tried to guide text-face generation through key points. We believe that using key points (point-level prior knowledge) as conditions is more user-friendly and more in line with the special characteristics of the face field, where hair changes greatly and facial features change little.

[0005] The existing keypoint-guided text-face generation model adopts an additional learnable branch scheme on the frozen Stable Diffusion (SD) model to achieve control over the generation process, which greatly improves the controllability of the SD model. However, the essence of this scheme is image editing and is subject to some limitations. The face image generation task requires the model to generate faces consistent with the given conditions, but according to the given text conditions, the SD branch may produce face images inconsistent with the key points, for example, the face may appear in different positions in the picture with different angles and different face shapes. Therefore, the trainable branch not only needs to learn how to generate faces based on the given key points, but also needs to learn how to suppress various inconsistencies generated by the SD branch, which makes the training of the model extremely challenging. Generally speaking, the greater the gap between the face generated by SD and the keypoint guidance, the greater the gap between the generated image and the expected image. Summary of the invention

[0006] The present invention is aimed at the problems existing in the prior art and provides a face generation method based on a key point guided text graph diffusion model. First, data information is collected, and the corresponding face images, text description information and face key points are constructed into training data pairs. Then, the text description information is encoded and converted into an embedding vector by a text encoder, and the face key point image information is mapped to the implicit space by a latent space encoder. Then, a Gaussian noise map is initialized and mapped to the implicit space by a latent space encoder, and the implicit space tensor of the Gaussian noise map and the implicit space tensor of the face key point image information are spliced. An implicit diffusion model is established, and the diffusion model is trained to iteratively denoise the spliced ​​implicit space tensor to obtain face images corresponding to the key points and text description information. A denoising loss module guided by a heat map is used to allocate additional training attention to the key point conditions to achieve high generation quality and high image-condition consistency at the same time. A latent space decoder is used to map the denoising result in the implicit space to the pixel space, and finally a face image with high generation quality and high image-condition consistency is reconstructed. The method of the present invention uses a diffusion model to generate a face image by guiding text description information through key points, and ensures that the face image is consistent with the provided face key points and text description information.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a face generation method based on a key point guided Vincent graph diffusion model at least comprises the following steps:

[0008] S1. Collect and process data: collect text description information and facial key points of different facial images, integrate the corresponding facial images, text description information and facial key points, and construct training data pairs;

[0009] S2, text description information processing: use the text encoder to encode the text description information corresponding to the face image and convert it into an embedding vector, which is used as the text condition input in the generation process;

[0010] S3, facial key point image processing: using the latent space encoder to map the facial key point image information to the implicit space, obtaining the implicit space tensor of the facial key point image information, so that the diffusion model can use the facial key points for calculation in the implicit space;

[0011] S4, implicit space tensor splicing: initialize the Gaussian noise map, map it to the implicit space to obtain the implicit space tensor of the Gaussian noise map, splice the implicit space tensor of the Gaussian noise map with the implicit space tensor of the facial key point image information obtained in step S3, and send it to the diffusion model;

[0012] S5. Establish an implicit diffusion model: The implicit diffusion model uses the pre-trained Stable Diffusion 2.0, which is a deep neural network with a UNet structure. Through a multi-step reverse diffusion process, samples that meet the distribution of training data are obtained;

[0013] S6, model training: perform model training on the implicit diffusion model established in step S5, and continuously iteratively denoise the spliced ​​implicit space tensor, make a difference between the predicted noise and the real noise, and obtain a difference map after decoding the difference through the implicit space decoder. Perform key point detection on the difference map to obtain an attention heat map, and the attention allocates additional training attention to the key point conditions to achieve high generation quality and high consistency between the image and the conditions at the same time;

[0014] S7, result output: initialize the Gaussian noise map, use the denoising network trained in step S6, gradually predict the noise added in the current step according to the facial key points and text description information, use the implicit sampling algorithm to iteratively denoise the noise map, use the latent space decoder to map the implicit space denoising result to the pixel space, and finally reconstruct the face image consistent with the facial key points and text description information.

[0015] As an improvement of the present invention, in step S1, different facial images are collected, cropped, aligned, and super-resolution operations are performed on the collected facial images, and corresponding text descriptions and facial key point information are collected based on the facial images, and the facial images are used as corresponding label images.

[0016] As an improvement of the present invention, the text encoder in step S2 is a CLIP model, and the text conditions are converted into high-dimensional embedding vectors after encoding, wherein the descriptive text information corresponding to the face includes gender, age, eyebrow shape, hairstyle, etc., and the high-dimensional embedding vectors are used for text condition input in the generation process.

[0017] As another improvement of the present invention, the latent space encoder ε in step S3 adopts a pre-trained VAE encoder model to transform the face key point image c composed of the face key point coordinates into L Mapped to the implicit space, and the implicit space tensor z is formed L =ε(c L ); the implicit space tensor is a compact representation of the original key point image to reduce the computational complexity of the subsequent diffusion model; the corresponding VAE decoder is used as the latent space decoder The latent space decoder is used to reconstruct the latent space tensor into a pixel space image, namely:

[0018]

[0019] As another improvement of the present invention, in step S4, the implicit space tensor of the Gaussian noise image is obtained by the same implicit space encoder as step S3, and the implicit space tensor of the Gaussian noise image is spliced ​​with the implicit space tensor of the face key point image information and sent to the diffusion model; the splicing is specifically as follows:

[0020] z=concat(z noise ,z L ; dim = 1)

[0021] Among them, z noise =ε(x noise ) is the implicit spatial tensor of the Gaussian noise map, x noise ~N(0,1) is the initialization Gaussian noise map, and ε is the latent space encoder.

[0022] As another improvement of the present invention, the implicit diffusion model in step S5 includes an encoder module, an intermediate layer module and a decoder module, each module is composed of a residual block, a spatial attention module, an upsampling module, a downsampling module and a jump connection; wherein the residual block receives the time step encoding and the output of the previous upsampling module, the spatial attention module receives the text encoding and the output of the previous residual block, and learns the matching relationship between text information and face images through cross attention based on the text encoding. The diffusion model is initialized using the Stable Diffusion 2.0 network parameters, and the encoder input form is modified to adapt to the spliced ​​latent space tensor.

[0023] As a further improvement of the present invention, the loss function of the implicit diffusion model in step S5 is:

[0024]

[0025] Where t is the time step, z t is the tensor obtained after t steps of noise addition to the concatenated tensor z0 ∈ is the real noise, ∈ θ is the noise predicted by the model, c is the text condition c T and the key point condition c L A collection of .

[0026] As a further improvement of the present invention, in step S6, the denoising loss module guided by the heat map is used to allocate additional training attention to the key point conditions of the difference map. The denoising loss module is a face key point detector pre-trained by STAR Loss, which generates a corresponding attention heat map according to the difference map and allocates additional training attention to the key point conditions.

[0027] Compared with the prior art, the present invention has the following technical advantages and effects:

[0028] (1) In step S5 of the present invention, an innovative diffusion model is proposed, which guides the face generation method of the text map through key points, and uses the powerful generation ability of the implicit diffusion model to finely control facial features, thereby ensuring that the generated face image is consistent with the face key points and text conditions.

[0029] (2) The method of the present invention proposes a denoising loss module guided by a heat map, which, through differentiated learning, strengthens the control of facial features while avoiding excessive damage to the generalization performance of the model.

[0030] (3) The present invention eliminates the conflicts inherent in the dual-branch architecture when processing various conditions, and can accurately control the generation of facial features, such as the shape and position of the facial features. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is an example diagram of the face image-text description information-face key point data set in step S1 of the method of the present invention;

[0032] Figure 2 is a schematic diagram of a face generation method based on a key point guided Vincent graph of the present invention, wherein:

[0033] (a) is the basic structure diagram of the implicit diffusion model;

[0034] (b) is the structure diagram of the denoising loss module guided by the heat map;

[0035] Figure 3 Schematic diagram of the difference between the predicted facial noise and the actual noise in step S6 of embodiment 2 of the present invention;

[0036] Figure 4 It is a comparison diagram of face generation results using different combinations of key points and text conditions in the method of the present invention;

[0037] Figure 5 is a comparison chart of face generation results of different models in the test example of the present invention, where:

[0038] (a) is a schematic diagram of the results of the method of the present invention;

[0039] (b) is a schematic diagram of the ControlNet result;

[0040] (c) is a schematic diagram of the results of T2I-Adapter;

[0041] (d) is a schematic diagram of the results of Stable Diffusion 1.5;

[0042] (e) is a schematic diagram of the results of Stable Diffusion 2.1;

[0043] Figure 6 is a comparison diagram of face images generated under relatively uncommon text conditions and face key point conditions selected by different models in the test example of the method of the present invention, wherein:

[0044] (a) is a schematic diagram of the results of the method of the present invention;

[0045] (b) is a schematic diagram of the ControlNet result;

[0046] (c) is a schematic diagram of the results of T2I-Adapter;

[0047] Figure 7 : is a schematic diagram of the visualization results of the comparison of the conditions of the face and the key points of the face generated by different models in the test example of the present invention, wherein:

[0048] (a) is a schematic diagram of the results of the method of the present invention;

[0049] (b) is a schematic diagram of the ControlNet result;

[0050] (c) is a schematic diagram of the results of T2I-Adapter;

[0051] Figure 8 It is a flow chart of the steps of the face generation method based on the key point guided Vincent map of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0053] Example 1

[0054] The face generation method based on the key point guided Wensheng graph diffusion model specifically includes the following steps:

[0055] Step S1: collect text description information and facial key points of different facial images, integrate the corresponding facial images, text description information and facial key points, and construct training data pairs;

[0056] Collect different face images, perform cropping, alignment, and super-resolution operations on the collected face images, and collect corresponding text descriptions and facial key point information based on the face images, and use the face images as corresponding label images.

[0057] Step S2: Encode the description information corresponding to the face using a text encoder and convert it into an embedding vector, which is used as the text condition input in the generation process;

[0058] The CLIP model is selected as the text encoder to conditionally encode the text and convert it into a high-dimensional embedding vector. The descriptive text information corresponding to the face includes gender, age, eyebrow shape, hairstyle, etc. The high-dimensional embedding vector is used for text condition input in the generation process.

[0059] Step S3: Mapping the facial key point image information to the implicit space using the latent space encoder, so that the diffusion model uses the facial key points for calculation in the implicit space;

[0060] The latent space encoder uses a pre-trained VAE encoder model to map the facial key point image composed of facial key point coordinates to the implicit space and construct an implicit space tensor. The diffusion model uses the facial key point information for calculation in the implicit space. The implicit space tensor is a compact representation of the original key point image to reduce the amount of calculation of the subsequent diffusion model.

[0061] Step S4: Initialize the Gaussian noise map, map it to the implicit space through the latent space encoder, concatenate the implicit space tensor of the Gaussian noise map and the implicit space tensor of the facial key point image information in step S3, and send them to the diffusion model;

[0062] In this step, the same VAE encoder model as in step S3 is used to map the Gaussian noise map to the implicit space and construct a latent space tensor.

[0063] Step S5: Establish an implicit diffusion model. The diffusion model uses the pre-trained Stable Diffusion 2.0. This model is a deep neural network with a UNet structure. It can be guided by text conditions and gradually "denoise" to generate new data. Given random noise, the model converts it into clear samples that conform to the distribution of training data through a multi-step inverse diffusion process. Compared with other generative models (such as GAN, VAE), the implicit diffusion model can more effectively handle complex and high-dimensional data distributions, and has greater advantages in image generation quality and diversity. The diffusion model contains an encoder, an intermediate layer, and a decoder. These modules are composed of residual blocks, spatial attention modules, upsampling modules, downsampling modules, and jump connections. The diffusion model is initialized using the Stable Diffusion 2.0 network parameters, and the encoder input form is modified to adapt to the concatenated latent space tensor.

[0064] The residual block in the diffusion model receives the time step encoding and the output of the previous module, and the spatial attention module receives the text encoding and the output of the previous module. It learns the matching relationship between text information and face images through cross attention based on the text encoding.

[0065] Step S6: Perform global training on the constructed diffusion model, continuously iteratively denoise the concatenated implicit spatial tensor, and make a difference between the predicted noise and the actual noise. The difference is decoded by the VAE decoder to obtain a fuzzy difference map, which is then sent to the denoising loss module guided by the heat map. This module is a face key point detector pre-trained by STAR Loss. The detector generates a corresponding attention heat map based on the difference map and allocates additional training attention to the key point conditions to achieve both high generation quality and high image and condition consistency.

[0066] Step S7: Use the latent space decoder to map the denoising result in the latent space to the pixel space, and finally reconstruct a face image with high generation quality and high image and condition consistency.

[0067] Example 2

[0068] In this embodiment, the input of the implicit diffusion model is the text description information and facial key points of the face, where the text description information is 40 facial features, including Arched Eyebrows, Attractive, Bald, etc., which are facial description sentences composed of simple grammars; the facial key point map is the facial feature information located by 98 coordinate points, including eyebrows, eyes, nose, mouth, and facial contour areas. The facial text description information and facial key points contain certain facial information, but cannot fully express all facial features.

[0069] This embodiment is similar to Figure 1The image distribution shown in , using 27000 pairs of face images - text description information - face key point information for training, and using 3000 pairs of face images - text description information - face key point information for testing. The overall step flow scheme of the present invention is as follows Figure 2 As shown in the figure, the face generation is assisted by the text diffusion model in a key point guided manner. The face key point map and Gaussian noise map are first mapped to the implicit space and spliced, and then sent to the diffusion model. The text encoding information is injected into each module of the diffusion model. The powerful generation ability of the implicit diffusion model is used to generate facial features, and finally a high-fidelity face image consistent with the face key points and text conditions is reconstructed.

[0070] Face generation method based on key point guided Wensheng graph diffusion model, such as Figure 8 As shown, the specific steps include:

[0071] Step S1: Collect 30,000 different face images, and perform operations such as cropping, alignment, and super-resolution on the images to unify the face images into 512×512 images. According to the characteristics of the face images, 40 labels are marked for each image, including Arched Eyebrows, Attractive, Bald, etc., and then the face feature description labels are connected into text description sentences through simple syntax. The key points of the face images are marked, and the facial feature information located by 98 coordinate points includes eyebrows, eyes, nose, mouth, and facial contour areas. In this way, a face image-text description information-face key point information data pair is constructed, such as Figure 1 As shown, the first 27,000 data pairs are used as training sets, and the last 3,000 data pairs are used as test sets.

[0072] Step S2: Select the CLIP model as the text encoder, input the text description sentence into the CLIP model, and obtain a high-dimensional vector converted from the text description. The high-dimensional vector is 77 word embedding vectors, and the feature dimension of each vector is 768. These high-dimensional embedding vectors are subsequently sent to the diffusion model as the text conditional input c in the generation process. T .

[0073] Step S3: The latent space encoder ε uses the pre-trained VAE encoder model to transform the facial key point image c composed of 98 facial key point coordinates L Mapped to the implicit space, and the implicit space tensor z is formed L =ε(c L ), the diffusion model uses the facial key point information for calculation in the implicit space. The implicit space tensor is a compact representation of the original key point image to reduce the amount of calculation of the subsequent diffusion model.

[0074] The corresponding VAE decoder acts as a latent space decoder The latent space decoder is used to reconstruct the latent space tensor into a pixel space image, i.e.

[0075] Step S4: Initialize Gaussian noise image x noise ~N(0,1), the same VAE encoder model as in step S3 is used as the latent space encoder, and it is mapped to the implicit space through the latent space encoder ε to obtain the implicit space tensor z of the Gaussian noise map noise =ε(x noise ), concatenate the implicit spatial tensor of the Gaussian noise image with the implicit spatial tensor of the facial key point image information in step S3 z = concat (z noise , z L ,; dim=1) and send it to the diffusion model.

[0076] Step S5: Establish an implicit diffusion model; the diffusion model uses the pre-trained Stable Diffusion2.0, which contains an encoder, an intermediate layer, and a decoder. These modules are composed of residual blocks, spatial attention modules, upsampling modules, downsampling modules, and jump connections. Use the Stable Diffusion2.0 network parameters to initialize the diffusion model, and modify the encoder input form to adapt to the concatenated latent space tensor. The diffusion model is trained using L2 distance as the loss function:

[0077]

[0078] Where t is the time step, z t is the tensor obtained after t steps of noise addition to the concatenated tensor z0 ∈ is the real noise, ∈ θ is the noise predicted by the model, c is the text condition c T And the key point condition c L A collection of .

[0079] The residual block in the diffusion model receives the time step encoding and the output of the previous module, and the spatial attention module receives the text encoding and the output of the previous module. It learns the matching relationship between text information and face images through cross attention based on the text encoding.

[0080] Step S6: Figure 2 As shown in (a), the training diffusion model continuously iteratively denoises the concatenated implicit spatial tensor to obtain the face image corresponding to the key points and text, and uses the denoising loss module guided by the heat map to allocate additional training attention to the key point conditions to achieve high generation quality and high consistency between the image and the condition. The specific process is as follows:

[0081] The constructed diffusion model is trained globally, and the prediction noise ∈ of the diffusion model is obtained during the training process. θ , the real noise ∈ and the predicted noise ∈ θ The difference is then passed through the decoder After decoding, a fuzzy difference image is obtained, such as Figure 3 Shown

[0082]

[0083] Then the difference map M is sent to the denoising loss module guided by the heat map, such as Figure 2 As shown in (b), this module is a face key point detector pre-trained by STAR Loss. The detector generates the corresponding attention heat map based on the difference map.

[0084] H=F(M)

[0085] Here, H∈h×w×k, h and w represent the height and width of the attention heatmap, k is the number of key points, and F is the attention heatmap estimator in the denoising loss module. To simplify the calculation, the attention heatmaps are summed according to the dimension k to generate a single attention heatmap. Figure 3 As shown, the difference between the predicted noise and the real noise ∈-∈ θ The larger it is, the more obvious the face image in the difference map M is, and the larger the corresponding attention heat map H value is. The attention heat map H is sent to the latent space encoder ε to obtain the heat map encoding:

[0086]

[0087] Finally, the objective function of the optimized diffusion model can be expressed as:

[0088]

[0089]

[0090] Here W a =ω·H E , ω is set to 0.1 by default.

[0091] Step S6: During inference, use the trained denoising network to extract the noise image z t Estimate the noise-free face image z0 in . Load the trained diffusion model parameters to perform iterative denoising in the latent space, calculate the noise added in the previous step based on the current step noise, and iterate continuously to finally get the clean image z0. The sampling formula is as follows:

[0092]

[0093] Among them, z t is the noise sampled from a standard normal distribution, is a time-varying hyperparameter, t is the time step, ∈ θ represents the diffusion model, σ is the variance, and the default value is 0. The sampling formula is used for 50 iterations to obtain z0, and finally z0 is mapped to the pixel space through the latent space decoder to obtain a high-fidelity face image consistent with the face key points and text conditions.

[0094] Figure 4 This is a comparison diagram of face results generated by the method of the present invention after selecting different key points and text condition combinations. Figure 4 It can be seen that 7 groups of key points and 5 groups of text conditions are combined to form 35 key point and text condition pairs. Through the method of the present invention, 35 face images are finally obtained. Figure 4 As can be seen from the example, the present invention can generate high-fidelity face images for any combination of key points and text conditions, even for relatively uncommon feature combinations, such as a young man with long hair and a beard (i.e. Figure 4 , line 6), the present invention still successfully generates high-quality facial images.

[0095] Test Case

[0096] The method of the present invention is compared with the results of different face generation models. Figure 5 As shown, Figure 5 The comparison chart of face generation results of different models is shown. Figure 5 The text above each row of images and the key point image in the first column of the row constitute a pair of text and face key point conditions, which are respectively sent to the present invention, the dual-branch conditional generation model: ControINet and T2I-Adapter and the Wensheng graph diffusion model: Stable Diffusion1.5 and Stable Diffusion2.1, and then the corresponding 5 faces are generated (i.e., the ae column in the figure). The second column in the figure is the real face corresponding to the text and face key point conditions. Figure 5 As shown in FIG. 1 , under the same conditions, compared with faces generated by other models, the present invention can generate high-fidelity face images that are more consistent with face key points and text conditions, and are very similar to real images, such as Figure 5In the fourth row, compared with other models, the faces generated by the present invention are consistent with the real face expressions, facial orientation, and eye movement direction. Although ControlNet and T2I-Aapter can also generate faces based on text and facial key points, their generation effects on facial features and hair details are poor, such as the fourth row of column (b), the faces generated by ControlNet have different orientations from the real faces; in the fifth row of column (b), the faces generated by T2I-Aapter do not generate the wide-mouthed expression similar to the real faces. The results of StableDiffusion 1.5 and StableDiffusion 2.1 are difficult to be consistent with the facial key point conditions, as shown in the fourth rows of columns (c) and (d).

[0097] The face images generated by the method of the present invention and other models are compared under the condition of selecting relatively uncommon texts. Figure 6 shown. Figure 6 The text condition "a young man with long hair and beard" is combined with 7 groups of facial key points to form 7 pairs of key points and text conditions, and then the facial images are generated by the method of the present invention, ControlNet and T2I-Adapter model respectively. The results generated by the method of the present invention are as follows Figure 6 As shown in (a), Figure 6 As can be seen in (a), under the conditions of 7 sets of facial key points, the method of the present invention generates facial images consistent with the text conditions. The generation results of ControlNet are shown in Figure 6 As shown in (b), the face images generated by the model have gender inconsistencies, such as Figure 6 (b) The fourth and seventh faces. The results generated by T2I-Adapter are as follows Figure 6 As shown in (c), the face images generated by this model also have gender inconsistency, such as Figure 6 (c) In the fourth picture, there is still inconsistency between the face and hair features, such as Figure 6 (c) The seventh face. Therefore, under the condition of selecting relatively uncommon texts, the method of the present invention has higher accuracy.

[0098] Figure 7 The figure is a visualization result diagram of the comparison between the generated face and the face key point conditions of the method of the present invention and other models. The three key point and text condition pairs are respectively sent to the face images generated by the method of the present invention, ControlNet and T2I-Adapter model. Figure 7 The green dots represent the key points of the face used to guide the face generation, while the red dots represent the key points of the face corresponding to the generated face. Figure 7As shown in (a), the face generated by the present invention has a high consistency with the face key point conditions. The key points corresponding to the generated face (i.e., the red dots in the figure) almost coincide with the face key point conditions (i.e., the green dots in the figure) or are on the extension line of the green dots. The ControlNet generated result is as follows Figure 7 (b) Compared with the present invention, the generation effect of ControlNet is poor, and the position of the red dot is greatly offset compared with the green dot, as shown in Figure 7 In the second row of column (b), the generated result of ControlNet is quite different from the original image. Not only does this image fail to meet the key point condition of the character opening his mouth wide and closing his left eye, it also fails to meet the requirement of the face turning slightly to the left. Figure 7 (c) shows the generation result of this model is slightly better than ControlNet, but still inferior to the present invention. Figure 7 (c) In the second row of column, although the image meets the requirement that the face is slightly turned to the left, it still fails to meet the key point condition that the person has an open mouth and closes the left eye.

[0099] In summary, the method of the present invention trains the diffusion model to iteratively denoise the spliced ​​implicit space tensor, obtains the face image corresponding to the key points and text, and uses the denoising loss module guided by the heat map to allocate additional training attention to the key point conditions to achieve high generation quality and high image and condition consistency at the same time; uses the latent space decoder to map the denoising result in the implicit space to the pixel space, and finally reconstructs the face image with high generation quality and high image and condition consistency. The present invention can generate face images by guiding the Wensheng graph diffusion model through key points, and generate high-fidelity face images that are highly consistent with key points and text descriptions based on prior information.

[0100] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A face generation method based on a key point guided Wensheng graph diffusion model, characterized in that , including at least the following steps: S1. Collect and process data: collect image information of different faces, wherein the image information includes at least text description information and facial key points, and construct training data pairs of facial image-text description information-facial key points from corresponding facial images, text description information and facial key points; S2, text description information processing: using a text encoder to encode the text description information corresponding to the face image and convert it into an embedding vector; S3, facial key point image processing: using the latent space encoder to map the facial key point image information to the implicit space, and obtain the implicit space tensor of the facial key point image information; S4, implicit space tensor splicing: initialize the Gaussian noise map, map it to the implicit space to obtain the implicit space tensor of the Gaussian noise map, and splice the implicit space tensor of the Gaussian noise map with the implicit space tensor of the facial key point image information obtained in step S3; S5. Establish an implicit diffusion model: The implicit diffusion model uses the pre-trained Stable Diffusion 2.0, which is a deep neural network with a UNet structure. Through a multi-step reverse diffusion process, samples that conform to the distribution of training data are obtained; S6, model training: perform model training on the implicit diffusion model established in step S5, and continuously iteratively denoise the spliced ​​implicit space tensor, make a difference between the predicted noise and the real noise, and obtain a difference map after decoding the difference through the latent space decoder. Perform key point detection on the difference map to obtain an attention heat map, and the attention allocates additional training attention to the key point conditions; S7, result output: initialize the Gaussian noise map, use the denoising network trained in step S6, gradually predict the noise added in the current step according to the facial key points and text description information, use the implicit sampling algorithm to iteratively denoise the noise map, use the latent space decoder to map the implicit space denoising result to the pixel space, and finally reconstruct the face image consistent with the facial key points and text description information.

2. The face generation method based on key point guided Vincent graph diffusion model as claimed in claim 1, characterized in that: In the step S1, at least cropping, alignment and super-resolution operations are performed on the face image, and the face image is used as the corresponding label image according to the text description information and face key points corresponding to the face image.

3. The face generation method based on key point guided Vincent graph diffusion model as claimed in claim 1, characterized in that: The text encoder in step S2 is a CLIP model, and the text conditions are converted into high-dimensional embedding vectors after encoding.

4. The face generation method based on key point guided Vincent graph diffusion model as claimed in claim 1, characterized in that: In step S3, the latent space encoder ε adopts a pre-trained VAE encoder model to transform the facial key point image c composed of the facial key point coordinates L Mapped to the implicit space, and the implicit space tensor z is formed L =ε(c L ); the corresponding VAE decoder is used as the latent space decoder The latent space decoder is used to reconstruct the latent space tensor into a pixel space image, namely:

5. The face generation method based on key point guided Vincent graph diffusion model as claimed in claim 4, characterized in that: In step S4, the implicit space tensor of the Gaussian noise image is obtained by the same implicit space encoder as step S3, and the implicit space tensor of the Gaussian noise image is concatenated with the implicit space tensor of the facial key point image information as follows: z=concat(z noise ,from L ;dim=1) Among them, z noise =ε(x noise ) is the implicit spatial tensor of the Gaussian noise map, x noise ~N(0,1) is the initialization Gaussian noise map, and ε is the latent space encoder.

6. The face generation method based on key point guided Vincent graph diffusion model as claimed in claim 1, characterized in that: The implicit diffusion model in step S5 includes an encoder module, an intermediate layer module and a decoder module, each of which is composed of a residual block, a spatial attention module, an upsampling module, a downsampling module and a jump connection; wherein the residual block receives the time step encoding and the output of the previous upsampling module, the spatial attention module receives the text encoding and the output of the previous residual block, and learns the matching relationship between text information and face images through cross attention based on the text encoding.

7. The face generation method based on key point guided Vincent graph diffusion model as claimed in claim 6, characterized in that: The loss function of the implicit diffusion model in step S5 is: Where t is the time step, z t is the tensor obtained after t steps of noise addition to the concatenated tensor z0 ∈ is the real noise, ∈ θ is the noise predicted by the model, c is the text condition c T And the key point condition c L A collection of .

8. The face generation method based on key point guided Vincent graph diffusion model as claimed in claim 1, characterized in that: In step S6, the denoising loss module guided by the heat map is used to allocate additional training attention to the key point conditions of the difference map. The denoising loss module is a face key point detector pre-trained by STAR Loss. It generates a corresponding attention heat map according to the difference map and allocates additional training attention to the key point conditions.

Images