Face generation method based on attribute information assistance

By marking face attributes in the face generation method and building a multi-branch network, the problem of uncontrollability of face generation in the stable diffusion method is solved, the generation quality and consistency are improved, and the occurrence of attribute errors is reduced.

CN118982857APending Publication Date: 2024-11-19HANGZHOU YUNQI WISDOM VISION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411145560.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The face generation method based on stable diffusion is prone to uncontrollable attribute errors when generating brand new virtual identity faces, such as the phenomenon of men and women, which affects the performance of face recognition.

Method used

By marking face attributes in the training image, a multi-branch face recognition network and a face generation network are built, face attribute information is added, and a multi-branch network and loss function is used for training, the generated network includes an encoder, a forward diffusion layer, Fusion layer, Unnet and decoder, input the sample image to be expanded to output the newly generated face image.

Benefits of technology

This increases the controllability and stability of the face generation process, reduces the occurrence of attribute errors, and improves the generation quality and consistency in the case of small face samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982857A_ABST
    Figure CN118982857A_ABST
Patent Text Reader

Abstract

The invention discloses a face generation method based on attribute information assistance, and relates to the technical field of face image processing. Comprising the following steps: S1, marking face attributes in a training image; s2, constructing a multi-branch face recognition network and a loss function of the multi-branch face recognition network, wherein the multi-branch face recognition network comprises a backbone network, face attribute branches and face recognition branches; s3, constructing a face generation network and a loss function of the face generation network, wherein the face generation network comprises an encoder, a forward diffusion layer, a fusion layer, a Unet and a decoder; s4, training the multi-branch face recognition network and the face generation network; and S5, inputting a to-be-expanded sample image into the trained multi-branch face recognition network and the face generation network, and outputting a newly generated face image. According to the method, additional attribute information is added, and the controllability and stability of face random generation are improved, so that errors in face random generation are reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of face image processing, and in particular to a face generation method based on the assistance of attribute information. Background Art

[0002] With the development of deep learning, face recognition technology is widely used in various fields. As face recognition performance continues to improve, the demand for large-scale face data is becoming increasingly urgent. However, due to public concerns about personal privacy, the widespread collection and use of face datasets for commercial training has been restricted. The emergence of face generation technology has effectively alleviated the problem of data scarcity in face recognition.

[0003] Common face generation methods include: methods based on Generative Adversarial Networks (GAN), which generates realistic face images through adversarial training of generators and discriminators, and representative methods include DiscoFaceGAN and StyleGAN; methods based on 3D face reconstruction and rendering, which combine 3D facial models and GAN to allow fine control of generated faces, such as expression, posture and lighting, and representative methods include StyleRig and Rotate-and-Render; methods based on stable diffusion (SD), which restore images by gradually denoising to generate high-quality face images, and have good stability and diversity, and representative methods include Idiff-Face, etc. After experiments and tests, the method based on stable diffusion is comprehensively superior to the other two methods in terms of authenticity, generation diversity, identity consistency and ease of training.

[0004] The method based on stable diffusion can not only expand the face data with a small sample size, but also generate a new virtual identity face. This method usually controls the identity of the generated face by inputting identity control information, and controls the generation diversity by inputting random noise, so as to achieve the effect of generating n different face pictures of the same person in n experiments. Experiments have found that when the method based on stable diffusion generates a new virtual identity face, due to the small amount of newly generated face information, it is easy to have uncontrollable attribute errors, such as the phenomenon of mixing men and women in the n pictures of the same identity, which affects the final face recognition performance. Summary of the invention

[0005] In order to solve at least one of the technical problems mentioned in the background technology, the purpose of the present invention is to provide a face generation method based on the assistance of attribute information, add additional attribute information, increase the controllability and stability of random face generation, and thus reduce the errors in the random generation of faces.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for generating a human face based on attribute information assistance comprises the following steps:

[0008] S1, labeling facial attributes in training images;

[0009] S2, constructing a multi-branch face recognition network and a loss function of the multi-branch face recognition network, wherein the multi-branch face recognition network includes a backbone network, a face attribute branch, and a face recognition branch;

[0010] S3, constructing a face generation network and a loss function of the face generation network, wherein the face generation network includes an encoder, a forward diffusion layer, a fusion layer, a Unet and a decoder;

[0011] S4, training the multi-branch face recognition network and the face generation network;

[0012] S5, input the sample image to be expanded into the trained multi-branch face recognition network and face generation network, and output the newly generated face image.

[0013] In certain embodiments of the present invention, the facial attributes include gender, age group, and skin color; gender, age group, and skin color are each provided with at least two options and each facial attribute belongs to only one of the options.

[0014] In some embodiments of the present invention, the structure of the multi-branch face recognition network is as follows:

[0015] The backbone network uses the conv1, conv2_x, conv3_x, conv4_x, and conv5_x layers of resnet50. The input is a 224×224×3 face image, and the output is a 2048-dimensional vector.

[0016] The face attribute branch uses a fully connected layer and a BN layer connected in sequence. The input is the 2048-dimensional vector output by the backbone network, and the output is an 8-dimensional face attribute vector f_attr;

[0017] The face recognition branch uses a fully connected layer and a BN layer connected in sequence. The input is the 2048-dimensional vector output by the backbone network, and the output is a 512-dimensional face embedding vector f_emb;

[0018] The face ID classifier uses a fully connected layer and a Softmax layer connected in sequence. The input is the face embedding vector f_emb output by the face recognition branch, and the output is the face ID vector f_id.

[0019] In some embodiments of the present invention, the loss function of the multi-branch face recognition network is:

[0020] L face =α*L attr +L id

[0021] Where: L face is the loss of the multi-branch face recognition network; L attr is the loss function of the face attribute branch; L id is the loss of the face ID classifier; α is L attr The weight coefficient of .

[0022] In some embodiments of the present invention, the loss L of the face attribute branch attr for:

[0023]

[0024] Among them, f_attr[i] is the i-th dimension of the face attribute vector output by the face attribute branch; q i is the i-th dimension of the real face attribute vector of the training image.

[0025] In some embodiments of the present invention, the loss L of the face ID classifier is id for:

[0026]

[0027] Where N is the total number of training images, i is the image sequence number, y is the real ID label of the training image, and f_id[i] is the i-th dimension of the face ID vector output by the global feature network.

[0028] In some embodiments of the present invention, the structure of the face generation network is as follows:

[0029] Encoder, which uses the encoder part of the variational autoencoder, with a 512×512×3 face image as input and a 64×64×4 latent space feature map as output;

[0030] The forward diffusion layer adopts a linear noise scheduling strategy, with the input being the 64×64×4 latent space feature map output by the encoder. The time step sequence and the corresponding noise intensity sequence are set by the linear scheduling strategy. For any time t in the time step sequence, Gaussian noise ∈ is sampled, and Gaussian noise is added to the latent space map according to the noise intensity at time t, and the 64×64×4 latent space noise map at time t is calculated.

[0031] The fusion layer uses a dropout layer, a concatenation layer, and a BN layer connected in sequence. The input is an 8-dimensional face attribute vector and a 512-dimensional face embedding vector. The face embedding vector is input into the dropout layer, and 8 dimensions are randomly dropped to obtain a 504-dimensional vector. The 504-dimensional vector is concatenated with the 8-dimensional face attribute vector and input into the BN layer. The output is a 512-dimensional conditional vector.

[0032] Unet, using UNet 2D Condition Model, takes as input the 64×64×4 latent space noise map output by the forward diffusion layer and its time t and the 512-dimensional condition vector output by the fusion layer. The condition vector is injected into multiple layers inside Unet through the cross attention module inside Unet to guide Unet to output a 64×64×4 noise residual map;

[0033] The cyclic denoising layer uses a denoising diffusion probability model. The initial input is the 64×64×4 latent space noise map output by the forward diffusion and the 64×64×4 noise residual map output by Unet. The noise residual map is removed from the input latent space noise map through the denoising diffusion probability model to obtain a 64×64×4 intermediate process noise map. The intermediate process noise map is used to replace the initial input latent space noise map. The Unet and cyclic denoising layer are repeated. The process is cycled N times and finally a 64×64×4 denoised latent space feature map is output;

[0034] Decoder,The decoder part of the variational autoencoder is used, with the input being a 64×64×4 denoised latent space feature map and the output being a 512×512×3 generated face image.

[0035] In some embodiments of the present invention, the loss function of the face generation network is:

[0036]

[0037] Among them, t is the t moment in the time step, x t is the latent space noise map at time t, ∈ is the Gaussian noise sampled at time t, ∈(x t ,t) is the noise residual map output by Unet; E[||·||22] represents the mean square error.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention marks facial attributes in training images, thereby increasing the controllability and stability of the random face generation process, thereby reducing uncontrollable attribute errors that are prone to occur due to less newly generated facial information. When the number of face samples is small, the sample size can be expanded according to actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a flow chart of the overall method of the present invention.

[0041] Figure 2 This is a structural diagram of the multi-branch face recognition network of the present invention.

[0042] Figure 3 Generate a network structure diagram for the face of the present invention.

[0043] Figure 4 A schematic diagram of multiple facial images generated according to an embodiment of the present invention.

[0044] Figure 5 Schematic diagram of the generation results of different face attribute vectors according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] See also Figure 1 This embodiment provides a method for generating a face based on attribute information, comprising the following steps:

[0047] S1, labeling facial attributes in training images.

[0048] Specifically, the facial attributes include gender (male / female), age group (children / youth / elderly), skin color (yellow / white / black), etc. The content in brackets represents the options for each attribute.

[0049] It is worth mentioning that each attribute is set with at least two options and each face attribute belongs to only one of the options.

[0050] The labeled facial attributes are concatenated into an 8-dimensional facial attribute vector. The 8 dimensions of the facial attribute vector represent [male, female, child, young, old, yellow, white, black]. If a dimension is labeled as true, the value of this dimension of the facial attribute vector is 1, otherwise it is 0. For example, if a face image is labeled as male, young, black, then its facial attribute vector is [1, 0, 0, 1, 0, 0, 0, 1].

[0051] S2, constructing a multi-branch face recognition network and a loss function of the multi-branch face recognition network, wherein the multi-branch face recognition network includes a backbone network, a face attribute branch and a face recognition branch.

[0052] like Figure 2 As shown, the structure of the multi-branch face recognition network is as follows:

[0053] The backbone network uses the conv1, conv2_x, conv3_x, conv4_x, and conv5_x layers of resnet50. The input is a 224×224×3 face image, and the output is a 2048-dimensional vector.

[0054] The face attribute branch uses a fully connected layer and a BN layer connected in sequence. The input is the 2048-dimensional vector output by the backbone network, and the output is an 8-dimensional face attribute vector f_attr;

[0055] The face recognition branch uses a fully connected layer and a BN layer connected in sequence. The input is the 2048-dimensional vector output by the backbone network, and the output is a 512-dimensional face embedding vector f_emb;

[0056] The face ID classifier uses a fully connected layer and a Softmax layer connected in sequence. The input is the face embedding vector f_emb output by the face recognition branch, and the output is the face ID vector f_id.

[0057] Construct the loss function of the face attribute branch and the face ID classifier, and the loss function of the weighted multi-branch face recognition network is:

[0058] L face =α*L attr +L id

[0059] Where: L face is the loss of the multi-branch face recognition network; L attr is the loss function of the face attribute branch, which uses the cross entropy between the face attribute vector output by the face attribute branch and the real face attribute vector of the training set image; L id is the loss of the face ID classifier, using the cross entropy of the face ID vector output by the face ID classifier and the real ID label of the training image; α is L attr The weight coefficient is preferably 0.15.

[0060] The loss L of the face attribute branch attr for:

[0061]

[0062] Among them, f_attr[i] is the i-th dimension of the face attribute vector output by the face attribute branch; q i is the i-th dimension of the real face attribute vector of the training image.

[0063] In some embodiments of the present invention, the loss L of the face ID classifier is id for:

[0064]

[0065] Where N is the total number of training images, i is the image sequence number, y is the real ID label of the training image, and f_id[i] is the i-th dimension of the face ID vector output by the global feature network.

[0066] S3, constructing a face generation network and a loss function of the face generation network, wherein the face generation network includes an encoder, a forward diffusion layer, a fusion layer, a Unet and a decoder.

[0067] like Figure 3 As shown, the structure of the face generation network includes an encoder, a forward diffusion layer, a fusion layer, a Unet, a back diffusion, and a decoder.

[0068] 4.1 Encoder,The encoder part of the variational auto encoder (VAE) is used. The input is a 512×512×3 face image and the output is a 64×64×4 latent space feature map.

[0069] 4.2 Forward diffusion layer, adopts linear noise scheduling strategy, the input is the 64×64×4 latent space feature map output by the encoder, and the time step sequence and the corresponding noise intensity sequence are set by the linear scheduling strategy; for any time t in the time step sequence, Gaussian noise ∈ is sampled, and Gaussian noise is added to the latent space map according to the noise intensity at time t. This step can directly calculate the 64×64×4 latent space noise map at time t.

[0070] 4.3 Fusion layer, using the dropout layer, concatenation layer, BN layer connected in sequence, the input is an 8-dimensional face attribute vector and a 512-dimensional face embedding vector, the face embedding vector is input into the dropout layer, 8 dimensions are randomly dropped out, and a 504-dimensional vector is obtained. The 504-dimensional vector is concatenated with the 8-dimensional face attribute vector and input into the BN layer, and the output is a 512-dimensional conditional vector.

[0071] 4.4Unet, using UNet 2D Condition Model, with the input being the 64×64×4 latent space noise map output by the forward diffusion layer and its time t and the 512-dimensional condition vector output by the fusion layer. The condition vector is injected into multiple layers inside Unet through the cross-attention module inside Unet to guide Unet to output a 64×64×4 noise residual map;

[0072] 4.5 cyclic denoising layer adopts denoising diffusion probability model (DDPM). The initial input is the 64×64×4 latent space noise map output by forward diffusion and the 64×64×4 noise residual map output by Unet. The noise residual map is removed from the input latent space noise map through the denoising diffusion probability model to obtain a 64×64×4 intermediate process noise map. The intermediate process noise map is used to replace the initial input latent space noise map. Repeat 4.4 Unet and 4.5 cyclic denoising layers. The process is repeated N times and finally a 64×64×4 denoised latent space feature map is output.

[0073] 4.6 Decoder, the decoder part of the variational autoencoder (VAE) is used, the input is a 64×64×4 denoised latent space feature map, and the output is a 512×512×3 generated face image.

[0074] The loss function of the face generation network is:

[0075]

[0076] Among them, t is the t moment in the time step, x t is the latent space noise map at time t, ∈ is the Gaussian noise sampled at time t, ∈(x t ,t) is the noise residual map output by Unet; represents the mean square error.

[0077] S4, training the multi-branch face recognition network and the face generation network;

[0078] S5, input the sample image to be expanded into the trained multi-branch face recognition network and face generation network, and output the newly generated face image.

[0079] like Figure 4 FIG. 1 shows multiple faces generated based on the same face ID vector according to an embodiment. Figure 5 What is shown is the generation result of different face attribute vectors based on the same face embedding vector according to an embodiment.

[0080] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A face generation method based on attribute information assistance, It is characterized in that The following steps are involved: S1, labeling facial attributes in training images; S2, constructing a multi-branch face recognition network and a loss function of the multi-branch face recognition network, wherein the multi-branch face recognition network includes a backbone network, a face attribute branch, and a face recognition branch; S3, constructing a face generation network and a loss function of the face generation network, wherein the face generation network includes an encoder, a forward diffusion layer, a fusion layer, a Unet and a decoder; S4, training the multi-branch face recognition network and the face generation network; S5, input the sample image to be expanded into the trained multi-branch face recognition network and face generation network, and output the newly generated face image.

2. The method for generating a human face based on attribute information according to claim 1, It is characterized in that The facial attributes include gender, age group, and skin color; gender, age group, and skin color are respectively set with at least two options and each facial attribute belongs to only one of the options.

3. The method for generating a human face based on attribute information according to claim 2, It is characterized in that The structure of the multi-branch face recognition network is as follows: The backbone network uses the conv1, conv2_x, conv3_x, conv4_x, and conv5_x layers of resnet50. The input is a 224×224×3 face image, and the output is a 2048-dimensional vector. The face attribute branch uses a fully connected layer and a BN layer connected in sequence. The input is the 2048-dimensional vector output by the backbone network, and the output is an 8-dimensional face attribute vector f_attr; The face recognition branch uses a fully connected layer and a BN layer connected in sequence. The input is the 2048-dimensional vector output by the backbone network, and the output is a 512-dimensional face embedding vector f_emb; The face ID classifier uses a fully connected layer and a Softmax layer connected in sequence. The input is the face embedding vector f_emb output by the face recognition branch, and the output is the face ID vector f_id.

4. The method for generating a human face based on attribute information according to claim 3, It is characterized in that The loss function of the multi-branch face recognition network is: Where: L face is the loss of the multi-branch face recognition network; L attr is the loss function of the face attribute branch; L id is the loss of the face ID classifier; α is L attr The weight coefficient of .

5. The method for generating a human face based on attribute information according to claim 4, It is characterized in that The loss of the face attribute branch for: Among them, f_attr[i] is the i-th dimension of the face attribute vector output by the face attribute branch; q i is the i-th dimension of the real face attribute vector of the training image.

6. The method for generating a human face based on attribute information according to claim 4, It is characterized in that The loss of the Face ID classifier for: Where N is the total number of training images, i is the image sequence number, y is the real ID label of the training image, and f_id[i] is the i-th dimension of the face ID vector output by the global feature network.

7. The method for generating a human face based on attribute information according to claim 1, It is characterized in that The structure of the face generation network is as follows: Encoder, which uses the encoder part of the variational autoencoder, with a 512×512×3 face image as input and a 64×64×4 latent space feature map as output; The forward diffusion layer adopts a linear noise scheduling strategy. The input is the 64×64×4 latent space feature map output by the encoder. The time step sequence and the corresponding noise intensity sequence are set by the linear scheduling strategy. For any time t in the time step sequence, the Gaussian noise is sampled. , add Gaussian noise to the latent space map according to the noise intensity at time t, and calculate the 64×64×4 latent space noise map at time t; The fusion layer uses a dropout layer, a concatenation layer, and a BN layer connected in sequence. The input is an 8-dimensional face attribute vector and a 512-dimensional face embedding vector. The face embedding vector is input into the dropout layer, and 8 dimensions are randomly dropped to obtain a 504-dimensional vector. The 504-dimensional vector is concatenated with the 8-dimensional face attribute vector and input into the BN layer. The output is a 512-dimensional conditional vector. Unet, using UNet 2D Condition Model, takes as input the 64×64×4 latent space noise map output by the forward diffusion layer and its time t and the 512-dimensional condition vector output by the fusion layer. The condition vector is injected into multiple layers inside Unet through the cross attention module inside Unet to guide Unet to output a 64×64×4 noise residual map; The cyclic denoising layer uses a denoising diffusion probability model. The initial input is the 64×64×4 latent space noise map output by the forward diffusion and the 64×64×4 noise residual map output by Unet. The noise residual map is removed from the input latent space noise map through the denoising diffusion probability model to obtain a 64×64×4 intermediate process noise map. The intermediate process noise map is used to replace the initial input latent space noise map. The Unet and cyclic denoising layer are repeated. The process is cycled N times and finally a 64×64×4 denoised latent space feature map is output; Decoder,The decoder part of the variational autoencoder is used, with the input being a 64×64×4 denoised latent space feature map and the output being a 512×512×3 generated face image.

8. The method for generating a human face based on attribute information according to claim 7, It is characterized in that The loss function of the face generation network is: Among them, t is the t moment in the time step, x t is the latent space noise map at time t, is the Gaussian noise sampled at time t, is the noise residual map output by Unet; E[||·||2 2] represents the mean square error.