A sample-guided diversified restoration method for face images
By introducing self-supervised attribute consistent learning and gradient space variant layer in the generative adversarial network, the problem of difficulty in capturing semantic relationships and single results in image repair in the prior art is solved, and diversified repair and high-quality output of face images are achieved.
Patent Information
- Application Number
- CN202210026270.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-11
- Publication Date
- 2025-04-29
- Estimated Expiration
- 2042-01-11
AI Technical Summary
Existing image repair methods are difficult to capture the semantic relationships of images, resulting in poor performance in complex images, and deep learning-based methods have problems such as single results and difficulty for users to guide output effects.
A diversified repair method of face images based on sample guidance is proposed. Using the generative adversarial network architecture, a self-supervised attribute consistency learning method and gradient space variation layer are introduced, and the attribute information of sample images is automatically encoded and face repair is guided.
The quality and diversity of face repair is achieved, ensuring that the repair area clearly displays the attribute information in the face pictures of the sample, and improving the diversity and user experience of the repair results.
Smart Images
Figure CN114549341B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image restoration, and in particular to a sample-guided diversified restoration method for facial images. Background Art
[0002] In image restoration methods, traditional restoration methods can use similar patches around the image to fill in the missing image areas, but these methods cannot capture the semantic relationship of the image well, resulting in these methods often performing poorly in complex images.
[0003] Although existing deep learning-based restoration methods can generate satisfactory results in missing areas, these methods have the following problems: First, the image restoration results of most methods tend to be single, and users cannot choose the restoration results they want; second, for methods that can output multiple restoration results, although the results are diverse, users cannot guide the desired output effect; third, some restoration methods can introduce some auxiliary information, such as sketches, strokes, and key points to control the generation of faces. However, these methods are too dependent on prior information, resulting in restoration results fitting the prior, limiting restoration diversity.
[0004] Therefore, in order to solve the above problems, the present invention proposes a diversified facial image restoration method based on sample guidance. Based on the generative adversarial network architecture, this method first introduces a self-supervised attribute consistency learning method, which can help the network learn the clear mapping of sample style encoding and corresponding facial attributes; secondly, a gradient space variation layer is proposed, which can use the relationship between pixel spatial positions to adjust the constraints on network parameters in the back propagation stage. The present invention can automatically encode the attributes of a given sample image, and use this attribute information to guide face restoration, ensuring the quality and diversity of face restoration, so that the repaired area can clearly show the attribute information in the sample face image. Summary of the invention
[0005] The technical problem to be solved by the embodiments of the present invention is to provide a diversified facial image restoration method based on sample guidance, which can use the facial attribute information of a given sample image to guide face restoration, and ensure the quality and diversity of face restoration, and ensure that the restoration area can clearly display the attribute information in the sample face image.
[0006] In order to solve the above technical problems, an embodiment of the present invention provides a method for diversified restoration of facial images based on sample guidance, the method comprising the following steps:
[0007] Step S1: Given a training set And the test set Among them, a irepresents the i-th image in the training set, 3 represents the number of color channels, the corresponding color channel d∈{red, green, blue}, and K represents the width or height of a single image; a j represents the jth image in the test set. N represents the training set I train The number of samples; H represents the test set I test The number of samples;
[0008] Step S2: construct a network model for realizing diverse face restoration based on sample guidance; the network model at least includes a mapping network, a style network, a generator network and a discriminator network; wherein the mapping network f is constructed using a fully connected neural network for generating random style codes, and its parameters are represented by θ f ; Construct a style network E to map the sample image into a sample style code, and its parameters are expressed as Construct a generator network G to combine random style coding, sample style coding and global style vector to repair the input image and obtain a face repair result containing sample attributes. The parameter of G is represented by θ g ; Construct a discriminator network D for generative adversarial training to assist the generator G in learning the data distribution of real samples. The parameter of D is represented by θ d ; Finally, all networks are initialized. Among them, the style network E uses the parameters of the pre-trained model Initialize and do not participate in the update of model parameters. Therefore, in the qth iteration training, the parameters of the mapping network f, the generator network G and the discriminator network D are set to
[0009] Step S3: During the qth iteration training process, traverse all samples of the training set. i , randomly extract a sample graph from the training set And generate a random binary mask To construct the corresponding input graph Generate a random Gaussian noise Input noise into the mapping network f to generate a random style code Input the sample image e into the style network to generate the corresponding sample style code The input graph r i , a binary mask m, and a random style encoding Mixed style encoding mixed with sample style encoding w+ Input into generator G to get the repair result
[0010] Step S4: Combine the generated adversarial loss Ladv , spatial variant-aware loss L LPIPS , identity loss L id , and attribute consistency loss L attri The loss value is calculated. Back propagation is performed based on the loss value, and the parameters of the mapping network f, the generator network G, and the discriminator network D are updated and adjusted using the stochastic gradient descent method.
[0011] Step S5: Using test set I test The method model of the present invention is evaluated. If the restoration quality index of the model in the test image is the highest at present, the model parameters are saved and It represents the optimal model of the current mapping network f, generator network G and discriminator network D. It is judged whether the training has reached the maximum number of iterations n. If it has reached the maximum number of iterations n, the training phase ends and jumps to step S6, otherwise jumps to step S3.
[0012] Step S6: Obtain the optimal model parameters of the mapping network f, the generator network G, and the discriminator network D.
[0013] As a further improvement, in step S1, the training set I train and test set I test They are all collections of images, and no image labels are required.
[0014] As a further improvement, in step S2, the mapping network f can be a fully connected layer (FC) neural network without an activation function, the purpose of which is to learn linear correlations and decouple facial attributes in the random latent space, such as the mapping network in StyleGAN (T.Karras, S.Laine, and T.Aila, ``A style-based generator architecture for generative adversarial networks,” in Proc.CVPR, 2019, pp.4401-4410.) and StyleGANv2 (T.Karras, S.Laine, M.Aittala, J.Hellsten, J.Lehtinen, and T.Aila, ``Analyzing and improving the image quality of styleGAN,” in Proc.CVPR, 2020, pp.8110-8119.). The style network E is a pre-trained style encoding network used to encode facial attributes into the corresponding sample space, such as the encoder of pSp (E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or, ``Encoding in style: a styleGAN encoder for image-to-image translation, ” in Proc. CVPR, 2021, pp. 2287-2296.). The encoder G is included in the generator G. en and decoder G de , a multi-style encoding method is introduced to combine random style encoding The sample style code w+ and the global style vector c are used to repair the input image to obtain a face repair result containing the sample attributes. The sample style code w+ and the global style vector c are encoded and fused for modulation and demodulation (modulation, demodulation, for the style layer, please refer to StyleGAN and StyleGANv2) to achieve sample-guided diversified restoration of face images. The generator network G can use the style encoding method of the present invention (introduced in step S3) based on Co-mod-GAN (S.Zhao, J.Cui, Y.Sheng, Y.Dong, X.Liang, EI Chang, and Y.Xu, ``Large scale image completion via co-modulated generative adversarial networks,” in Proc.ICLR, 2021.). The discriminator network D can directly use the discriminator network of Co-mod-GAN.
[0015] As a further improvement, in step S3, given the i-th training sample a i , random sample graph Random binary mask Next, input the graph The construction formula is as follows:
[0016] r i =a i ⊙(1-m),
[0017] The value range of each scalar in m is [0,1]. A value of 0 indicates a non-masked area, and a value of 1 indicates a masked area. represents a tensor whose values are all 1, and ⊙ is the Hadamard Product. Through the broadcast mechanism, i In the case of m, the information of the pixels corresponding to the masked area will be erased and represented as 0. In the case of random style encoding The calculation formula is as follows:
[0018]
[0019] in, Represents the extended stochastic style latent space. Sample style encoding The calculation formula is as follows:
[0020]
[0021] Where w+∈W+, W+ represents the style latent space, and T represents the subscript index set of the corresponding style layer. Then, the random style is encoded and sample style encoding Perform style mixing to obtain the fused style encoding The formula is as follows:
[0022]
[0023] Among them, φ is a binary label vector that indicates which style vector will be mixed. The calculation formula is as follows:
[0024]
[0025] Among them, the generator G contains the encoder G en and decoder G de , a multi-style encoding method is introduced to combine random style encoding The sample style encoding w+ and the global style vector c are used to obtain the face restoration result containing the sample attributes. The calculation formula of the corresponding style representation vector is:
[0026]
[0027] Among them, the style representation vector v idx It will be used for modulation and demodulation later. represents the learnable affine transformation function (affine transformation) corresponding to the idx-th style layer. c represents the encoder G in the generator G de For input r i and m to encode the global style vector (for modulation, demodulation, and affine transformation, please refer to StyleGAN and StyleGANv2).
[0028] As a further improvement, in step S4, the present invention first proposes a spatial variant gradient layer (SVGL), which is defined as:
[0029] P(x,M x ) = x,
[0030]
[0031] Where x represents the input feature, which can be an image or a feature map. x Represents the spatial weight mask. The above formula defines the forward propagation calculation and the partial derivative formula of the gradient spatial variation layer P(·,·) for back propagation. In the forward calculation, P(·,·) does not change any input information, but introduces the corresponding spatial weight mask during back propagation to adjust the spatial constraints of each output pixel according to the spatial relationship.
[0032] Next, the present invention proposes a spatial variant perception loss L LPIPS , the formula is as follows:
[0033]
[0034] where F(·) is a pre-trained feature extractor such as the VGG network. is a confidence weight mask (confidence weight mask, please refer to: Y.Wang, X.Tao, X.Qi, Xi.Shen, and J.Jia, ``Image inpainting via generative multi-columnconvolutional neural networks, "in Proc.NeurIPS, 2018, pp.331-340.) after linear function normalization (Min-Max scaling). The value range of each scalar is [0,1]. The value is close to 1, and the higher the weight is. When the original image a i When it is equal to the sample e, there is no contradiction in the constraints guided by the facial attributes, so the perceptual loss (LPIPS loss, please refer to: R.Zhang, P.Isola, AAEfros, E.Shechtman, and O.Wang, ``The unreasonable effectiveness of deepfeatures as a perceptual metric,” in Proc.CVPR, 2018, pp.586-595.) can be used directly. When the original image a i When it is not equal to the sample e, the mask boundary can be constrained, and the output in the middle of the mask can be relaxed to force the repair result. The pixels around the mask and the original image a i There is a higher perceived similarity.
[0035] The property consistent loss L of the present invention attri As shown below:
[0036]
[0037] in, is the restored image output by the style network E The style code obtained by encoding, are the pre-trained parameters of the style network E. is the reverse weight mask, and its calculation formula is:
[0038]
[0039] That is, based on the confidence weight mask, the inverse weight mask is calculated. i When it is equal to sample e, directly repair the result Encode and calculate each Vector Sum The mean square error loss of the vector. i When it is not equal to the sample e, it is necessary to combine the inverse weight mask to encourage the area far away from the mask boundary to present the attributes of the sample face while maintaining boundary consistency. Generate adversarial loss L adv (GAN loss) Please refer to StyleGAN, identity loss L id (identity loss)Please refer to pSp.
[0040] Will fight against the loss L adv , spatial variant-aware loss L LPIPS , identity loss L id , and attribute consistency loss L attri By performing common constraints, the final objective function of the method of the present invention is obtained, and its formula is:
[0041]
[0042] where θ g ,θ f ,θ d and Respectively represent the parameters of the generator network G, mapping network f, discriminator network D and style network E. By performing maximum minimization training on the mapping network f, the pre-trained style network E, the generator network G and the discriminator network D, the formula is as follows:
[0043]
[0044] Among them, The parameters of the style network are not involved in training. During the inference process, only the mapping network f, style network E and generator network G are needed to perform sample-guided diverse image restoration. The discriminator network D is used to distinguish between true and false samples during training. By generating adversarial methods, the model can learn the distribution of real samples and improve the restoration quality.
[0045] Back propagation is performed according to the loss value, and the parameters of the mapping network f, the generator network G, and the discriminator network D at the qth iteration are adjusted using the stochastic gradient descent method. and To update and adjust, the specific steps are:
[0046] According to the final objective function, the chain derivation rule is used to adjust the parameters of the mapping network f, the generator network G, and the discriminator network D at the qth iteration. and Calculate the gradient and use the stochastic gradient descent method to reduce the corresponding network parameters and To update, the formula is as follows:
[0047]
[0048] Among them, η represents the learning rate in the hyperparameter; and They represent the gradients of the parameters of the corresponding mapping network f, generator network G, and discriminator network D at the qth iteration, respectively, and are obtained by the chain rule.
[0049] Wherein, in step S5, using test set I test The method model of the present invention is evaluated, and the specific steps are:
[0050] The test set Each sample in is traversed one by one, and each sample is calculated using the calculation method described in step S3. The corresponding repair result set is obtained For the test set I test and the repair result set Y testCalculate the Fr′echetinception distance indicator. (FID, please refer to: M.Heusel, H.Ramsauer, T.Unterthiner, B.Nessler, and S.Hochreiter, ``GANs trained by a two time-scale update ruleconverge to a local nash equilibrium,” in Proc.NIPS, 2017, pp.6626-6637.). Here we define the optimal model network parameters and The FID indicator is fid best , if fid q >fid best , then let fid best =fid q .
[0051] Compared with the prior art, the present invention has the following beneficial effects:
[0052] 1. Compared with the existing image restoration methods, the present invention can use the attribute information of a given sample image to guide face restoration, and ensure the quality and diversity of face restoration, ensure that the restoration area can clearly show the attribute information in the sample face image, and has good value and prospects in practical applications;
[0053] 2. The present invention proposes a diversified facial image restoration method based on sample guidance. Based on the generative adversarial network architecture, this method first introduces a self-supervised attribute consistency learning method, which can help the network learn to embed a clear mapping of sample style and facial attributes; secondly, a gradient space variation layer is proposed, which can use the relationship between the spatial positions of pixels to adjust the constraints of the loss function on the network parameters in the back-propagation stage. The gradient space variation layer is a general constraint guidance layer. This module has strong versatility and can be applied to any loss function, so that the loss function can dynamically adjust the gradient penalty of the corresponding parameters according to the spatial relationship of the pixels. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, without paying creative labor, other drawings obtained based on these drawings still belong to the scope of the present invention.
[0055] Figure 1A flowchart of a method for diversified restoration of facial images based on sample guidance provided by an embodiment of the present invention;
[0056] Figure 2 An architecture diagram of a sample-guided diversified facial image restoration method provided by an embodiment of the present invention; DETAILED DESCRIPTION
[0057] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.
[0058] like Figure 1 FIG. 1 is a flowchart of a method for diversified restoration of facial images based on sample guidance according to an embodiment of the present invention. The method comprises the following steps:
[0059] Step S1: Given a training set And the test set Among them, a i represents the i-th image in the training set, 3 represents the number of color channels, the corresponding color channel d∈{red, green, blue}, and K represents the width or height of a single image; a j represents the jth image in the test set. N represents the training set I train The number of samples; H represents the test set I test The number of samples; among them, the training set I train and test set I test They are all collections of images, and no image labels are required.
[0060] Step S2: construct a network model for realizing diverse restoration of face images based on sample guidance; the network model at least includes a mapping network, a style network, a generator network and a discriminator network; wherein the mapping network f is constructed using a fully connected neural network for generating random style codes, and its parameters are represented by θ f ; Construct a style network E to map the sample image into a sample style code, and its parameters are expressed as Construct a generator network G to combine random style coding, sample style coding and global style vector to repair the input image and obtain a face repair result containing sample attributes. The parameter of G is represented by θ g ; Construct a discriminator network D for generative adversarial training to assist the generator G in learning the data distribution of real samples. The parameter of D is represented by θ d ; Finally, all networks are initialized. Among them, the style network E uses the parameters of the pre-trained model Initialize and do not participate in the update of model parameters. Therefore, in the qth iteration training, the parameters of the mapping network f, the generator network G and the discriminator network D are set to
[0061] Among them, the mapping network f can be a fully connected layer (FC) neural network without an activation function. The purpose is to learn linear correlations and decouple facial attributes in the random latent space, such as the mapping network in StyleGAN (T.Karras, S.Laine, and T.Aila, ``A style-based generator architecture for generative adversarial networks,” in Proc.CVPR, 2019, pp.4401-4410.) and StyleGANv2 (T.Karras, S.Laine, M.Aittala, J.Hellsten, J.Lehtinen, and T.Aila, ``Analyzing and improving the image quality of styleGAN,” in Proc.CVPR, 2020, pp.8110-8119.). The style network E is a pre-trained style encoding network used to encode facial attributes into the corresponding sample space, such as the encoder of pSp (E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or, ``Encoding in style: a styleGAN encoder for image-to-image translation, ” in Proc. CVPR, 2021, pp. 2287-2296.). The encoder G is included in the generator G. en and decoder G de , a multi-style encoding method is introduced to combine random style encoding The sample style code w+ and the global style vector c are used to repair the input image to obtain a face repair result containing the sample attributes. The sample style code w+ and the global style vector c are encoded and fused for modulation and demodulation (modulation, demodulation, for style layer, please refer to StyleGAN and StyleGANv2) of the style layer to achieve sample-guided diverse face restoration. The generator network G can use the style embedding method of the present invention (introduced in step S3) based on Co-mod-GAN (S.Zhao, J.Cui, Y.Sheng, Y.Dong, X.Liang, EI Chang, and Y.Xu, ``Large scale image completion via co-modulated generative adversarial networks,” in Proc.ICLR, 2021.). The discriminator network D can directly use the discriminator network of Co-mod-GAN.
[0062] Step S3: During the qth iteration training process, traverse all samples of the training set. i , randomly extract a sample graph from the training set And generate a random binary mask To construct the corresponding input graph Generate a random Gaussian noise Input noise into the mapping network f to generate a random style code Input the sample image e into the style network to generate the corresponding sample style code The input graph r i , a binary mask m, and a random style encoding Mixed style encoding mixed with sample style encoding w+ Input into generator G to get the repair result
[0063] like Figure 2 As shown, given the i-th training sample a i , random sample graph Random binary mask Next, input the graph The construction formula is as follows:
[0064] r i =a i ⊙(1-m),
[0065] The value range of each scalar in m is [0,1]. A value of 0 indicates a non-masked area, and a value of 1 indicates a masked area. represents a tensor whose values are all 1, and ⊙ is the Hadamard Product. Through the broadcast mechanism, i In the case of m, the information of the pixels corresponding to the masked area will be erased and represented as 0. In the case of random style encoding The calculation formula is as follows:
[0066]
[0067] in, Represents the extended stochastic style latent space. Sample style encoding The calculation formula is as follows:
[0068]
[0069] Where w+∈W+, W+ represents the style latent space, and T represents the subscript index set of the corresponding style layer. Then, the random style is encoded and sample style encoding Perform style mixing to obtain the fused style encoding The formula is as follows:
[0070]
[0071] Among them, φ is a binary label vector, which is used to indicate which style vector will be fused. The calculation formula is as follows:
[0072]
[0073] Among them, the generator G contains the encoder G en and decoder G de , a multi-style encoding method is introduced to combine random style encoding The sample style encoding w+ and the global style vector c are used to obtain the face restoration result containing the sample attributes. The calculation formula of the corresponding style representation vector is:
[0074]
[0075] Among them, the style representation vector v idx It will be used for modulation and demodulation later. represents the learnable affine transformation corresponding to the idx-th style layer. c represents the encoder G in the generator G. de For input r i and m to encode the global style vector (for modulation, demodulation, and affine transformation, please refer to StyleGAN and StyleGANv2).
[0076] Step S4: Combine the generated adversarial loss L adv , spatial variant-aware loss L LPIPS , identity loss L id , and attribute consistency loss L attri The loss value is calculated. Back propagation is performed based on the loss value, and the parameters of the mapping network f, the generator network G, and the discriminator network D are updated and adjusted using the stochastic gradient descent method.
[0077] like Figure 2 As shown in (right), the present invention first proposes a spatial variant gradient layer (SVGL), which is defined as:
[0078] P(x,M x ) = x,
[0079]
[0080] Where x represents the input feature, which can be an image or a feature map. x Represents the spatial weight mask. The above formula defines the forward propagation calculation and the partial derivative formula of the gradient spatial variation layer P(·,·) for back propagation. In the forward calculation, P(·,·) does not change any input information, but introduces the corresponding spatial weight mask during back propagation to adjust the spatial constraints of each output pixel according to the spatial relationship.
[0081] Next, the present invention proposes a spatial variant perception loss L LPIPS , the formula is as follows:
[0082]
[0083] where F(·) is a pre-trained feature extractor such as the VGG network. is a confidence weight mask (confidence weight mask, please refer to: Y.Wang, X.Tao, X.Qi, Xi.Shen, and J.Jia, ``Image inpainting via generative multi-columnconvolutional neural networks, "in Proc.NeurIPS, 2018, pp.331-340.) after linear function normalization (Min-Max scaling). The value range of each scalar is [0,1]. The value is close to 1, and the higher the weight is. When the original image a i When it is equal to the sample e, there is no contradiction in the constraints guided by the facial attributes, so the perceptual loss (LPIPS loss, please refer to: R.Zhang, P.Isola, AAEfros, E.Shechtman, and O.Wang, ``The unreasonable effectiveness of deepfeatures as a perceptual metric,” in Proc.CVPR, 2018, pp.586-595.) can be used directly. When the original image a i When it is not equal to the sample e, the mask boundary can be constrained, and the output in the middle of the mask can be relaxed to force the repair result. The pixels around the mask and the original image a i There is a higher perceived similarity.
[0084] The property consistent loss L of the present invention attri As shown below:
[0085]
[0086] in, It is the restored image output by the style network E The style code obtained by encoding, are the pre-trained parameters of the style network E. is the reverse weight mask, and its calculation formula is:
[0087]
[0088] That is, based on the confidence weight mask, the inverse weight mask is calculated. i When it is equal to sample e, directly repair the result Encode and calculate each Vector Sum The mean square error loss of the vector. i When it is not equal to the sample e, it is necessary to combine the inverse weight mask to encourage the area far away from the mask boundary to present the attributes of the sample face while maintaining boundary consistency. Generate adversarial loss L adv (GAN loss) Please refer to StyleGAN, identity loss L id (identity loss)Please refer to pSp.
[0089] Will fight against the loss L adv , spatial variant-aware loss L LPIPS , identity loss L id , and attribute consistency loss L attri By performing common constraints, the final objective function of the method of the present invention is obtained, and its formula is:
[0090]
[0091] where θ g ,θ f ,θ d and They represent the parameters of the generator network G, the mapping network f, the discriminator network D, and the style network E. The corresponding λ id , LPIPS , attri Respectively represent the weights of each loss function, which are set to λ in the present invention. id =0.1,λ LPIPS =0.5,λ attri = 0.1. By performing maximum and minimum training on the mapping network f, the pre-trained style network E, the generator network G, and the discriminator network D, the parameter training and optimization of the mapping network f, the generator network G, and the discriminator network D are achieved. The formula is as follows:
[0092]
[0093] Among them, The parameters of the style network are not involved in training. During the inference process, only the mapping network f, style network E and generator network G are needed to perform sample-guided diverse image restoration. The discriminator network D is used to distinguish between real and fake samples during training, and through the generative adversarial approach, the model can learn the distribution of real samples and improve the restoration quality.
[0094] Back propagation is performed according to the loss value, and the parameters of the mapping network f, the generator network G, and the discriminator network D at the qth iteration are adjusted using the stochastic gradient descent method. and To update and adjust, the specific steps are:
[0095] According to the final objective function, the chain derivation rule is used to adjust the parameters of the mapping network f, the generator network G, and the discriminator network D at the qth iteration. and Calculate the gradient and use the stochastic gradient descent method to reduce the corresponding network parameters and To update, the formula is as follows:
[0096]
[0097] Among them, η represents the learning rate in the hyperparameter; and They represent the gradients of the parameters of the corresponding mapping network f, generator network G, and discriminator network D at the qth iteration, respectively, and are obtained by the chain rule.
[0098] Step S5: Using test set I test The method model of the present invention is evaluated. If the restoration quality index (FID) of the model in the test image is the highest at present, the model parameters are saved and It represents the optimal model of the current mapping network f, generator network G and discriminator network D. It is judged whether the training has reached the maximum number of iterations n. If it has reached the maximum number of iterations n, the training phase ends and jumps to step S6, otherwise jumps to step S3.
[0099] Using test set I test The method model of the present invention is evaluated, and the specific steps are:
[0100] First, the test set Copy and randomly shuffle to get a set of sample images At the same time, a fixed set of binary masks is generated Traverse the data, and each traversal starts from the test set I test 、Sample Atlas I exemplar and binary mask set I mask Take a sample from each of them to get the test input picture a j 、Sample picture b j and binary mask image m j , calculate these three samples using the calculation method described in step S3. Obtain the corresponding repair result set For the test set I test and the repair result set Y testCalculate the Fr′echetinception distance indicator. (FID, please refer to: M.Heusel, H.Ramsauer, T.Unterthiner, B.Nessler, and S.Hochreiter, ``GANs trained by a two time-scale update ruleconverge to a local nash equilibrium,” in Proc.NIPS, 2017, pp.6626-6637.). Here we define the optimal model network parameters and The FID indicator is fid best , if fid q >fid best , then let fid best =fid q It should be noted that the sample atlas I exemplar and binary mask set I mask It is generated only once and used in subsequent tests to ensure that the conditions are the same for each test.
[0101] Step S6, obtaining the optimal model parameters of the mapping network f, the generator network G, and the discriminator network D, and using the mapping network f, the style network E, and the generator network G for sample-guided diversified restoration of face images.
[0102] The implementation of the embodiments of the present invention has the following beneficial effects:
[0103] 1. Compared with the existing image restoration methods, the present invention can use the attribute information of a given sample image to guide face restoration, and ensure the quality and diversity of face restoration, ensure that the restoration area can clearly show the attribute information in the sample face image, and has good value and prospects in practical applications;
[0104] 2. The present invention proposes a diversified facial image restoration method based on sample guidance. Based on the generative adversarial network architecture, this method first introduces a self-supervised attribute consistency learning method, which can help the network learn to embed a clear mapping of sample style and facial attributes; secondly, a gradient space variation layer is proposed, which can use the relationship between the spatial positions of pixels to adjust the constraints of the loss function on the network parameters in the back-propagation stage. The gradient space variation layer is a general constraint guidance layer. This module has strong versatility and can be applied to any loss function, so that the loss function can dynamically adjust the gradient penalty of the corresponding parameters according to the spatial relationship of the pixels.
[0105] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, CD-ROM, etc.
[0106] The above disclosure is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A sample-guided diversified facial image restoration method, characterized in that: The method comprises at least the following steps: Step S1, obtaining a training set and a test set; wherein the training set is used to train and adjust the model parameters, and the test set is used to evaluate and test the model; Step S2, constructing a network model for realizing diversified restoration of face images based on sample guidance; the network model at least includes a mapping network, a style network, a generator network and a discriminator network; wherein the mapping network is used to generate random style codes; the style network is used to map the sample image into the sample style code; the output information of the mapping network and the style network is mixed and input to the generator network; the generator network is used to restore the input image by combining the random style code, the sample style code and the global style code to obtain a face restoration result containing the sample attributes; the discriminator network is used to generate adversarial training to assist the generator network in learning the data distribution of real samples; Initialize all networks; the style network is initialized with the parameters of the pre-trained model, and the parameters of the style network are not updated during the network model training process; Step S3, in any iterative training process, first generate and train a binary mask of the same size as the image, then erase the image information at the corresponding position in the mask, and input the mask and the image into the generator network; at the same time, the mapping network randomly samples the Gaussian distribution and maps the Gaussian distribution into a random style code; the style network maps the given sample image into a sample style code; the encoder in the generator network encodes the input image to obtain the corresponding global style vector and multi-scale feature map, and embeds the random style code, the sample style code and the global style vector into the decoder in the generator network to control the generation of facial attributes; at the same time, the multi-scale features in the encoder of the generator network are fused with the feature maps of the corresponding scales in the decoder of the generator network through jump connections; through the above calculations, the generator network outputs a repaired face image; Step S4: Combine the generated adversarial loss L adv , spatial variant-aware loss L LPIPS , identity loss L id , and attribute consistency loss L attri Calculate the loss value; perform back propagation based on the loss value, and use the stochastic gradient descent method to update and adjust the parameters of the mapping network, generator network, and discriminator network; Step S5, using the test set to evaluate the network model constructed by the above method; if the restoration quality index of the network model in the test image is the highest currently, then save the model parameters; determine whether the training is completed, if it is completed, jump to step S6, otherwise jump to step S3; Step S6, obtaining the optimal model parameters of the mapping network, the generator network and the discriminator network, and using the mapping network, the generator network and the pre-trained style network for sample-guided diverse face restoration.
2. The example-guided diversified facial image restoration method according to claim 1, characterized in that: In step S1, both the training set and the test set are image sets, and image labels are not required.
3. The example-guided diversified facial image restoration method according to claim 1, characterized in that: In step S2, the mapping network is a fully connected neural network without an activation function; the style network is a pre-trained style encoding network for encoding facial attributes into the corresponding sample style space; the generator network includes an encoder G en and decoder G de ; The discriminator network is a binary classification network used to distinguish real samples from restored samples.
4. The example-guided diversified facial image restoration method according to claim 1, characterized in that: In step S3, the calculation formula for combining random style coding, sample style coding and global style coding to repair the input image to obtain a representation code of a face repair result containing sample attributes is: in, represents the mixed style vector, idx represents the subscript of the style layer in the network, and represents the idx-th style layer in the generator network; w idx is the corresponding sample style vector, is the corresponding random style vector, φ idx v is the introduced binary label vector, which is used to indicate which style vector is used in the idx layer; idx Indicates the final multi-style representation code, which will be used for subsequent style modulation and demodulation. represents the learnable radiometric transformation function corresponding to the idx-th style layer; c represents the global style vector that encodes the input in the generator network.
5. The example-guided diversified facial image restoration method according to claim 4, characterized in that: In step S4, a gradient spatial variation layer is used to construct a loss function, which is defined as: P(x,M x )=x, Among them, x represents the input feature, which is an image or a feature map; M x represents the spatial weight mask corresponding to the input; the above formula defines the forward propagation calculation and the partial derivative formula of the back propagation of the gradient spatial variation layer P(·,·); in the forward calculation, P(·,·) does not change any information of the input, but in the back propagation, the corresponding spatial weight mask M x Introduce it to realize the output adjustment constraints of different position coordinates.
6. The example-guided diversified facial image restoration method according to claim 5, characterized in that: In step S4, the spatial variant perception loss L LPIPS The formula is as follows: where F(·) is the pre-trained feature extractor, is the repair result finally output by the generator network; a i represents the corresponding original image; e represents the corresponding sample image; m w is the confidence weight mask normalized by a linear function, and its value range is [0,1]; when the original image a i When the original image a is equal to the sample e, there is no contradiction in the constraints guided by the face attributes, and the perceptual loss is used directly; i When it is not equal to the sample e, the mask boundary is constrained, and the output in the middle of the mask is relaxed, thus forcing the pixels around the mask to have a higher perceptual similarity with the original image.
7. The example-guided diversified facial image restoration method according to claim 6, characterized in that: In step S4, the attribute consistency loss L attri As shown below: in, The restored image is output by the style network The style code obtained by encoding, The restored image is output by the style network The style vector obtained by encoding; are the pre-trained parameters of the style network; is the reverse weight mask; when the original image a i When it is equal to sample e, directly Encode and calculate each stylelayer and The mean square error loss; when the original image a i When it is not equal to the sample e, it is necessary to combine the inverse weight mask to encourage the areas far away from the mask boundary to present the attributes of the sample face while maintaining boundary consistency.
8. The example-guided diversified facial image restoration method according to claim 1, characterized in that: In step S4, the adversarial loss L adv , spatial variant-aware loss L LPIPS , identity loss L id , and attribute consistency loss L attri By performing common constraints, the formula of the final objective function is obtained as follows: Among them, θ g ,θ f ,θ d and Represent the parameters of the generator network, mapping network, discriminator network and style network respectively; λ id , LPIPS and λ attri They are the corresponding identity losses L id , spatial variant-aware loss L LPIPS And attribute consistency loss L attri The weight parameters of the network are: By performing joint maximum minimization training on the mapping network, the pre-trained style network, the generator network, and the discriminator network, the formula is as follows: in, It is the parameter of the style network and does not participate in training. During the inference process, the mapping network, style network and generator network are used to perform sample-guided diverse face image restoration. The discriminator network is used to distinguish true and false samples during training, and the model can learn the distribution of real samples by generating adversarial methods.