Face anonymization method based on stylegan encoder and skin color matching generative adversarial network

Abstract

The application discloses a face anonymization method based on a StyleGAN encoder and a skin color matching generative adversarial network, which comprises the following steps: converting an original face image into a semantic segmentation image by using a pre-trained face semantic segmentation model BiSeNet; extracting pose latent representation of the semantic segmentation image by using a translation encoder, and extracting identity latent representation information of a reference face image by using an identity encoder; fusing the pose latent representation information and the identity latent representation information to obtain a new latent representation which maintains the original face pose and is different from the original face identity; converting the injected new latent representation into a preliminary anonymous face image by using a StyleGAN generator; and fusing the generated anonymous face image and the background of the original image by using a skin color matching generative adversarial network to realize final face anonymization. The application improves the anonymity of the anonymous picture, face detection performance, image generation quality, diversity, face expression recognition performance and picture integrity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face image anonymization, and particularly relates to a face anonymization method based on a StyleGAN encoder and a skin color matching generative adversarial network. BACKGROUND

[0002] In the current digital era, with the development of artificial intelligence technology and deep learning, face recognition technology has been widely applied in many fields, such as security monitoring, identity authentication, intelligent devices and social media. However, the accompanying privacy problem has also attracted widespread attention, especially the misuse of face data may lead to personal privacy leakage. Therefore, face anonymization method has important research value and development prospect.

[0003] Traditional face anonymization methods mainly include pixelization, blurring and occlusion technology, etc. These methods usually significantly reduce the usability or aesthetics of the image, affecting the user experience. At the same time, with the progress of face recognition technology, traditional anonymization methods gradually cannot effectively prevent face recognition systems from recognizing and restoring. Therefore, anonymization technology based on deep learning and generative adversarial network (GAN) emerges as the times require. The generative adversarial network can generate realistic face images through the adversarial training of the generator and the discriminator, such as CIAGAN, which mixes landmark identity and target identity, and uses a twin identity-guided discriminator for anonymous face, this method is suitable for low-resolution face anonymization, but the generated anonymous face is not realistic enough, and there may be a bad face; DeepPrivacy automatically anonymizes the face in the image while preserving the original data distribution, but there is a uniqueness problem, mapping each identity to a unique fake identity, not allowing different images to be generated, in addition, there is a problem of poor preservation of non-private information in the original image; GANonymization is a new face anonymization framework based on pix2pix, which realizes the anonymization of the original face by mapping the face scatter plot to the face, and preserves the pose and expression information of the face. Although this method solves the problem of loss of non-identity information of the face in the previous anonymization method, it does not solve the problem of changing the background of the generated anonymous face at the same time.

[0004] In summary, the existing face anonymization methods based on generative adversarial network mainly have the problems of low quality of generated face images, poor diversity, poor non-private information preservation ability, and background loss. Therefore, it is necessary to design a face anonymization method that can better preserve the face detection and facial expression recognition performance while ensuring anonymity, has good integrity, and can preserve the background information unrelated to the face. SUMMARY

[0005] The present application aims to provide a face anonymization method which can confuse human eyes and machines, has strong anonymity, high face detection rate, high image generation quality, diverse background and strong face expression preservation capability.

[0006] The technical solution for achieving the object of the present application is a face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network, comprising the following steps:

[0007] Step 1: using a pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map;

[0008] Step 2: using a translation encoder to extract the pose latent representation of the semantic segmentation map;

[0009] Step 3: using an identity encoder to extract the identity latent representation information of the reference face image;

[0010] Step 4: fusing the pose latent representation information and the identity latent representation information to obtain a new latent representation which maintains the original face pose and has a distinguished identity from the original face;

[0011] Step 5: using a StyleGAN generator to convert the injected new latent representation into a preliminary anonymous face image;

[0012] Step 6: using a skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image and the background of the original image to realize the final face anonymization.

[0013] Further, the semantic segmentation map in step 1 includes 19 parts, namely background, skin, nose, glasses, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, mouth, upper lip, lower lip, hair, hat, earring, neck, necklace and clothes.

[0014] Further, the translation encoder in step 2 adopts a three-level feature pyramid encoder of pSp architecture, the pyramid encoder adopts IR-SE50 as the backbone, and the mapping module is composed of 18 small convolutional networks style2map, the small convolutional networks numbered 1-3 are used to extract "coarse" level details, the small convolutional networks numbered 4-7 are used to extract "medium" level details, and the small convolutional networks numbered 8-18 are used to extract "fine" level details.

[0015] Further, in step 2, the pose latent representation of the semantic segmentation map is extracted using the translation encoder, specifically as follows:

[0016] Step 2.1: using the CelebaAMask-HQ dataset as the training dataset, and using the method of step 1 to extract the face semantic segmentation map of the training data;

[0017] Step 2.2, extracting pose latent representation by the translation encoder to be trained;

[0018] Step 2.3, passing the complete pose latent representation into the pre-trained StyleGAN generator, and generating a new face image with the same pose semantics as the original image by the StyleGAN generator;

[0019] Step 2.4, fixing the parameters of the StyleGAN generator, and training the translation encoder by the total loss function Training the translation encoder:

[0020]

[0021] wherein λ1, λ2, λ3 are constants, is a pixel-level loss function, is a perceptual loss function, is a regularization term loss function;

[0022] Step 2.5, extracting the pose latent representation of the semantic segmentation map by the trained translation encoder.

[0023] Further, in step 2.4, the total loss function for training the translation encoder is The calculation method is as follows:

[0024] Let the latent representation vector w extracted by the translation encoder after inputting the original face image x be trans

[0025] w trans = E trans (seg(x))

[0026] wherein seg represents the face semantic segmentation model BiSeNet, E trans represents the translation encoder;

[0027] Then, the latent representation vector w is passed through the StyleGAN generator to obtain the generated result y trans trans

[0028]

[0029] wherein pSp represents the overall architecture of the translation encoder plus the StyleGAN generator, G stylegan represents the StyleGAN generator, w trans represents the latent representation vector extracted by the translation encoder, represents the average value of w trans

[0030] ​​​​The core of model training is the selection of loss function. The training loss function of the translation encoder is a weighted combination of multiple objectives, which is as follows:

[0031] First, the pixel-level loss is used to optimize the encoder to generate a new image similar to the original image in pixels. The pixel-level loss is defined as:

[0032]

[0033] Then, the perceptual loss LPIPS is used to provide better image quality than other perceptual losses by learning perceptual similarity. The perceptual loss LPIPS function is defined as:

[0034]

[0035] where F represents a perceptual feature extractor.

[0036] Finally, a regularization term is used to encourage the latent semantic representation vector output by the translation encoder to be closer to the average latent representation vector. The regularization term loss function is defined as:

[0037]

[0038] In summary, the total loss function for training the translation encoder is defined as:

[0039]

[0040] where λ1, λ2, λ3 are constants used to define the weighting values of the loss function.

[0041] Further, the identity encoder is used in step 3 to extract the identity latent representation information of the reference face image, as follows:

[0042] Step 3.1, use the method of step 1 to extract the semantic segmentation map of the face of the training data;

[0043] Step 3.2, use the translation encoder to encode the semantic segmentation map of the input original image to obtain the semantic latent representation;

[0044] Step 3.3, use the identity encoder to encode the target image to obtain the identity latent representation;

[0045] Step 3.4, fuse the semantic latent representation and the identity latent representation by concatenation to obtain the fused representation;

[0046] Step 3.5, input the fusion representation into the StyleGAN generator to generate an anonymous face image with the original face pose and different from the original face identity;

[0047] Step 3.6, fix the parameters of StyleGAN and the translation encoder, and use the total loss function of training the identity encoder Training the identity encoder:

[0048]

[0049] where λ3, λ4, λ5, λ6 are weight constants, λ3 is consistent with the previous, represents the identity loss, which is used to encourage the identity encoder to effectively encode the identity information; x, t represents the original face image and the reference target face image;

[0050] Step 3.7, using the trained identity encoder to extract the identity latent representation information of the reference face image.

[0051] Further, the total loss function of training the identity encoder in step 3.6 The calculation method is as follows:

[0052] Before training the identity encoder, the translation encoder needs to be pre-trained, and the training process requires the two encoders to work together: the translation encoder encodes the semantic segmentation map of the input original image to obtain the semantic latent representation, and the identity encoder encodes the target image to obtain the identity latent representation; Then fuse the two representations and input them into the StyleGAN generator to generate an anonymous face image; The fusion of the two latent representations is realized by concatenation, and the latent representation vectors of layers 1-8 are provided by the translation encoder, and the latent representation vectors of layers 9-18 are provided by the identity encoder;

[0053] Let the reference target image be t, then the output identity latent representation vector of the identity encoder w id is represented as:

[0054] w id = E id (t)

[0055] where E id represents the identity encoder;

[0056] The fused latent representation vector of w mix is represented as:

[0057]

[0058] where subscript i represents the i-th component of the latent representation vector;

[0059] In training the identity encoder, the pre-trained translation encoder needs to be fixed, only updating the parameters of the identity encoder;

[0060] The total loss function for training the identity encoder is also a weighted combination of multiple objectives, as follows:

[0061] First, the pixel-level loss, LPIPS loss, and regularization loss:

[0062]

[0063] where y mix represents the anonymous face image generated by the fused latent representation vector;

[0064] Then, to encourage the identity encoder to encode identity information, an identity loss is introduced:

[0065]

[0066] where R represents the pre-trained face recognition model;

[0067] In summary, the total loss function for training the identity encoder is defined as:

[0068]

[0069] where λ3, λ4, λ5, λ6 are constants used to define the weighting values of the loss function; compared with the translation encoder training, the weights of the loss and LPIPS loss are adjusted, corresponding to λ4 and λ5 respectively, the regularization term loss and the weight constant do not change; λ6 is the weight constant of the unit loss.

[0070] Further, in step 6, the generated anonymous face image and the background of the original image are fused using the skin color matching generative adversarial network SCMGAN to realize the final face anonymization, as follows:

[0071] Step 6.1, use the method of step 1 to extract the semantic segmentation map of the face of the training data;

[0072] Step 6.2, use the Gaussian blur method to obtain the blurred version of the original face image in the training data set;

[0073] Step 6.3, change the contrast, brightness, saturation and hue of the face part of the original face image to obtain the to-be-matched face image that is not coordinated with the background;

[0074] Step 6.4, use the total loss function to train SCMGAN, the training process is decoupled from the training of the encoder:

[0075]

[0076] wherein represents an adversarial loss, represents a zero-centered gradient penalty loss, represents a pixel-level loss, represents a VGG-based perceptual loss, and λ7 and λ8 represent weight constants;

[0077] Step 6.5, input the images in steps 6.1-6.3 into the trained SCMGAN to obtain a face image with completed skin color matching.

[0078] Further, the total loss function for training the SCMGAN in step 6.4 is The calculation method is specifically as follows:

[0079] The SCMGAN has three inputs: (1) a spliced image of an anonymous face image and an original image background; (2) a semantic segmentation image of the face of the original face image; and (3) a blurred original face image. The training of the SCMGAN can be completed through a skin color matching proxy task;

[0080] First, the adversarial loss is calculated, which is specifically as follows:

[0081]

[0082] wherein G SCM is the generator of the SCMGAN, D SCM is the discriminator, b is the label of the real data, a is the label of the false data, x is the real data, and i is the input of the generator;

[0083] Then, the zero-centered gradient penalty loss is used to stabilize the model training:

[0084]

[0085] The pixel-level loss is used to match the color of the original face and the generated face:

[0086]

[0087] wherein I represents the input of the generator;

[0088] Finally, the VGG-based perceptual loss function is used to determine the feature difference between the target and the generated output, and the VGG loss is defined as follows:

[0089]

[0090] wherein μ icorresponding to the weight of each layer feature, C p ,W p ,H p respectively corresponding to the channel, width and height dimensions of the image, VGG represents the VGG-19 model;

[0091] In summary, the total loss function required for training SCMGAN is:

[0092]

[0093] Where λ7, λ8 are weight constants.

[0094] A face anonymization device based on a StyleGAN encoder and a skin color matching generative adversarial network, the device is used to implement the face anonymization method based on the StyleGAN encoder and the skin color matching generative adversarial network, the device comprises, wherein:

[0095] A semantic segmentation module is used to convert the original face image into a semantic segmentation image by using a pre-trained face semantic segmentation model BiSeNet.

[0096] A pose latent representation extraction module is used to extract the pose latent representation of the semantic segmentation image by using a translation encoder.

[0097] An identity latent representation extraction module is used to extract the identity latent representation information of the reference face image by using an identity encoder.

[0098] A fusion module is used to fuse the pose latent representation information and the identity latent representation information to obtain a new latent representation which maintains the original face pose and has a distinguished identity from the original face.

[0099] A StyleGAN anonymization generator module is used to convert the injected new latent representation into a preliminary anonymous face image by using a StyleGAN generator.

[0100] An anonymous face skin color matching module is used to fuse the generated anonymous face image and the background of the original image by using a skin color matching generative adversarial network SCMGAN to realize the final face anonymization.

[0101] Compared with the prior art, the present application has the following advantages: (1) the translation encoder and the identity encoder are used to extract the semantic latent representation and the identity latent representation respectively, which not only retains the background information, the face semantic segmentation information, the facial expression information and other information irrelevant to the identity, but also ensures the quality and anonymity of the image generation; (2) the anonymous face not only confuses machines, but also has good human eye confusion effect, and has good anonymity effect; (3) the face detection and facial expression recognition performance are retained, the background information irrelevant to the face is retained, and good integrity is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0102] Figure 1 This is a flowchart illustrating a face anonymization method based on a StyleGAN encoder and a skin color matching generative adversarial network according to the present invention.

[0103] Figure 2 This is a flowchart illustrating the overall process of the present invention.

[0104] Figure 3 This is a structural block diagram of the translation encoder and identity encoder in this invention.

[0105] Figure 4 This is a structural block diagram of SCMGAN in this invention.

[0106] Figure 5 This is an anonymization effect diagram of the method of the present invention and existing methods on the FFHQ dataset in the embodiments of the present invention.

[0107] Figure 6 The image shows the facial expression preservation effect of the method of the present invention and the existing method on the Affect dataset in the embodiments of the present invention. Detailed Implementation

[0108] like Figure 1 , Figure 2 As shown, the present invention provides a face anonymization method based on a StyleGAN encoder and a skin color matching generative adversarial network, comprising the following steps:

[0109] Step 1: Use the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map;

[0110] Step 2: Extract the pose latent representation of the semantic segmentation map using a translation encoder;

[0111] Step 3: Use an identity encoder to extract the latent identity representation information of the reference face image;

[0112] Step 4: Integrate pose latent representation information and identity latent representation information to obtain a new latent representation that retains the original face pose while having a different identity from the original face.

[0113] Step 5: Using the StyleGAN generator, the injected new latent representations are transformed into preliminary anonymized face images;

[0114] Step 6: Use the skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image with the background of the original image to achieve the final face anonymization.

[0115] As a specific example, in step 1, the pre-trained face semantic segmentation model BiSeNet is used to convert the original face image into a semantic segmentation map;

[0116] The semantic segmentation map includes 19 parts, respectively background, skin, nose, glasses, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, mouth, upper lip, lower lip, hair, hat, earring, neck, necklace and clothes.

[0117] As a specific example, in step 2, the pose latent representation of the semantic segmentation map is extracted by using a translation encoder;

[0118] As shown in Figure 3 The translation encoder adopts a three-level feature pyramid encoder of pSp architecture, the pyramid encoder adopts IR-SE50 as the backbone, the mapping module is composed of 18 small convolutional networks, the small convolutional networks numbered 1-3 are used to extract "coarse" level details, the small convolutional networks numbered 4-7 are used to extract "medium" level details, and the small convolutional networks numbered 4-7 8-18 are used to extract "fine" level details.

[0119] As a specific example, in step 2, the pose latent representation of the semantic segmentation map is extracted by using a translation encoder, specifically as follows:

[0120] Step 2.1, using CelebaAMask-HQ dataset as training dataset, using the method of step 1 to extract the face semantic segmentation map of the training data;

[0121] Step 2.2, using the translation encoder to be trained to extract the pose latent representation;

[0122] Step 2.3, passing the complete pose latent representation into the pre-trained StyleGAN generator, and using the StyleGAN generator to generate a new face image with the same pose semantics as the original image;

[0123] Step 2.4, fixing the parameters of the StyleGAN generator, using the total loss function Train the translation encoder:

[0124]

[0125] wherein λ1, λ2, λ3 are constants, is a pixel-level loss function, is a perceptual loss function, is a regularization term loss function;

[0126] Step 2.5, using the trained translation encoder to extract the pose latent representation of the semantic segmentation map.

[0127] As a specific example, in step 2.4, the total loss function of training the translation encoder is The calculation method is specifically as follows:

[0128] The input image x is set to extract the latent representation vector w through the translation encoder trans For:

[0129] w trans =E trans (seg(x))

[0130] Wherein seg represents the face semantic segmentation model BiSeNet, E trans represents the translation encoder.

[0131] Then the latent representation vector w trans is obtained through the StyleGAN generator to obtain the generated result y trans For:

[0132]

[0133] Wherein pSp represents the overall architecture of the translation encoder plus the StyleGAN generator, G stylegan represents the StyleGAN generator, E trans represents the translation encoder, w trsns represents the latent representation vector extracted by the translation encoder, represents the average value of w trans ;

[0134] The core of model training is the selection of loss function, and the training loss function of the translation encoder is a weighted combination of multiple objectives, which is specifically as follows:

[0135] Firstly, the pixel-level loss is used to optimize the encoder and generate a new image similar to the original image in pixels, and the pixel-level loss is defined as:

[0136]

[0137] Then, the perceptual loss LPIPS is used to provide better image quality than other perceptual losses by learning perceptual similarity, and the perceptual loss LPIPS function is defined as:

[0138]

[0139] Wherein F represents a perceptual feature extractor;

[0140] Finally, a regularization term is used to promote the latent semantic representation vector output by the translation encoder to be closer to the average latent representation vector, and the regularization term loss function is defined as:

[0141]

[0142] In summary, the total loss function for training the translation encoder is defined as:

[0143]

[0144] where λ1, λ2, λ3 are constants used to define the weighted values of the loss function.

[0145] As a specific example, in step 3, the identity encoder is used to extract the identity latent representation information of the reference face image.

[0146] The identity encoder has a similar model structure as the translation encoder, but it focuses on capturing identity information while ignoring various information related to semantic segmentation mapping.

[0147] As a specific example, in step 3, the identity encoder is used to extract the identity latent representation information of the reference face image, which is as follows:

[0148] Step 3.1, use the method of step 1 to extract the face semantic segmentation map of the training data;

[0149] Step 3.2, use the translation encoder to encode the semantic segmentation map of the input original image to obtain the semantic latent representation;

[0150] Step 3.3, use the identity encoder to encode the target image to obtain the identity latent representation;

[0151] Step 3.4, fuse the semantic latent representation and the identity latent representation by concatenating and splicing to obtain the fused representation;

[0152] Step 3.5, input the fused representation into the StyleGAN generator to generate an anonymous face image with the original face pose but different from the original face identity;

[0153] Step 3.6, fix the parameters of StyleGAN and translation encoder, and use the total loss function for training the identity encoder Train the identity encoder:

[0154]

[0155] where λ3, λ4, λ5, λ6 are constants, λ3 is consistent with requirement 5, is the identity loss calculated for the face recognition model, x is the original face image, and t is the target image.

[0156] Step 3.7, use the trained identity encoder to extract the identity latent representation information of the reference face image.

[0157] As a specific example, the total loss function for training the identity encoder described in step 3.6 The calculation method is as follows:

[0158] Before training the identity encoder, the translation encoder needs to be pre-trained, and the training process requires the cooperation of the two encoders: the translation encoder encodes the semantic segmentation map of the input original image to obtain a semantic latent representation, and the identity encoder encodes the target image to obtain an identity latent representation; Then fuse the two representations and input them into the StyleGAN generator to generate an anonymous face image; The fusion of the two latent representations is realized by concatenation splicing, and the latent representation vectors of layers 1-8 are provided by the translation encoder, while the latent representation vectors of layers 9-18 are provided by the identity encoder;

[0159] Let the reference target image be t, then the identity encoder w id The output identity latent representation vector is represented as:

[0160] w id = E id (t)

[0161] Where E id represents the identity encoder.

[0162] The fused latent representation vector w mix is represented as:

[0163]

[0164] Where subscript i represents the i-th dimension component of the latent representation vector;

[0165] When training the identity encoder, the pre-trained translation encoder needs to be fixed, and only the parameters of the identity encoder are updated;

[0166] The total loss function for training the identity encoder is also a weighted combination of multiple objectives, which is as follows:

[0167] First, the pixel-level loss, LPIPS loss and regularization loss in the pre-training stage of the translation encoder are retained:

[0168]

[0169] Where y mix represents the anonymous face image generated by the fused latent representation vector.

[0170] Then, in order to encourage the identity encoder to encode identity information, an identity loss is introduced:

[0171]

[0172] wherein R represents a pre-trained face recognition model.

[0173] In summary, the total loss function of training the identity encoder is defined as:

[0174]

[0175] wherein λ3, λ4, λ5, λ6 are constants for defining the weight values of the loss function; compared with the translation encoder training, the weight values of the loss and LPIPS loss are only slightly adjusted, corresponding to λ4 and λ5 respectively, and the regularization term loss and its weight constant do not change; λ6 is the weight constant of the identity loss.

[0176] As a specific example, in step 4, the pose latent representation information and the identity latent representation information are fused to obtain a new latent representation which maintains the original face pose and has a distinguished identity from the original face;

[0177] The latent representation vector is fused as w mix = w

[0178]

[0179] wherein subscript i represents the i-th dimensional component of the latent representation vector;

[0180] As a specific example, in step 5, the injected new latent representation is converted into a preliminary anonymous face image by the StyleGAN generator;

[0181] The preliminary anonymous face image y mix is:

[0182]

[0183] As a specific example, in step 6, the generated anonymous face image and the background of the original image are fused by using the skin color matching generative adversarial network SCMGAN to realize the final face anonymization, as shown in Figure 4 , specifically as follows:

[0184] Step 6.1, the semantic segmentation map of the face part of the training data is extracted by the method of step 1;

[0185] Step 6.2, the blurred version of the original face image in the training data set is obtained by using the Gaussian blur method;

[0186] Step 6.3, the contrast, brightness, saturation and hue of the face part of the original face image are changed to obtain a to-be-matched face image which is not coordinated with the background;

[0187] Step 6.4. Total loss function for training SCMGAN Training SCMGAN, the training process is decoupled from the training of the encoder:

[0188]

[0189] wherein represents the adversarial loss, represents the zero-centered gradient penalty loss, represents the pixel-level loss, represents the VGG-based perceptual loss, and λ7 and λ8 represent weight constants.

[0190] Step 6.5. Input the images in steps 6.1-6.3 into the trained SCMGAN to obtain the completed skin color matching face image.

[0191] As a specific example, the total loss function for training SCMGAN described in step 6.4 is The calculation method is as follows:

[0192] SCMGAN has three inputs: (1) the spliced image of the anonymous face image and the original image background; (2) the semantic segmentation map of the original face image face; (3) the blurred original face image, and the training of SCMGAN can be completed through the skin color matching proxy task;

[0193] First, calculate the adversarial loss, as follows:

[0194]

[0195] wherein G SCM is the generator of SCMGAN, D SCM is the discriminator, b is the label of real data, a is the label of false data, x is the real data, and i is the input of the generator;

[0196] Then use the zero-centered gradient penalty loss to stabilize the model training:

[0197]

[0198] And use the pixel-level loss to match the color of the original face and the generated face:

[0199]

[0200] wherein I represents the input of the generator;

[0201] Finally, the VGG-based perceptual loss function is used to determine the feature difference between the target and the generated output, and the VGG loss is defined as follows:

[0202]

[0203] wherein μ i corresponding to each layer feature, C p ,W p ,H p respectively correspond to the channel, width and height dimensions of the image, and VGG represents the VGG-19 model.

[0204] In summary, the total loss function required for training the SCMGAN is:

[0205]

[0206] wherein λ7, λ8 are weight constants.

[0207] The application also provides a face anonymization device based on a StyleGAN encoder and a skin color matching generative adversarial network, which is used to implement the face anonymization method based on the StyleGAN encoder and the skin color matching generative adversarial network.

[0208] A semantic segmentation module is configured to convert an original face image into a semantic segmentation image by using a pre-trained face semantic segmentation model BiSeNet.

[0209] A pose latent representation extraction module is configured to extract a pose latent representation of the semantic segmentation image by using a translation encoder.

[0210] An identity latent representation extraction module is configured to extract identity latent representation information of the reference face image by using an identity encoder.

[0211] A fusion module is configured to fuse the pose latent representation information and the identity latent representation information to obtain a new latent representation that maintains the original face pose and is different from the original face identity.

[0212] A StyleGAN anonymization generator module is configured to convert the injected new latent representation into a preliminary anonymous face image by using a StyleGAN generator.

[0213] An anonymous face skin color matching module is configured to fuse the generated anonymous face image and the background of the original image by using a skin color matching generative adversarial network SCMGAN to achieve face anonymization.

[0214] The application will be further described in detail below in combination with the accompanying drawings and specific embodiments.

[0215] Embodiment

[0216] In this embodiment, the FFHQ dataset is used, and 10,000 face images are randomly selected as a test set to test the method of the application. Figure 1、 Figure 2 As shown, the face anonymization method of the present application based on StyleGAN encoder and skin color matching generative adversarial network is used to anonymize 10,000 randomly selected face images in the FFHQ dataset, and the specific steps are as follows:

[0217] Step 1, obtain the FFHQ dataset, randomly select 10,000 face images as the test set, and preprocess the test data to crop them to 256x256. Use the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map, which includes 19 parts, i.e. background, skin, nose, glasses, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, mouth, upper lip, lower lip, hair, hat, earring, neck, necklace and clothes.

[0218] The original face image is blurred by a Gaussian blur algorithm to obtain a blurred original face.

[0219] Step 2, use the translation encoder to extract the pose latent representation of the semantic segmentation map; the trained translation encoder can well extract the pose and expression information in the semantic segmentation map, which is used to retain the pose and expression information of the original face in the anonymous face;

[0220] Step 3, use the identity encoder to extract the identity latent representation information of the reference face image; the trained identity encoder can well extract the identity information in the reference face image, which is used to provide reference information different from the original identity in the anonymous face generation process, and guide the generation of face anonymization;

[0221] Step 4, fuse the pose latent representation information and the identity latent representation information to obtain a new latent representation that retains the pose of the original face and has a different identity from the original face;

[0222] The position of fusion is observed by replacing dimensions. Replace certain dimensions of the latent representation vector obtained by the translation encoder with an average latent representation vector. Specifically, the first dimension is retained, the 1-2 dimensions are retained, the 1-3 dimensions are retained, and so on. Then, the replaced latent representation vector is input into the StyleGAN generator to obtain 18 different generation results. By observation, starting from the 3rd dimension, the generated image begins to show the original facial direction; starting from the 4th dimension, the generated image begins to be similar to the original facial hair; starting from the 5th dimension, the generated image begins to match the mouth shape of the original image; starting from the 7th dimension, the generated image begins to match the gender of the original image; starting from the 8th dimension, the generated image is very similar to the semantic segmentation map of the original face; after the 8th dimension, more subtle facial changes are observed. As can be seen, the first 8 dimensions are mainly used to maintain the posture information, and the latter dimensions are used for more subtle facial changes, so the identity is spliced after the 8th dimension;

[0223] Step 5, injecting the new latent representation into the StyleGAN generator to convert it into a preliminary anonymous face image;

[0224] Step 6, using SCMGAN to fuse the generated anonymous face image and the background of the original image to realize the final face anonymity.

[0225] The existing face detection models Dlib and SSH are used to detect the anonymous face image, and the detection rate is calculated. The existing face recognition models cosface and arcface are used to compare the similarity between the original face and the anonymous face, and the anonymity effect is evaluated. The original face image and the anonymous face image are input into the face recognition model to obtain the identity feature vector, the cosine distance of the two identity feature vectors is calculated, and a threshold of 0.4 is set. If it is lower than 0.4, it means that the anonymity is successful, otherwise the anonymity fails, and the anonymity failure rate is calculated. The anonymous face detection rate and anonymity are shown in Table 1, and the anonymity effect is shown in Figure 5 Table 1. Original represents the original face image, Pixelization 8by 8 and Pixelization 16by 16 represent different sizes of pixelization, Blur 9by 9 and Blur 17by 17 represent different sizes of blurring, CIA represents the anonymous face generated by CIAGAN, DP2 represents the anonymous face generated by DeepPrivacy2, and Ours represents the anonymous face generated by the present application.

[0226] Table 1 Comparison of anonymous face detection rate and anonymity of the method of the present application on the FFHQ test data set

[0227]

[0228] Different reference face images are used to generate different anonymous versions, and the diversity of the anonymous faces is calculated. The diversity is embodied by the LPIPS index, which is a deep learning-based image quality evaluation method that uses a pre-trained convolutional neural network to learn the perceptual difference of image features. The larger the value, the better the diversity of the anonymous face. The comparison results of the diversity of the anonymous face are shown in Table 2.

[0229] Table 2 Comparison of the diversity of the anonymous face of the method of the present application on the FFHQ test data set

[0230] Models LPIPS(↑) CIA 0.0257 Ours 0.0276

[0231] The image quality evaluation algorithm is used to evaluate the generation quality of the anonymous face. In order to evaluate the quality of the generated image, the FID, SSIM and PSNR indexes are mainly used. Among them, FID is an index used to evaluate the quality and diversity of the generation model. The smaller the value, the more diverse and higher the quality of the generated image. SSIM measures the similarity of two images, including brightness, contrast and structure. The closer the value to 1, the higher the image quality; the closer the value to 0, the lower the image quality. PSNR measures the quality of the image by comparing the peak signal-to-noise ratio of the original image and the compressed image, evaluates the effect of the compression algorithm, and the higher the PSNR value, the better the image quality. The comparison results of the quality of the anonymous image are shown in Table 3.

[0232] Table 3 Comparison of the quality of the anonymous image of the method of the present application on the FFHQ test data set

[0233] Models FID(↓) SSIM(↑) PSNR(↑) Pixelization 8by 8 226.58 0.54 19.43 Pixelization 16by 16 326.78 0.48 16.69 Blur 9by 9 188.66 0.58 20.39 Blur 17by 17 319.48 0.52 17.91 CIA 25.69 0.73 21.96 DP2 10.61 0.78 20.87 Ours 6.99 0.86 24.92

[0234] The facial expression recognizer ResMaskingNet is trained using the expression recognition data set AffectNet, and the expression retention effect of the anonymous face is evaluated by calculating the distance of the original face and the anonymous face on each expression. The closer the distance, the better the effect of the anonymous face in retaining the expression. The comparison results of the emotion retention ability of the anonymous image are shown in Table 4, and the expression retention effect is shown in Figure 6

[0235] Table 4 Comparison of the emotion retention ability of the anonymous image of the method of the present application on the AffectNet data set

[0236]

[0237] ​Compared with traditional anonymization methods and the latest learning-based methods CIAGAN and DeepPrivacy2 in terms of anonymity, face detection performance, image generation quality, diversity and facial expression recognition performance, it can be seen that the method of the present application better preserves the face detection and facial expression recognition performance while ensuring anonymity. At the same time, the method of the present application shows better image generation quality and diversity. In addition, the method of the present application is also compared with the latest GANonymization method in terms of expression preservation, and the results show that the present application has advantages in more expressions, preserves background information unrelated to the face, and is adapted to it, with better integrity.

[0238] The above is only the preferred embodiment of the present application, and it should be pointed out that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered within the scope of protection of the present application.

Claims

1. A face anonymization method based on a StyleGAN encoder and a skin color matching generative adversarial network, characterized in that, Includes the following steps: Step 1: Use the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map; Step 2: Extract the pose latent representation of the semantic segmentation map using a translation encoder; Step 3: Use an identity encoder to extract the latent identity representation information of the reference face image; Step 4: Integrate pose latent representation information and identity latent representation information to obtain a new latent representation that retains the original face pose while having a different identity from the original face. Step 5: Using the StyleGAN generator, the injected new latent representations are transformed into preliminary anonymized face images; Step 6: Use the Skin Color Matching Generative Adversarial Network (SCMGAN) to fuse the generated anonymous face image with the background of the original image to achieve final face anonymity, as detailed below: Step 6.1: Extract semantic segmentation maps of the faces in the training data using the method in Step 1; Step 6.2: Use the Gaussian blur method to obtain a blurred version of the original face image in the training dataset; Step 6.3: Change the contrast, brightness, saturation and hue of the face portion of the original face image to obtain a face image to be matched that does not match the background; Step 6.4: Use the total loss function of training SCMGAN Training SCMGAN decouples the training process from the encoder training: ； in Indicating resistance to loss, This represents the zero-center gradient penalty loss. Indicates pixel-level loss. This represents the perceptual loss based on VGG. and Represents the weighting constant; Step 6.5: Input the images from steps 6.1 to 6.3 into the trained SCMGAN to obtain the face image with completed skin color matching; Step 6.4 Training the total loss function of SCMGAN The calculation method is as follows: SCMGAN has three inputs: (1) a stitched image of an anonymous face image and the background of the original image; (2) a semantic segmentation image of the face of the original face image; and (3) the original face image after blurring. Training SCMGAN can be accomplished through a skin color matching proxy task. First, calculate the adversarial loss, as follows: ； ； ； in For the generator of SCMGAN, For the discriminator, Labels for real data Labels for fake data For real data, For the generator's input; Then, zero-center gradient penalty loss is used to stabilize model training: ； And use pixel-level Loss is used to match the colors of the original face and the generated face: ； in Indicates the input to the generator; Finally, a VGG-based perceptual loss function is used to determine the feature differences between the target and the generated output. The VGG loss is defined as follows: ； in The weights corresponding to the features of each layer, , , These correspond to the image's channels, width, and height dimensions, respectively. This represents the VGG-19 model; In summary, the total loss function required to train SCMGAN is: ； in , It is a weighting constant.

2. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that, The semantic segmentation map in Step 1 consists of 19 parts: background, skin, nose, glasses, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, mouth, upper lip, lower lip, hair, hat, earring, neck, necklace, and clothes.

3. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that, The translation encoder in step 2 adopts a three-level feature pyramid encoder with a pSp architecture. The pyramid encoder uses IR-SE50 as the backbone. The mapping module consists of 18 small convolutional networks style2map. The small convolutional networks numbered 1-3 are used to extract "coarse" level details, the small convolutional networks numbered 4-7 are used to extract "medium" level details, and the small convolutional networks numbered 8-18 are used to extract "fine" level details.

4. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that, In step 2, the pose latent representation of the semantic segmentation map is extracted using a translation encoder, as follows: Step 2.1: Using the CelebaAMask-HQ dataset as the training dataset, extract the face semantic segmentation map from the training data using the method in Step 1; Step 2.2: Extract latent pose representations using the translation encoder to be trained; Step 2.3: Input the complete pose latent representation into the pre-trained StyleGAN generator and use the StyleGAN generator to generate a new face image with the same pose semantics as the original image. Step 2.4: Fix the parameters of the StyleGAN generator and use the total loss function trained on the translation encoder. Training the translation encoder: ； in, , , It is a constant. For pixel-level loss functions, For the perceptual loss function, The loss function is the regularization term; Step 2.5: Extract the pose latent representation of the semantic segmentation map using the trained translation encoder.

5. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 4, characterized in that, In step 2.4, the total loss function for training the translation encoder is... The calculation method is as follows: Set the input as the original face image The latent representation vector extracted after translation encoder for: ； in BiSeNet represents the face semantic segmentation model. Indicates a translation encoder; Then the latent representation vector The generated result is obtained after passing through the StyleGAN generator. for: ； in This represents the overall architecture of the translation encoder plus the StyleGAN generator. This represents the StyleGAN generator. This represents the latent representation vector extracted by the translation encoder. express The average value; The core of model training is the choice of loss function. The training loss function of the translation encoder is a weighted combination of multiple objectives, as follows: First, utilize pixel-level loss To optimize the encoder so that it generates a new image that is pixel-wise similar to the original image, pixel-level loss Defined as: ； Then, perceptual loss LPIPS is used, which provides better image quality than other perceptual losses by learning perceptual similarity. The perceptual loss LPIPS function... Defined as: ； in Represents a perceptual feature extractor; Finally, a regularization term is used to make the latent semantic representation vector output by the translation encoder closer to the average latent representation vector. The regularization term loss function... Defined as: ； In summary, the total loss function for training the translation encoder is defined as: ； in , , This is a constant used to define the weighting values ​​of the loss function.

6. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that, Step 3 utilizes an identity encoder to extract latent identity representation information from the reference face image, as detailed below: Step 3.1: Extract the face semantic segmentation map from the training data using the method in Step 1; Step 3.2: Encode the semantic segmentation map of the input original image using a translation encoder to obtain a semantic latent representation; Step 3.3: Encode the target image using an identity encoder to obtain a latent identity representation; Step 3.4: Merge the semantic latent representation and the identity latent representation by concatenation to obtain the fused representation; Step 3.5: Input the fused representation into the StyleGAN generator to generate an anonymous face image that has the original face pose but is different from the original face identity; Step 3.6: Fix the parameters of StyleGAN and the translation encoder, and use the total loss function trained on the identity encoder. Training the identity encoder: ； in , , , is a weighting constant, representing the identity loss, used to encourage the identity encoder to effectively encode identity information; Represents the original face image and the reference target face image; Step 3.7: Use the trained identity encoder to extract the latent identity representation information of the reference face image.

7. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 6, characterized in that, The total loss function for training the identity encoder in step 3.6 The specific calculation method is as follows: Before training the identity encoder, the translation encoder needs to be pre-trained. The training process requires the two encoders to work together: the translation encoder encodes the semantic segmentation map of the input original image to obtain the semantic latent representation, and the identity encoder encodes the target image to obtain the identity latent representation. Then, the two representations are fused and input into the StyleGAN generator to generate anonymized face images. The fusion of the two latent representations is achieved by concatenation. The latent representation vectors of layers 1-8 are provided by the translation encoder, while the latent representation vectors of layers 9-18 are provided by the identity encoder. Set the reference target image as Then the identity encoder The output identity latent representation vector is represented as: ； in Indicates identity encoder; Then merge The latent representation vector is represented as: ； Subscript The third latent representation vector Dimensional components; When training the identity encoder, the pre-trained translation encoder needs to be fixed, and only the parameters of the identity encoder need to be updated. The overall loss function for training the identity encoder is also a weighted combination of multiple objectives, as follows: First, preserve the pixel-level data from the pre-training phase of the translation encoder. Loss, LPIPS loss, and regularization loss: ； ； ； ； in An anonymous face image generated from the fused latent representation vectors; Then, to encourage the identity encoder to encode identity information, an identity loss is introduced: ； in This represents a pre-trained face recognition model; In summary, the total loss function for training the identity encoder is defined as: ； in , , , This is a constant used to define the weighting values ​​of the loss function; Compared to translation encoder training, The weights for the loss and LPIPS loss have been adjusted, corresponding to... and The regularization loss and weight constant remain unchanged; It is the weighting constant for unit loss.

8. A face anonymization device based on a StyleGAN encoder and a skin color matching generative adversarial network, characterized in that, The apparatus is used to implement the face anonymization method based on a StyleGAN encoder and a skin color matching generative adversarial network as described in any one of claims 1 to 7, wherein the apparatus comprises: The semantic segmentation module uses the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map. The pose latent representation extraction module uses a translation encoder to extract the pose latent representation of the semantic segmentation map; The identity latent representation extraction module uses an identity encoder to extract the identity latent representation information of a reference face image; The fusion module is used to fuse pose potential representation information and identity potential representation information to obtain a new potential representation that retains the original face pose but has a different identity from the original face. The StyleGAN anonymization generator module transforms the injected new latent representations into preliminary anonymized face images through the StyleGAN generator. The anonymous face skin color matching module uses the skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image with the background of the original image, thus achieving the final face anonymization.

Images