Face anonymity method based on StyleGAN encoder and skin color matching generative adversarial network

Abstract

The invention discloses a face anonymous method based on a StyleGAN encoder and a skin color matching generative adversarial network. The method comprises the following steps: converting an original face image into a semantic segmentation map by using a pre-trained face semantic segmentation model BiSeNet; utilizing a translation encoder to extract posture potential representation of the semantic segmentation map, and utilizing an identity encoder to extract identity potential representation information of the reference face image; fusing the posture potential characterization information and the identity potential characterization information to obtain a new potential characterization which keeps the original face posture and is different from the original face identity, and converting the injected new potential characterization into a preliminary anonymous face image through a StyleGAN generator; and fusing the generated anonymous face image and the background of the original image by using a skin color matching generative adversarial network to realize final face anonymity. The anonymity of the anonymous picture, the face detection performance, the image generation quality, the diversity, the facial expression recognition performance and the picture integrity are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of face image anonymization processing, and in particular to a face anonymization method based on a StyleGAN encoder and a skin color matching generative adversarial network. Background Art

[0002] In the current digital age, with the development of artificial intelligence technology and deep learning, face recognition technology has been widely used in many fields, such as security monitoring, identity authentication, smart devices and social media. However, the accompanying privacy issues have also aroused widespread concern, especially the abuse of face data may lead to personal privacy leakage. Therefore, face anonymization methods have important research value and development prospects.

[0003] Traditional face anonymization methods mainly include pixelation, blurring, and occlusion techniques, which usually significantly reduce the usability or aesthetics of the image and affect the user experience. At the same time, with the advancement of face recognition technology, traditional anonymization methods are gradually unable to effectively prevent the recognition and restoration of face recognition systems. Therefore, anonymization technology based on deep learning and generative adversarial networks (GAN) came into being. Generative adversarial networks can generate realistic facial images through adversarial training of generators and discriminators. For example, CIAGAN mixes landmark identities and target identities, and uses twin identities to guide the discriminator for anonymous faces. This method is suitable for low-resolution face anonymity, but the generated anonymous faces are not realistic enough and may have bad faces. DeepPrivacy automatically anonymizes faces in images while retaining the original data distribution, but there is a uniqueness problem. Each identity is mapped to a unique false identity, and different images are not allowed to be generated. In addition, there is a problem of poor preservation of non-private information in the original image. GANonymization is a new face anonymization framework based on pix2pix. This framework anonymizes the original face by mapping the face scatter plot to the face, and retains the face's posture and expression information. Although this method solves the problem of loss of non-identity information of the face in previous anonymization methods, it does not simultaneously solve the problem of changes in the background of the generated anonymous faces.

[0004] In summary, the existing face anonymity methods based on generative adversarial networks mainly have the following problems: low quality of generated face images, poor diversity, poor ability to retain non-privacy information, and background loss. Therefore, it is necessary to design a face anonymity method that can better retain the performance of face detection and facial expression recognition while ensuring anonymity, has good integrity, and can retain background information not related to the face. Summary of the invention

[0005] The purpose of the present invention is to provide a face anonymization method that can confuse human eyes and machines, has strong anonymity, high face detection rate, high image generation quality, and strong background diversity and facial expression preservation capabilities.

[0006] The technical solution to achieve the purpose of the present invention is: a face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network, comprising the following steps:

[0007] Step 1: Use the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map;

[0008] Step 2: Use the translation encoder to extract the pose potential representation of the semantic segmentation map;

[0009] Step 3: Use the identity encoder to extract the identity potential representation information of the reference face image;

[0010] Step 4: Fuse the posture latent representation information and the identity latent representation information to obtain a new latent representation that maintains the original face posture and is different from the original face identity;

[0011] Step 5: Convert the injected new latent representation into a preliminary anonymous face image through the StyleGAN generator;

[0012] Step 6: Use the skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image and the background of the original image to achieve the final face anonymity.

[0013] Furthermore, the semantic segmentation map in step 1 includes 19 parts, namely background, skin, nose, glasses, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, mouth, upper lip, lower lip, hair, hat, earrings, neck, necklace and clothes.

[0014] Furthermore, the translation encoder in step 2 adopts a three-level feature pyramid encoder with a pSp architecture. The pyramid encoder uses IR-SE50 as the backbone, and the mapping module consists of 18 small convolutional networks style2map. The small convolutional networks numbered 1-3 are used to extract "coarse" level details, the small convolutional networks numbered 4-7 are used to extract "medium" level details, and the small convolutional networks numbered 8-18 are used to extract "fine" level details.

[0015] Furthermore, in step 2, the translation encoder is used to extract the pose potential representation of the semantic segmentation map, as follows:

[0016] Step 2.1, use the CelebaAMask-HQ dataset as the training dataset, and use the method in step 1 to extract the face semantic segmentation map of the training data;

[0017] Step 2.2, extracting posture latent representation using the translation encoder to be trained;

[0018] Step 2.3: Pass the complete posture potential representation into the pre-trained StyleGAN generator, and use the StyleGAN generator to generate a new face image with the same posture semantics as the original image;

[0019] Step 2.4: Fix the parameters of the StyleGAN generator and use the total loss function of the training translation encoder Train the translation encoder:

[0020]

[0021] Among them, λ1, λ2, λ3 are constants, is the pixel-level loss function, is the perceptual loss function, is the regularization loss function;

[0022] Step 2.5: Use the trained translation encoder to extract the pose potential representation of the semantic segmentation map.

[0023] Furthermore, in step 2.4, the total loss function for training the translation encoder is The calculation method is as follows:

[0024] Assume that the input original face image x is passed through the translation encoder and then the latent representation vector w is extracted trans for:

[0025] w trans =E trans (seg(x))

[0026] Where seg represents the face semantic segmentation model BiSeNet, E trans represents the translation encoder;

[0027] Then the potential representation vector w trans After passing through the StyleGAN generator, the generated result y is obtained trans for:

[0028]

[0029] Where pSp represents the overall architecture of the translation encoder plus the StyleGAN generator, G stylegan represents the StyleGAN generator, w trans represents the latent representation vector extracted by the translation encoder, Indicates w trans The average value of

[0030] The core of model training is the selection of loss function. The training loss function of the translation encoder is a weighted combination of multiple objectives, as follows:

[0031] First, we use pixel-level loss To optimize the encoder to generate a new image that is similar to the original image in pixels, pixel-level loss Defined as:

[0032]

[0033] Then, the perceptual loss LPIPS is used to provide better image quality than other perceptual losses by learning perceptual similarity. The perceptual loss LPIPS function Defined as:

[0034]

[0035] Where F represents the perceptual feature extractor;

[0036] Finally, a regularization term is used to make the latent semantic representation vector output by the translation encoder closer to the average latent representation vector. The regularization term loss function is Defined as:

[0037]

[0038] In summary, the total loss function for training the translation encoder is defined as:

[0039]

[0040] Among them, λ1, λ2, λ3 are constants used to define the weighted values ​​of the loss function.

[0041] Furthermore, in step 3, the identity encoder is used to extract the identity potential representation information of the reference face image, as follows:

[0042] Step 3.1, using the method in step 1 to extract the face semantic segmentation map of the training data;

[0043] Step 3.2, using the translation encoder to encode the semantic segmentation map of the input original image to obtain the semantic latent representation;

[0044] Step 3.3, use the identity encoder to encode the target image to obtain the identity latent representation;

[0045] Step 3.4, fusing the semantic latent representation and the identity latent representation by serial concatenation to obtain a fused representation;

[0046] Step 3.5, input the fused representation into the StyleGAN generator to generate an anonymous face image with the original face posture but different from the original face identity;

[0047] Step 3.6: Fix the parameters of StyleGAN and translation encoders and use the total loss function of training identity encoder Train the identity encoder:

[0048]

[0049] Among them, λ3,λ4,λ5,λ6 are weight constants, λ3 is consistent with the previous text, represents the identity loss, which is used to encourage the identity encoder to effectively encode the identity information; x,t represents the original face image and the reference target face image;

[0050] Step 3.7: Use the trained identity encoder to extract the identity potential representation information of the reference face image.

[0051] Furthermore, the total loss function for training the identity encoder in step 3.6 is The calculation method is as follows:

[0052] Before training the identity encoder, the translation encoder needs to be pre-trained. The training process requires the two encoders to work together: the translation encoder encodes the semantic segmentation map of the input original image to obtain the semantic latent representation, and the identity encoder encodes the target image to obtain the identity latent representation; then the two representations are fused and input into the StyleGAN generator to generate anonymous face images; the fusion of the two latent representations is achieved by serial splicing, the latent representation vectors of layers 1-8 are provided by the translation encoder, and the latent representation vectors of layers 9-18 are provided by the identity encoder;

[0053] Assume the reference target image is t, then the identity encoder w id The output identity latent representation vector is expressed as:

[0054] w id =E id (t)

[0055] Where E id represents the identity encoder;

[0056] Then fusion w mix The potential representation vector of is expressed as:

[0057]

[0058] Where the subscript i represents the component of the i-th dimension of the potential representation vector;

[0059] When training the identity encoder, the pre-trained translation encoder needs to be fixed and only the parameters of the identity encoder need to be updated;

[0060] The total loss function for training the identity encoder is also a weighted combination of multiple objectives, as follows:

[0061] First, keep the pixel-level translation encoder pre-training stage Loss, LPIPS loss and regularization loss:

[0062]

[0063] where y mix Anonymous face image generated by representing the fused latent representation vector;

[0064] Then, to encourage the identity encoder to encode the identity information, the identity loss is introduced:

[0065]

[0066] Where R represents the pre-trained face recognition model;

[0067] In summary, the total loss function for training the identity encoder is defined as:

[0068]

[0069] Among them, λ3,λ4,λ5,λ6 are constants used to define the weighted values ​​of the loss function; compared with the translation encoder training, The weights of the loss and LPIPS loss are adjusted, corresponding to λ4 and λ5 respectively, and the regularization loss and weight constant remain unchanged; λ6 is the weight constant of the unit loss.

[0070] Furthermore, in step 6, the skin color matching generative adversarial network SCMGAN is used to fuse the generated anonymous face image and the background of the original image to achieve the final face anonymity, as follows:

[0071] Step 6.1, extracting the semantic segmentation map of the face of the training data using the method in step 1;

[0072] Step 6.2, using Gaussian blur method to obtain the blurred version of the original face image in the training data set;

[0073] Step 6.3, changing the contrast, brightness, saturation and hue of the face part of the original face image to obtain a face image to be matched that is inconsistent with the background;

[0074] Step 6.4: Using the total loss function for training SCMGAN Training SCMGAN, the training process is decoupled from the training of the encoder:

[0075]

[0076] in Represents resistance to loss, represents the zero center gradient penalty loss, represents pixel-level loss, represents the perceptual loss based on VGG, λ7 and λ8 represent weight constants;

[0077] Step 6.5: Input the images in steps 6.1 to 6.3 into the trained SCMGAN to obtain a face image with skin color matching.

[0078] Furthermore, the total loss function for training SCMGAN in step 6.4 is The calculation method is as follows:

[0079] SCMGAN has three inputs: (1) the mosaic of the anonymous face image and the original image background; (2) the semantic segmentation map of the original face image; (3) the original face image after blurring. Training SCMGAN can be completed through the skin color matching proxy task.

[0080] First, calculate the adversarial loss as follows:

[0081]

[0082] Among them G SCM is the generator of SCMGAN, D SCM is the discriminator, b is the label of the real data, a is the label of the fake data, x is the real data, and i is the input of the generator;

[0083] Then use zero center gradient penalty loss to stabilize model training:

[0084]

[0085] And use pixel-level Loss to match the colors of the original and generated faces:

[0086]

[0087] Where I represents the input of the generator;

[0088] Finally, a VGG-based perceptual loss function is used to determine the feature differences between the target and generated outputs. The VGG loss is defined as follows:

[0089]

[0090] where μ iThe weight corresponding to each layer of features, C p ,W p ,H p They correspond to the channel, width, and height dimensions of the image, respectively. VGG represents the VGG-19 model.

[0091] In summary, the total loss function required to train SCMGAN is:

[0092]

[0093] Where λ7,λ8 are weight constants.

[0094] A face anonymization device based on StyleGAN encoder and skin color matching generative adversarial network, the device is used to implement the face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network, the device includes:

[0095] The semantic segmentation module uses the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map;

[0096] The pose latent representation extraction module uses the translation encoder to extract the pose latent representation of the semantic segmentation map;

[0097] The identity latent representation extraction module uses the identity encoder to extract the identity latent representation information of the reference face image;

[0098] A fusion module is used to fuse the posture latent representation information and the identity latent representation information to obtain a new latent representation that maintains the original face posture and is different from the original face identity;

[0099] The StyleGAN anonymous generator module converts the injected new latent representation into a preliminary anonymous face image through the StyleGAN generator;

[0100] The anonymous face skin color matching module uses the skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image with the background of the original image to achieve the final face anonymity.

[0101] Compared with the prior art, the present invention has the following significant advantages: (1) It utilizes a translation encoder and an identity encoder to extract semantic latent representation and identity latent representation respectively, which not only retains information irrelevant to identity, such as background information, facial semantic segmentation information, and facial expression information, but also ensures the quality and anonymity of image generation; (2) Anonymous faces can not only confuse machines, but also have a good confusing effect for human eyes, and have a good anonymity effect; (3) It retains the performance of face detection and facial expression recognition, can retain background information irrelevant to the face, and has good integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 It is a flow chart of a face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network of the present invention.

[0103] Figure 2 It is a flowchart of the overall process of the present invention.

[0104] Figure 3 It is a structural block diagram of the translation encoder and identity encoder in the present invention.

[0105] Figure 4 This is a structural block diagram of SCMGAN in the present invention.

[0106] Figure 5 This is a diagram showing the anonymous effects of the method of the present invention and the existing method on the FFHQ dataset in the embodiments of the present invention.

[0107] Figure 6 This is a diagram of expression preservation effects of the method of the present invention and the existing method on the Affect dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0108] like Figure 1 , Figure 2 As shown, the present invention provides a face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network, comprising the following steps:

[0109] Step 1: Use the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map;

[0110] Step 2: Use the translation encoder to extract the pose potential representation of the semantic segmentation map;

[0111] Step 3: Use the identity encoder to extract the identity potential representation information of the reference face image;

[0112] Step 4: Fuse the posture latent representation information and the identity latent representation information to obtain a new latent representation that maintains the original face posture and is different from the original face identity;

[0113] Step 5: Convert the injected new latent representation into a preliminary anonymous face image through the StyleGAN generator;

[0114] Step 6: Use the skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image and the background of the original image to achieve the final face anonymity.

[0115] As a specific example, in step 1, the pre-trained face semantic segmentation model BiSeNet is used to convert the original face image into a semantic segmentation map;

[0116] The semantic segmentation map includes 19 parts, namely background, skin, nose, glasses, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, mouth, upper lip, lower lip, hair, hat, earrings, neck, necklace and clothes.

[0117] As a specific example, in step 2, the translation encoder is used to extract the pose potential representation of the semantic segmentation map;

[0118] like Figure 3 As shown, the translation encoder adopts a three-level feature pyramid encoder with a pSp architecture. The pyramid encoder uses IR-SE50 as the backbone. The mapping module consists of 18 small convolutional networks. The small convolutional networks numbered 1-3 are used to extract "coarse" level details, the small convolutional networks numbered 4-7 are used to extract "medium" level details, and the small convolutional networks numbered 4-7, 8-18 are used to extract "fine" level details.

[0119] As a specific example, the use of the translation encoder to extract the potential representation of the posture of the semantic segmentation map in step 2 is as follows:

[0120] Step 2.1, use the CelebaAMask-HQ dataset as the training dataset, and use the method in step 1 to extract the face semantic segmentation map of the training data;

[0121] Step 2.2, extracting posture latent representation using the translation encoder to be trained;

[0122] Step 2.3: Pass the complete posture potential representation into the pre-trained StyleGAN generator, and use the StyleGAN generator to generate a new face image with the same posture semantics as the original image;

[0123] Step 2.4: Fix the parameters of the StyleGAN generator and use the total loss function of the training translation encoder Train the translation encoder:

[0124]

[0125] Among them, λ1, λ2, λ3 are constants, is the pixel-level loss function, is the perceptual loss function, is the regularization loss function;

[0126] Step 2.5: Use the trained translation encoder to extract the pose potential representation of the semantic segmentation map.

[0127] As a specific example, the total loss function for training the translation encoder described in step 2.4 is The calculation method is as follows:

[0128] Suppose the input image x is passed through the translation encoder to extract the latent representation vector w trans for:

[0129] w trans =E trans (seg(x))

[0130] Where seg represents the face semantic segmentation model BiSeNet, E trans Represents a translation encoder.

[0131] Then the potential representation vector w trans After passing through the StyleGAN generator, the generated result y is obtained trans for:

[0132]

[0133] Where pSp represents the overall architecture of the translation encoder plus the StyleGAN generator, G stylegan represents the StyleGAN generator, E trans represents the translation encoder, w trsns represents the latent representation vector extracted by the translation encoder, Indicates w trans The average value of

[0134] The core of model training is the selection of loss function. The training loss function of the translation encoder is a weighted combination of multiple objectives, as follows:

[0135] First, we use pixel-level loss To optimize the encoder and generate a new image that is similar to the original image in pixels, the pixel-level loss Defined as:

[0136]

[0137] Then, the perceptual loss LPIPS is used to provide better image quality than other perceptual losses by learning perceptual similarity. The perceptual loss LPIPS function Defined as:

[0138]

[0139] Where F represents the perceptual feature extractor;

[0140] Finally, a regularization term is used to make the latent semantic representation vector output by the translation encoder closer to the average latent representation vector. The regularization term loss function is Defined as:

[0141]

[0142] In summary, the total loss function for training the translation encoder is defined as:

[0143]

[0144] Among them, λ1, λ2, λ3 are constants used to define the weighted values ​​of the loss function.

[0145] As a specific example, in step 3, the identity encoder is used to extract the identity potential representation information of the reference face image;

[0146] The identity encoder described has a similar model structure to the translation encoder, but it focuses on capturing the identification information and ignores various information related to the semantic segmentation map.

[0147] As a specific example, the use of the identity encoder to extract the identity potential representation information of the reference face image in step 3 is as follows:

[0148] Step 3.1, using the method in step 1 to extract the face semantic segmentation map of the training data;

[0149] Step 3.2, using the translation encoder to encode the semantic segmentation map of the input original image to obtain the semantic latent representation;

[0150] Step 3.3, use the identity encoder to encode the target image to obtain the identity latent representation;

[0151] Step 3.4, fusing the semantic latent representation and the identity latent representation by serial concatenation to obtain a fused representation;

[0152] Step 3.5, input the fused representation into the StyleGAN generator to generate an anonymous face image with the original face posture but different from the original face identity;

[0153] Step 3.6: Fix the parameters of StyleGAN and translation encoders and use the total loss function of training identity encoder Train the identity encoder:

[0154]

[0155] Among them, λ3, λ4, λ5, λ6 are constants, and λ3 is consistent with requirement 5. Identity loss calculated for the face recognition model, x is the original face image, and t is the target image.

[0156] Step 3.7: Use the trained identity encoder to extract the identity potential representation information of the reference face image.

[0157] As a specific example, the total loss function for training the identity encoder described in step 3.6 is The calculation method is as follows:

[0158] Before training the identity encoder, the translation encoder needs to be pre-trained. The training process requires the two encoders to work together: the translation encoder encodes the semantic segmentation map of the input original image to obtain the semantic latent representation, and the identity encoder encodes the target image to obtain the identity latent representation; then the two representations are fused and input into the StyleGAN generator to generate anonymous face images; the fusion of the two latent representations is achieved by serial splicing, the latent representation vectors of layers 1-8 are provided by the translation encoder, and the latent representation vectors of layers 9-18 are provided by the identity encoder;

[0159] Assume the reference target image is t, then the identity encoder w id The output identity latent representation vector is expressed as:

[0160] w id =E id (t)

[0161] Where E id Represents an identity encoder.

[0162] Then fusion w mix The potential representation vector of is expressed as:

[0163]

[0164] Where the subscript i represents the component of the i-th dimension of the potential representation vector;

[0165] When training the identity encoder, the pre-trained translation encoder needs to be fixed and only the parameters of the identity encoder need to be updated;

[0166] The total loss function for training the identity encoder is also a weighted combination of multiple objectives, as follows:

[0167] First, keep the pixel-level translation encoder pre-training stage Loss, LPIPS loss and regularization loss:

[0168]

[0169] where y mix Represents the anonymous face image generated by the fused latent representation vector.

[0170] Then, to encourage the identity encoder to encode the identity information, the identity loss is introduced:

[0171]

[0172] Where R represents the pre-trained face recognition model.

[0173] In summary, the total loss function for training the identity encoder is defined as:

[0174]

[0175] Among them, λ3,λ4,λ5,λ6 are constants used to define the weighted values ​​of the loss function; compared with the translation encoder training, The weights of the loss and LPIPS loss are only slightly adjusted, corresponding to λ4 and λ5 respectively, and the regularization loss and its weight constants remain unchanged; λ6 is the weight constant of the identity loss.

[0176] As a specific example, in step 4, the posture latent representation information and the identity latent representation information are fused to obtain a new latent representation that maintains the original face posture and has a different identity from the original face;

[0177] Fusion mix The potential representation vector of is expressed as:

[0178]

[0179] Where the subscript i represents the component of the i-th dimension of the potential representation vector;

[0180] As a specific example, in step 5, the injected new latent representation is converted into a preliminary anonymous face image through the StyleGAN generator;

[0181] Initial anonymous face image y mix for:

[0182]

[0183] As a specific example, in step 6, the generated anonymous face image and the background of the original image are fused using the skin color matching generative adversarial network SCMGAN to achieve the final face anonymity, such as Figure 4 As shown, the details are as follows:

[0184] Step 6.1, extracting the semantic segmentation map of the face of the training data using the method in step 1;

[0185] Step 6.2, using Gaussian blur method to obtain the blurred version of the original face image in the training data set;

[0186] Step 6.3, changing the contrast, brightness, saturation and hue of the face part of the original face image to obtain a face image to be matched that is inconsistent with the background;

[0187] Step 6.4: Using the total loss function for training SCMGAN Training SCMGAN, the training process is decoupled from the training of the encoder:

[0188]

[0189] in Represents resistance to loss, represents the zero center gradient penalty loss, represents pixel-level loss, represents the VGG-based perceptual loss, and λ7 and λ8 represent weight constants.

[0190] Step 6.5: Input the images in steps 6.1 to 6.3 into the trained SCMGAN to obtain a face image with skin color matching.

[0191] As a specific example, the total loss function for training SCMGAN described in step 6.4 is The calculation method is as follows:

[0192] SCMGAN has three inputs: (1) the mosaic of the anonymous face image and the original image background; (2) the semantic segmentation map of the original face image; (3) the original face image after blurring. Training SCMGAN can be completed through the skin color matching proxy task.

[0193] First, calculate the adversarial loss as follows:

[0194]

[0195] Among them G SCM is the generator of SCMGAN, D SCM is the discriminator, b is the label of the real data, a is the label of the fake data, x is the real data, and i is the input of the generator;

[0196] Then use zero center gradient penalty loss to stabilize model training:

[0197]

[0198] And use pixel-level Loss to match the colors of the original and generated faces:

[0199]

[0200] Where I represents the input of the generator;

[0201] Finally, a VGG-based perceptual loss function is used to determine the feature difference between the target and generated outputs. The VGG loss is defined as follows:

[0202]

[0203] where μ i The weight corresponding to each layer of features, C p ,W p ,H p They correspond to the channel, width, and height dimensions of the image, respectively. VGG represents the VGG-19 model.

[0204] In summary, the total loss function required to train SCMGAN is:

[0205]

[0206] Where λ7,λ8 are weight constants.

[0207] The present invention also provides a face anonymization device based on StyleGAN encoder and skin color matching generative adversarial network, which is used to implement the face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network, and the device includes:

[0208] The semantic segmentation module uses the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map;

[0209] The pose latent representation extraction module uses the translation encoder to extract the pose latent representation of the semantic segmentation map;

[0210] The identity latent representation extraction module uses the identity encoder to extract the identity latent representation information of the reference face image;

[0211] A fusion module is used to fuse the posture latent representation information and the identity latent representation information to obtain a new latent representation that maintains the original face posture and is different from the original face identity;

[0212] The StyleGAN anonymous generator module converts the injected new latent representation into a preliminary anonymous face image through the StyleGAN generator;

[0213] The anonymous face skin color matching module uses the skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image with the background of the original image to achieve the final face anonymity.

[0214] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0215] Example

[0216] This embodiment uses the FFHQ dataset and randomly selects 10,000 face images as a test set to test the method of the present invention. Figure 1, Figure 2 As shown, the face anonymization method based on the StyleGAN encoder and the skin color matching generative adversarial network of the present invention is used to perform anonymization operation on 10,000 face images randomly selected from the FFHQ dataset. The specific steps are as follows:

[0217] Step 1: Get the FFHQ dataset, randomly select 10,000 face images as the test set, preprocess the test data, and crop them to a size of 256 × 256. Use the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map, which includes 19 parts, namely background, skin, nose, glasses, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, mouth, upper lip, lower lip, hair, hat, earrings, neck, necklace, and clothes.

[0218] The original face image is blurred by using Gaussian blur algorithm to obtain the blurred original face.

[0219] Step 2: Use the translation encoder to extract the potential representation of the posture of the semantic segmentation map; the translation encoder after training can well extract information such as posture and expression in the semantic segmentation map, which is used to retain the posture and expression information of the original face in the anonymous face;

[0220] Step 3: Use the identity encoder to extract the identity potential representation information of the reference face image; the trained identity encoder can well extract the identity information in the reference face image, which is used to provide reference information different from the original identity in the anonymous face generation process, and guide the generated face anonymization;

[0221] Step 4: Fusing the posture latent representation information and the identity latent representation information to obtain a new latent representation that maintains the original face posture and is different from the original face identity;

[0222] The position of fusion is observed by replacing dimensions. Some dimensions of the latent representation vector obtained by the translation encoder are replaced with the average latent representation vector. Specifically, the first dimension is retained, 1-2 dimensions are retained, 1-3 dimensions are retained, etc.; then the replaced latent representation vector is input into the StyleGAN generator, and 18 different generation results are obtained; by observation, starting from the third dimension, the generated image begins to show the original facial direction; starting from the fourth dimension, the generated image begins to be similar to the original facial hair; starting from the fifth dimension, the generated image begins to match the mouth shape of the original image; starting from the seventh dimension, the generated image begins to match the gender of the original image; starting from the eighth dimension, the generated image is very similar to the semantic segmentation map of the original face; after the eighth dimension, more subtle facial changes are observed. It can be seen that the first eight dimensions are mainly used to maintain posture information, and the following dimensions are used for more subtle facial changes, so it is chosen to splice the identity after the eighth dimension;

[0223] Step 5: Use the StyleGAN generator to transform the injected new latent representation into a preliminary anonymous face image.

[0224] Step 6: Use SCMGAN to fuse the generated anonymous face image and the background of the original image to achieve the final face anonymity.

[0225] The existing face detection models Dlib and SSH are used to detect anonymous face images respectively, and the detection rate is calculated. The existing face recognition models cosface and arcface are used to compare the similarity between the original face and the anonymous face, and the anonymity effect is evaluated. The original face image and the anonymous face image are input into the face recognition model respectively to obtain the identity feature vector, and the cosine distance of the two identity feature vectors is calculated. A threshold of 0.4 is set. If it is lower than 0.4, it indicates that the anonymity is successful, otherwise the anonymity fails. The anonymity failure rate is calculated. The anonymous face detection rate and anonymity are shown in Table 1, and the anonymity effect is shown in Table 1. Figure 5 Original represents the original face image, Pixelization 8by 8 and Pixelization 16by 16 represent pixelation of different sizes, Blur9by 9 and Blur 17by 17 represent blurring of different sizes, CIA represents the anonymous face generated by CIAGAN, DP2 represents the anonymous face generated by DeepPrivacy2, and Ours represents the anonymous face generated by the present invention.

[0226] Table 1 Comparison of anonymous face detection rate and anonymity of the proposed method on the FFHQ test dataset

[0227]

[0228] Different reference face images are used to generate different anonymous versions and calculate the diversity of anonymous faces. Diversity is reflected by the LPIPS indicator. LPIPS is an image quality assessment method based on deep learning. It uses a pre-trained convolutional neural network to learn the perceptual differences of image features. The larger the value, the better the diversity of anonymous faces. The comparison results of anonymous face diversity are shown in Table 2.

[0229] Table 2 Comparison of the diversity of anonymous faces of the proposed method on the FFHQ test dataset

[0230] Models LPIPS(↑) CIA 0.0257 Ours 0.0276

[0231] The image quality assessment algorithm is used to evaluate the generation quality of anonymous faces. In order to evaluate the quality of generated images, the FID, SSIM and PSNR indicators are mainly used. Among them, FID is an indicator used to evaluate the quality and diversity of the generated model. The smaller the value, the more diverse the generated images and the higher the quality. SSIM measures the similarity of two images, including brightness, contrast and structure. The closer the value is to 1, the higher the image quality; the closer the value is to 0, the lower the image quality. PSNR measures the quality of the image by comparing the peak signal-to-noise ratio of the original image and the compressed image, and evaluates the effect of the compression algorithm. The higher the PSNR value, the better the image quality. The comparison results of the anonymous image quality are shown in Table 3.

[0232] Table 3 Comparison of anonymous image quality of the proposed method on the FFHQ test dataset

[0233] Models FID(↓) SSIM(↑) PSNR(↑) Pixelization 8by 8 226.58 0.54 19.43 Pixelization 16 by 16 326.78 0.48 16.69 Blur 9by 9 188.66 0.58 20.39 Blur 17by 17 319.48 0.52 17.91 CIA 25.69 0.73 21.96 DP2 10.61 0.78 20.87 Ours 6.99 0.86 24.92

[0234] The facial expression recognition model ResMaskingNet is trained using the expression recognition dataset AffectNet. The expression retention effect of the anonymous face is evaluated by calculating the distance between the original face and the anonymous face in each expression. The closer the distance, the better the anonymous face is in retaining this expression. The comparison results of the emotion retention ability of anonymous images are shown in Table 4. Figure 6 As shown in Figure 1. GA represents the GANonymization method.

[0235] Table 4 Comparison of the anonymous image emotion retention ability of the proposed method on the AffectNet dataset

[0236]

[0237] By comparing the traditional anonymization method and the latest learning-based methods CIAGAN and DeepPrivacy2 in terms of anonymity, face detection performance, image generation quality, diversity and facial expression recognition performance, it can be seen that the method of the present invention better retains face detection and facial expression recognition performance while ensuring anonymity. At the same time, the method of the present invention shows better image generation quality and diversity. In addition, the method of the present invention is also compared with the latest GANonymization method in terms of expression retention. The results show that the present invention has advantages in more expressions, retains background information unrelated to the face, and is adapted to it and has better integrity.

[0238] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network, characterized in that: The steps include: Step 1: Use the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map; Step 2: Use the translation encoder to extract the pose potential representation of the semantic segmentation map; Step 3: Use the identity encoder to extract the identity potential representation information of the reference face image; Step 4: Fuse the posture latent representation information and the identity latent representation information to obtain a new latent representation that maintains the original face posture and is different from the original face identity; Step 5: Convert the injected new latent representation into a preliminary anonymous face image through the StyleGAN generator; Step 6: Use the skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image and the background of the original image to achieve the final face anonymity.

2. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that: The semantic segmentation map in step 1 includes 19 parts, namely background, skin, nose, glasses, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, mouth, upper lip, lower lip, hair, hat, earrings, neck, necklace and clothes.

3. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that: The translation encoder in step 2 adopts a three-level feature pyramid encoder with a pSp architecture. The pyramid encoder uses IR-SE50 as the backbone. The mapping module consists of 18 small convolutional networks style2map. The small convolutional networks numbered 1-3 are used to extract "coarse" level details, the small convolutional networks numbered 4-7 are used to extract "medium" level details, and the small convolutional networks numbered 8-18 are used to extract "fine" level details.

4. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that: In step 2, the translation encoder is used to extract the pose potential representation of the semantic segmentation map as follows: Step 2.1, use the CelebaAMask-HQ dataset as the training dataset, and use the method in step 1 to extract the face semantic segmentation map of the training data; Step 2.2, extracting posture latent representation using the translation encoder to be trained; Step 2.3: Pass the complete posture potential representation into the pre-trained StyleGAN generator, and use the StyleGAN generator to generate a new face image with the same posture semantics as the original image; Step 2.4: Fix the parameters of the StyleGAN generator and use the total loss function of the training translation encoder Train the translation encoder: Among them, λ1, λ2, λ3 are constants, is the pixel-level loss function, is the perceptual loss function, is the regularization loss function; Step 2.5: Use the trained translation encoder to extract the pose potential representation of the semantic segmentation map.

5. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 4, characterized in that: In step 2.4, the total loss function for training the translation encoder is The calculation method is as follows: Assume that the input original face image x is passed through the translation encoder and then the latent representation vector w is extracted trans for: w trans =E trans (sec(x)) Where seg represents the face semantic segmentation model BiSeNet, E trans represents the translation encoder; Then the latent representation vector w trans After passing through the StyleGAN generator, the generated result y is obtained trans for: Where pSp represents the overall architecture of the translation encoder plus the StyleGAN generator, G stylegan represents the StyleGAN generator, w trans represents the latent representation vector extracted by the translation encoder, Indicates w trans The average value of The core of model training is the selection of loss function. The training loss function of the translation encoder is a weighted combination of multiple objectives, as follows: First, we use pixel-level loss To optimize the encoder to generate a new image that is similar to the original image in pixels, pixel-level loss Defined as: Then, the perceptual loss LPIPS is used to provide better image quality than other perceptual losses by learning perceptual similarity. The perceptual loss LPIPS function Defined as: Where F represents the perceptual feature extractor; Finally, a regularization term is used to make the latent semantic representation vector output by the translation encoder closer to the average latent representation vector. The regularization term loss function is Defined as: In summary, the total loss function for training the translation encoder is defined as: Among them, λ1, λ2, and λ3 are constants used to define the weighted values ​​of the loss function.

6. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that: In step 3, the identity encoder is used to extract the identity potential representation information of the reference face image, as follows: Step 3.1, using the method in step 1 to extract the face semantic segmentation map of the training data; Step 3.2, using the translation encoder to encode the semantic segmentation map of the input original image to obtain the semantic latent representation; Step 3.3, use the identity encoder to encode the target image to obtain the identity latent representation; Step 3.4, fusing the semantic latent representation and the identity latent representation by serial concatenation to obtain a fused representation; Step 3.5, input the fused representation into the StyleGAN generator to generate an anonymous face image with the original face posture but different from the original face identity; Step 3.6: Fix the parameters of StyleGAN and translation encoders and use the total loss function of training identity encoder Train the identity encoder: Where λ3,λ4,λ5,λ6 are weight constants, representing identity loss, which are used to encourage the identity encoder to effectively encode identity information; x,t represents the original face image and the reference target face image; Step 3.7: Use the trained identity encoder to extract the identity potential representation information of the reference face image.

7. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 6, characterized in that: The total loss function for training the identity encoder in step 3.6 The calculation method is as follows: Before training the identity encoder, the translation encoder needs to be pre-trained. The training process requires the two encoders to work together: the translation encoder encodes the semantic segmentation map of the input original image to obtain the semantic latent representation, and the identity encoder encodes the target image to obtain the identity latent representation; then the two representations are fused and input into the StyleGAN generator to generate anonymous face images; the fusion of the two latent representations is achieved by serial splicing, the latent representation vectors of layers 1-8 are provided by the translation encoder, and the latent representation vectors of layers 9-18 are provided by the identity encoder; Assume the reference target image is t, then the identity encoder w id The output identity latent representation vector is expressed as: w id =E id (t) Where E id represents the identity encoder; Then fusion w mix The potential representation vector of is expressed as: Where the subscript i represents the component of the i-th dimension of the potential representation vector; When training the identity encoder, the pre-trained translation encoder needs to be fixed and only the parameters of the identity encoder need to be updated; The total loss function for training the identity encoder is also a weighted combination of multiple objectives, as follows: First, keep the pixel-level translation encoder pre-training stage Loss, LPIPS loss and regularization loss: where y mix Anonymous face image generated by representing the fused latent representation vector; Then, to encourage the identity encoder to encode the identity information, the identity loss is introduced: Where R represents the pre-trained face recognition model; In summary, the total loss function for training the identity encoder is defined as: Among them, λ3,λ4,λ5,λ6 are constants used to define the weighted values ​​of the loss function; compared with the translation encoder training, The weights of the loss and LPIPS loss are adjusted, corresponding to λ4 and λ5 respectively, and the regularization loss and weight constant remain unchanged; λ6 is the weight constant of the unit loss.

8. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 1, characterized in that: In step 6, the skin color matching generative adversarial network SCMGAN is used to fuse the generated anonymous face image and the background of the original image to achieve the final face anonymity, as follows: Step 6.1, extracting the semantic segmentation map of the face of the training data using the method in step 1; Step 6.2, using Gaussian blur method to obtain the blurred version of the original face image in the training data set; Step 6.3, changing the contrast, brightness, saturation and hue of the face part of the original face image to obtain a face image to be matched that is inconsistent with the background; Step 6.4: Using the total loss function for training SCMGAN Training SCMGAN, the training process is decoupled from the training of the encoder: in Represents resistance to loss, represents the zero center gradient penalty loss, represents pixel-level loss, represents the perceptual loss based on VGG, λ7 and λ8 represent weight constants; Step 6.5: Input the images in steps 6.1 to 6.3 into the trained SCMGAN to obtain a face image with skin color matching.

9. The face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to claim 8, characterized in that: The total loss function for training SCMGAN in step 6.4 The calculation method is as follows: SCMGAN has three inputs: (1) the mosaic of the anonymous face image and the original image background; (2) the semantic segmentation map of the original face image; (3) the original face image after blurring. Training SCMGAN can be completed through the skin color matching proxy task. First, calculate the adversarial loss as follows: Among them G SCM is the generator of SCMGAN, D SCM is the discriminator, b is the label of the real data, a is the label of the fake data, x is the real data, and i is the input of the generator; Then use zero center gradient penalty loss to stabilize model training: And use pixel-level Loss to match the colors of the original and generated faces: Where I represents the input of the generator; Finally, a VGG-based perceptual loss function is used to determine the feature differences between the target and generated outputs. The VGG loss is defined as follows: where μ i Corresponding to the weight of each layer of features, C p ,W p ,H p They correspond to the channel, width, and height dimensions of the image, respectively. VGG represents the VGG-19 model. In summary, the total loss function required to train SCMGAN is: Where λ7,λ8 are weight constants.

10. A face anonymization device based on StyleGAN encoder and skin color matching generative adversarial network, characterized in that: The device is used to implement the face anonymization method based on StyleGAN encoder and skin color matching generative adversarial network according to any one of claims 1 to 9, and the device includes: The semantic segmentation module uses the pre-trained face semantic segmentation model BiSeNet to convert the original face image into a semantic segmentation map; The pose latent representation extraction module uses the translation encoder to extract the pose latent representation of the semantic segmentation map; The identity latent representation extraction module uses the identity encoder to extract the identity latent representation information of the reference face image; A fusion module is used to fuse the posture latent representation information and the identity latent representation information to obtain a new latent representation that maintains the original face posture and is different from the original face identity; The StyleGAN anonymous generator module converts the injected new latent representation into a preliminary anonymous face image through the StyleGAN generator; The anonymous face skin color matching module uses the skin color matching generative adversarial network SCMGAN to fuse the generated anonymous face image with the background of the original image to achieve the final face anonymity.

Images