Face image privacy protection method, device, storage medium and program product for generating an adversarial makeup
By combining the trained makeup transfer model with the encoded fine-tuning generator, natural and high-quality adversarial makeup samples are generated, which solves the insufficient privacy protection in face verification applications in existing technologies and achieves efficient facial privacy protection.
Patent Information
- Application Number
- CN202410892374.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2044-07-04
AI Technical Summary
While existing technologies can generate natural and high-quality adversarial makeup samples, they are unable to achieve a high protection success rate in face verification applications.
The makeup transfer model is trained to transfer the reference makeup to the face image to be protected. The encoding fine-tuning generator is then used to generate face images with adversarial makeup. The makeup transfer generator and the fine-tuning generator are combined to optimize the generation process to achieve high quality and privacy protection.
Without significantly changing the user's appearance, the generated adversarial makeup samples have a high protection success rate in face verification, with high image quality and few artifacts, making them less likely to be detected by malicious attacks.
Smart Images
Figure CN119152076B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of makeup migration and face privacy protection, and particularly relates to a face image privacy protection method for adversarial makeup generation, a computer device, a storage medium and a program product. BACKGROUND
[0002] The rise of deep learning has triggered a paradigm shift in face recognition technology, making it widely used in social media, surveillance and biometrics. Recent research shows that adversarial samples can successfully evade the detection mechanism of face recognition systems.
[0003] There is prior art (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 15014-15023, “Protecting Facial Privacy: Generating Adversarial Identity Masks via Style-robust Makeup Transfer”) which focuses on a method of generating adversarial makeup masks, which has good performance in face privacy protection applications. However, the adversarial disturbance in the makeup conversion process produces unexpected artifacts and cannot maintain image quality.
[0004] Another prior art (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2023), pp. 20595-20605, “CLIP2Protect: Protecting Facial Privacy Using Text-Guided Makeup via Adversarial Latent Search”) proposes an adversarial method guided by makeup description text, which can be applied to different target identities by finding adversarial latent encodings in the latent space of the generation model without the need for large-scale makeup datasets or model retraining. However, in the application of face verification, this solution does not perform optimally.
[0005] There is still a lack of a comprehensive face privacy method that can generate natural and high-quality adversarial makeup samples while still achieving high protection success rates in face verification applications. SUMMARY
[0006] In view of the above technical problems and the deficiencies in the field, the present application provides a kind of face image privacy protection method of adversarial makeup generation.The present application first trains a makeup transfer model capable of finer makeup transfer, uses the model to transfer reference makeup to the face image to be protected, then encodes the face image to be protected, and uses the encoding fine-tuning generator to regenerate the face image to be protected with reference makeup, finally, by introducing the adversarial makeup into the face image encoding and generating the face image with adversarial makeup, a face privacy protection model and method of adversarial makeup generation are realized, so that natural and high-quality adversarial makeup samples can be generated while the processed face image has a high protection success rate in the application field of face verification.
[0007] A kind of face image privacy protection method of adversarial makeup generation, comprising:
[0008] Collecting face images with makeup and without makeup and performing face alignment to obtain a dataset LC;
[0009] According to the face dense feature points of the face image in the dataset LC, the face is subjected to affine transformation to obtain a dataset AF;
[0010] Extracting the segmentation analysis image parse of the face image without makeup x s and reference image y r and parse s and parse r , obtain face analysis dataset FP;Reference image y r is a face image with makeup;
[0011] Use LC, AF and FP three datasets to train a makeup transfer generator model, transfer the makeup of reference image y r to face image x s , obtain the makeup transferred image x r ;
[0012] Encode face image x s to obtain latent encoding w s , fine-tune generator G DC map the latent encoding w s of face image x s to makeup transferred image x r , obtain generated makeup image x gr ;
[0013] For any makeup-free face image requiring privacy protection, fix the fine-tuned generator G DC , and use the fine-tuned generator G DC to generate an image and a selected target face image zt The distribution distance of the privacy-protected face image without makeup is minimized as the target, and the latent code of the privacy-protected face image without makeup is trained and updated to a mixed latent code w mix The mixed latent code w mix The trained fine-tuning generator G DC is input, and an adversarial makeup face image with a privacy protection function is generated.
[0014] In the present application, makeup and no makeup can be understood as a pair of opposite concepts, and no makeup image can be understood as an image that needs to be transferred to makeup, and makeup image can be understood as an image used to provide makeup.
[0015] The face image privacy protection method for adversarial makeup generation, the face dense feature points can be obtained by using Face++API.
[0016] The face image privacy protection method for adversarial makeup generation, the segmentation analysis image parse s and parse r can be extracted by using face parsing method.
[0017] The face image privacy protection method for adversarial makeup generation, the makeup transfer generator model is trained by using LC, AF and FP three data sets, and the makeup of the reference image y r is transferred to the face image x s , to obtain the image x r after makeup transfer. Specifically, it can include:
[0018] The model structure of SSAT is adopted, the makeup transfer GAN model is trained by using LC and AF two data sets and segmentation analysis images parse s and parse r , to obtain the makeup transfer generator G MT .
[0019] The makeup transfer generator G MT transfers the makeup of the reference image y r to the face image x s , to obtain the image x r after makeup transfer.
[0020] The face image privacy protection method for adversarial makeup generation, for any face image, the corresponding latent code can be obtained by using e4e encoder.
[0021] The face image privacy protection method for adversarial makeup generation, the fine-tuning generator G DC updates the latent code w s of the face image x sThe image x migrated to the makeup r , to obtain the generated makeup image x gr The model structure of StyleGAN2 can be used in the process of generating the makeup image x total1 :
[0022] L total1 = L perspective + L2
[0023] Wherein, L perspective is the perceptual loss term of the generated makeup image x gr and the image x r migrated to the makeup calculated by lpips, L2 is the color loss term of the generated makeup image x gr and the image x r migrated to the makeup calculated by cross entropy loss.
[0024] The privacy protection method of the face image of the adversarial makeup generation can train the latent encoding of the makeup-free face image needing privacy protection and update it to the hybrid latent encoding w total2 based on the total loss L mix :
[0025] L total2 = L adv + L latent + L makeup
[0026] Wherein, L adv is the adversarial loss term of calculating the distribution distance of the generated image of the fine-tuned generator G DC and the target face image z t , L latent is the latent encoding loss term of calculating the L2 norm of the latent encoding of the makeup-free face image needing privacy protection and the latent encoding w t of the target face image z t , L makeup is the makeup loss term of calculating the region histogram matching loss of the generated image of the fine-tuned generator G DC and the image x r migrated to the makeup.
[0027] The present application also provides a computer device comprising a memory and a processor, the memory is used to store a computer program, the processor is used to execute the computer program stored in the memory, and the computer program runs to make the processor execute the privacy protection method of the face image of the adversarial makeup generation.
[0028] The application further provides a computer readable storage medium, wherein a program or instructions are stored on the computer readable storage medium, and the program or instructions, when executed by a computer device, cause the computer device to perform the face image privacy protection method for generating an adversarial makeup.
[0029] The application further provides a computer program product, comprising a computer program, wherein the computer program, when executed by a computer device, causes the computer device to perform the face image privacy protection method for generating an adversarial makeup.
[0030] Compared with the prior art, the application has the following beneficial effects:
[0031] 1. The application designs a method for generating an adversarial sample that can naturally combine a makeup effect and an adversarial disturbance without significantly changing the appearance of a user, has a high protection success rate in an application scenario of face verification, and effectively protects the facial privacy of the user.
[0032] 2. The generating method of the application can generate an adversarial sample with higher image quality and fewer artifacts, and is more difficult to be found by malicious attackers while ensuring image quality and natural effect. BRIEF DESCRIPTION OF DRAWINGS
[0033] Figure 1 A flowchart of a face image privacy protection method for generating an adversarial makeup according to the application. DETAILED DESCRIPTION
[0034] The application will be further described below in combination with the drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application.
[0035] In combination with Figure 1 A face image privacy protection method for generating an adversarial makeup, comprising the following steps:
[0036] (1) performing affine transformation on a face according to dense feature points of the face in a LC data set to obtain a data set AF, and extracting segmented and parsed images of the face and a reference image to obtain a data set FP, specifically comprising the following steps:
[0037] (1-1) collecting face images with makeup and face images without makeup, and performing face alignment on the images to obtain a data set LC;
[0038] (1-2) obtaining dense feature points of the face in the data set LC by using Face++ API, and performing affine transformation on the makeup / no makeup face according to the dense feature points to obtain a data set AF;
[0039] (1-3) extracting a face image without makeup x by using a face parsing methods and LC, i.e. a reference image y r s and parse r get a face parsing dataset FP.
[0040] (2) Train the makeup transfer generator model using the three datasets LC, AF and FP, transfer the makeup of the reference image y r to the face image x s , and get the makeup transferred image x r , which specifically includes the following steps:
[0041] (2-1) Use the SSAT model structure, use the two datasets LC and AF and the segmentation parsing image parse s and parse r to train the makeup transfer GAN model, and get the makeup transfer generator G MT with more refined makeup color transfer.
[0042] (2-2) Use the makeup transfer generator G MT to transfer the makeup of the reference image y r to the face image x s , and get the makeup transferred image x r .
[0043] For specific SSAT model structure construction, see Sun, Z., Chen, Y., & Xiong, S. (2022). SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2), 2325-2334.
[0044] The makeup transferred image generated by the trained makeup transfer generator obtained in step (2) of the embodiment shows that, compared with the makeup transferred image obtained by directly using the pre-trained SSAT model in the existing literature for makeup transfer, the trained makeup transfer generator model of the present application can successfully apply the makeup style in the reference image to the original image, and can more accurately transfer the makeup color.
[0045] (3) Encode the face image x s to obtain the latent encoding w s , fine-tune the generator G DC to get the face image x s latent code w s image x after makeup transfer r , to obtain the generated makeup image x gr , specifically comprising the steps of:
[0046] (3-1) encoding the face image x s using an e4e encoder to obtain the latent code w s ;
[0047] (3-2) using the model structure of StyleGAN2, based on the training total loss term L total1 , fine-tuning the generator G DC to generate an image x s with makeup according to the latent code w s of the face image x gr . Wherein the training total loss term L total1 is specifically as follows:
[0048] L total1 = L perspective + L2
[0049] Wherein, L perspective is the perceptual loss term of the generated makeup image x gr and the image x r after makeup transfer calculated by lpips, and L2 is the color loss term of the generated makeup image x gr and the image x r after makeup transfer calculated by cross-entropy loss.
[0050] The construction method of e4e encoder is described in Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., & Cohen-Or, D. (2021). Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4), 1-14.
[0051] For the specific model structure construction of StyleGAN2, see Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020). Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 8110-8119).
[0052] (4) For any face image without makeup that needs privacy protection, the fine-tuned generator G after training is fixed. DC , to fine-tune the generator G DC The generated image and the selected target face image z t The goal is to minimize the distribution distance of the face image without makeup that needs privacy protection, and then train the latent code of the face image without makeup and update it to the mixed latent code w mix , the mixed latent code w mix Input the trained fine-tuned generator G DC ,generating adversarial makeup face images with privacy preservation.
[0053] In the present invention, for any face image, the e4e encoder can be used to obtain its corresponding latent code. For example, the e4e encoder can be used to convert the target face image z t Encode to obtain the potential code w t .
[0054] In step (4), based on the total training loss term L total2 The latent code of the makeup-free face image that needs privacy protection is trained and updated to the hybrid latent code w mix :
[0055] L total2 =L adv +L latent +L makeup
[0056] Among them, L adv Is to calculate the fine-tuned generator G DC The generated image and target face image z t The adversarial loss term of the distribution distance, L latent It is to calculate the potential code of the face image without makeup that needs privacy protection and the target face image z t The latent code w t The latent coding loss term of the L2 paradigm, Lmakeup is a fine-tuned generator G DC of generated images and images x r after makeup transfer.
[0057] L makeup is a region histogram matching loss, see Risser, E., Wilmot, P., & Barnes, C. (2017). Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv: 1701.08893 for calculation method.
[0058] The generation result of the face image privacy protection method for adversarial makeup generation in the embodiment is less than 75 points in the Face++ API test in the face verification application scenario. It can be seen that the present application can naturally combine the makeup effect and the adversarial disturbance while maintaining the quality of the generated image, has a high protection success rate in the face verification application scenario, and effectively protects the face privacy of the user.
[0059] A computer device, comprising a memory for storing a computer program and a processor for executing the computer program stored in the memory, wherein the computer program runs to make the processor execute the face image privacy protection method for adversarial makeup generation.
[0060] A computer readable storage medium, wherein a program or instruction is stored on the computer readable storage medium, and the program or instruction makes a computer device execute the face image privacy protection method for adversarial makeup generation when the program or instruction is executed by the computer device.
[0061] A computer program product, comprising a computer program, and the computer program makes a computer device execute the face image privacy protection method for adversarial makeup generation when the computer program is executed by the computer device.
[0062] In addition, it should be understood that, after reading the above description of the present application, those skilled in the art can make various modifications or changes to the present application, and these equivalent forms also fall within the scope defined by the appended claims of the present application.
Claims
1. A method for protecting privacy of a human face image for a counteractive makeup generation, characterized by, The application relates to a method for protecting the privacy of a face image with a generated makeup, and a computer device and a computer program product. Collecting face images with makeup and without makeup and performing face alignment to obtain a dataset LC; Performing affine transformation on the face according to the face dense feature points of the face images in the dataset LC to obtain a dataset AF; extracting face images x without makeup from a dataset LC s and reference images y r segmentation analysis images parse s and parse r , obtaining a face analysis dataset FP; reference images y r are face images with makeup; The makeup transfer generator model is trained by using three data sets of LC, AF and FP, and the makeup of a reference image y r is transferred to a face image x s , to obtain a makeup-transferred image x r ; The face image x s Encode to obtain the potential code w s , fine-tune the generator G DC The face image x s The latent code w s Mapped to the image x after makeup transfer r , get the generated makeup image x gr ; For any face image without makeup that needs privacy protection, the fine-tuned generator G after training is fixed DC , to fine-tune the generator G DC The generated image and the selected target face image z t The goal is to minimize the distribution distance, based on the total training loss term L total2 The latent code of the makeup-free face image that needs privacy protection is trained and updated to the hybrid latent code w mix , the mixed latent code w mix Input the trained fine-tuned generator G DC ,generate adversarial makeup face images with privacy protection; L total2 = L adv + L latent + L makeup wherein L adv is an adversarial loss term of distribution distance, L DC is a latent encoding loss term of L2 norm, L t is a makeup loss term of region histogram matching loss, L latent is a latent encoding loss term of L2 norm, L t is an adversarial loss term of distribution distance, L t is a latent encoding loss term of L2 norm, L makeup is an adversarial loss term of distribution distance, L DC is a latent encoding loss term of L2 norm, L r is a makeup loss term of region histogram matching loss, L 2. The privacy preserving method of a human face image for a confrontational makeup generation according to claim 1, characterized in that, The face dense feature points are obtained by using Face++ API. 3.The privacy protection method of a human face image for a generated adversarial makeup according to claim 1, wherein, segmentation parse s and parse r extracted using face parsing method.
4. The privacy preserving method of a human face image for a confrontational makeup generation according to claim 1, wherein, The makeup transfer generator model is trained by using three data sets of LC, AF and FP, and the makeup of a reference image y r is transferred to a face image x s , to obtain a makeup-transferred image x r Specifically, the method comprises the following steps. The model structure of SSAT is adopted, and LC and AF two data sets and segmentation analysis images parse s and parse r The makeup transfer GAN model is trained to obtain a makeup transfer generator G MT ; Makeup transfer generator G MT Reference image y r is transferred to the face image x s , and a makeup-transferred image x r is obtained.
5. The privacy preserving method of a human face image for a confrontational makeup generation according to claim 1, wherein, For any face image, a corresponding latent code is obtained by using an e4e encoder.
6. The privacy preserving method of a human face image for a confrontational makeup generation according to claim 1, wherein, Fine-tuning the generator G DC The face image x s The latent code w s Mapped to the image x after makeup transfer r , get the generated makeup image x gr In the process of using the StyleGAN2 model structure and based on the total training loss term L total1 : L total1 = L perspective + L2 wherein L perspective is a perceptual loss term of the generated makeup image x gr and the makeup-migrated image x r is calculated using lpips, L2 is a color loss term of the generated makeup image x gr and the makeup-migrated image x r is calculated using cross-entropy loss.
7. A computer device comprising a memory for storing a computer program and a processor for executing the computer program stored in the memory, characterized in that, The computer program runs to enable the processor to perform the privacy protection method of the face image with a generated makeup according to any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores programs or instructions, and when the programs or instructions are executed by the computer device, the computer device executes the privacy protection method of the face image with a generated makeup according to any one of claims 1-6.
9. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the computer device, the computer device executes the privacy protection method of the face image with a generated makeup according to any one of claims 1-6.