Facial manipulation active defense method based on adversarial attack and frequency manipulation
By training the perturbation generator using generative adversarial attacks and frequency manipulation, the limitations of white-box defense methods and the problem of image quality degradation are solved, achieving efficient defense against various deepfake models in black-box scenarios while maintaining image quality.
Patent Information
- Application Number
- CN202411939291.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-11-21
- Estimated Expiration
- 2044-12-26
AI Technical Summary
Existing active defense methods are mostly based on white-box settings, which cannot effectively defend against unknown deepfake models. Furthermore, their performance degrades when faced with multiple tampering models, and they fail to effectively consider the differences in sensitivity of the human visual system, which leads to a decrease in image quality.
By training a perturbation generator through generative adversarial attacks, combined with multi-teacher distillation and frequency-aware loss, imperceptible adversarial perturbations are generated, focusing on high-frequency regions of face images. This trains a student perturbation generator to adapt to various deepfake models, improving defense generalization ability and image quality.
It effectively defends against various deepfake models in black-box scenarios, maintains image quality, improves the ease of use and generalization ability of the defense, and significantly enhances the defense effect.
Smart Images
Figure CN119992617B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to facial security and adversarial attack technology, and more particularly to an active defense method for facial tampering based on adversarial attacks and frequency manipulation. Background Technology
[0002] With the rapid development of deep learning technology, highly realistic face-swapped videos and images have become increasingly common. This type of technology is known as "deepfake," and although it has positive uses in entertainment, artistic creation, and other fields, its misuse also poses serious social security risks.
[0003] To mitigate the risks posed by deepfake and face tampering technologies, researchers have proposed many excellent countermeasures. However, the vast majority of these are forensic methods based on post-detection, such as MesoNet, XceptionNet, and Face X-ray. While these methods achieve high accuracy in detecting forged face images or videos, detection occurs after the face image has been altered, potentially indicating the impact and harm already caused. Therefore, developing effective deepfake defense mechanisms, especially proactive defense mechanisms, has become a crucial and urgent issue.
[0004] Existing active defense methods based on adversarial perturbations are typically designed for white-box scenarios, requiring the calculation of perturbations based on the internal information of the deepfake model. These methods experience a significant performance drop in black-box scenarios. Furthermore, research has found that adversarial perturbations in low-frequency regions (such as the cheeks) are more easily perceived by the human eye, but current active defense methods do not consider this characteristic, randomly adding perturbations across the entire facial area, which can potentially degrade facial image quality. Therefore, designing an active defense method that exhibits high generalization ability and remains imperceptible under black-box conditions is an important but still under-researched problem. Summary of the Invention
[0005] The purpose of this invention is to provide an active defense method against facial tampering based on adversarial attacks and frequency manipulation.
[0006] The technical solution for achieving the objective of this invention is: a proactive defense method for facial tampering based on adversarial attacks and frequency manipulation, characterized by comprising the following steps:
[0007] Teacher perturbation generator training phase:
[0008] Step 1: Train a perturbation generator based on generative adversarial attacks to generate adversarial perturbations from face images;
[0009] Step 2: Add adversarial perturbations to the face image, and input it along with the original face image into the face tampering model to calculate the adversarial loss;
[0010] Step 3: Input the original face image and the perturbated face image into the discriminator and calculate the generation loss;
[0011] Step 4: Train the perturbation generator and discriminator alternately using adversarial loss and generative loss;
[0012] Step 5: Train multiple teacher perturbation generators for forgery models of different depths using adversarial loss and generative loss;
[0013] Student perturbation generator training phase:
[0014] Step 6: Input the face image into the perturbation generator, and calculate the multi-teacher distillation loss using the perturbation output by the teacher perturbation generator and the perturbation generated by the student perturbation generator;
[0015] Step 7: Perform wavelet decomposition on the face image with added adversarial perturbation and the original face image, and calculate the frequency perception loss for the low-frequency part;
[0016] Step 8: Train the student perturbation generator using adversarial loss, multi-teacher distillation loss, and frequency domain perception loss;
[0017] Testing phase:
[0018] Step 9: Input the face image into the student perturbation generator to generate adversarial perturbations, and superimpose them onto the face image to defend against face tampering.
[0019] Compared with the prior art, the significant advantages of this invention are:
[0020] (1) This invention considers active defense against deepfakes in black-box scenarios. Compared with the previous active defense method based on white-box settings, it can defend against deepfakes even without knowing the deepfake model information, thus improving the ease of use of the active defense method in real-world scenarios.
[0021] (2) This invention considers the generalization of active defense models to various deepfake models. Teacher perturbation generators are trained for different deepfake models. Multi-teacher distillation loss is added when training student perturbation generators. Compared with the adversarial perturbation added by previous methods, which can only effectively defend against a single deepfake model, this method solves the problem that traditional defense methods can only defend against a single deepfake method.
[0022] (3) The present invention takes into account the different sensitivities of the human visual system to different frequency domains of an image, and limits the perturbation to the high frequency region of the face image. The human visual system is less likely to detect the adversarial perturbation added here. Compared with the previous method, this method improves the visual quality of the protected image. Attached Figure Description
[0023] Figure 1This is a flowchart illustrating the complete technical solution of the present invention.
[0024] Figure 2 This is a model framework diagram of the present invention. Detailed Implementation
[0025] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0026] Currently, existing proactive defense methods against face tampering have the following problems:
[0027] 1) Existing proactive defense methods are mostly based on white-box settings, which have strong limitations in practical applications when facing unknown tampering models.
[0028] 2) Existing methods mostly only consider defense against a single tampering model, and will suffer strong performance degradation when facing the tampering risk of multiple tampering models.
[0029] 3) Existing methods do not take into account the differences in sensitivity of the human visual system when adversarial perturbations are added, which can lead to a decrease in the visual quality of the protected face image.
[0030] To address the aforementioned problems, this invention proposes an active defense method for face tampering based on adversarial attacks and frequency domain manipulation. The method includes: generating adversarial perturbations using a perturbation generator; applying these perturbations to a facial image to generate a protected facial image; inputting the original and protected facial images into a face tampering model to obtain the tampered face, and calculating adversarial and generative losses; training the perturbation generator, training multiple teacher perturbation generators for different tampering models; designing a frequency-aware loss based on wavelet transform and inverse transform to concentrate the adversarial perturbations in high-frequency regions insensitive to the human eye; training a student perturbation generator using the adversarial loss, multi-teacher distillation loss, and frequency-aware loss; and adding perturbations to the facial image using the student perturbation generator for active defense. This invention effectively defends against various face tampering methods by generating universal and imperceptible adversarial perturbations, demonstrating significant superiority, especially in black-box scenarios. The active defense method of this invention has good generalization ability and can reduce the visual quality loss of the protected facial image, significantly improving the defense effect and usability against face tampering.
[0031] The specific steps of the present invention will now be described with reference to the accompanying drawings.
[0032] like Figure 1 As shown, a proactive defense method for facial tampering based on adversarial attacks and frequency manipulation includes the following steps:
[0033] Teacher perturbation generator training phase:
[0034] Step 1: Train a perturbation generator based on generative adversarial attacks to generate adversarial perturbations from face images;
[0035] Step 2: Add adversarial perturbations to the face image, and input it along with the original face image into the face tampering model to calculate the adversarial loss;
[0036] Step 3: Input the original face image and the perturbated face image into the discriminator and calculate the generation loss;
[0037] Step 4: Train the perturbation generator and discriminator alternately using adversarial loss and generative loss;
[0038] Step 5: Train multiple teacher perturbation generators for forgery models of different depths using adversarial loss and generative loss;
[0039] Student perturbation generator training phase:
[0040] Step 6: Input the face image into the perturbation generator, and calculate the multi-teacher distillation loss using the perturbation output by the teacher perturbation generator and the perturbation generated by the student perturbation generator;
[0041] Step 7: Perform wavelet decomposition on the face image with added adversarial perturbation and the original face image, and calculate the frequency perception loss for the low-frequency part;
[0042] Step 8: Train the student perturbation generator using adversarial loss, multi-teacher distillation loss, and frequency domain perception loss;
[0043] Testing phase:
[0044] Step 9: Input the face image into the student perturbation generator to generate adversarial perturbations, and superimpose them onto the face image to defend against face tampering.
[0045] Furthermore, a perturbation generator is trained based on generative adversarial attacks. Generative adversarial attacks mainly consist of three modules: a perturbation generator (PG), a discriminator (D), and a deep face forgery model (M). The perturbation generator generates adversarial perturbations θ for a face image x, which are then superimposed onto the original face data x to obtain the protected face image x′. The discriminator distinguishes between the original face data x and the protected face image x′. The deep face forgery model is the target model for active defense, used to generate forged face images y and y′ based on x and x′.
[0046] Furthermore, during the model training phase, the face spoofing model is fixed, and training is only performed on the perturbation generator and the discriminator. The training objective of the perturbation generator is to: maintain the visual consistency between the protected face image x′ and the original face image x; and maximize the difference between the spoofed face image y and the protected spoofed face image y′. That is, to prevent the tampering model from successfully tampering with the face image without compromising its visual quality. The training objective of the discriminator is to distinguish the protected face image x′ from the original face image x as much as possible.
[0047] Furthermore, based on the forged face image M(x,c) j ) and the protected fake face image M(x′,c j ) Calculate the adversarial loss L adv :
[0048]
[0049] c j The target attributes for facial manipulation are represented, such as hair color and facial expression. j represents the number of attributes. M(x,c) j The forgery model represents the model based on the original face image x and the target attribute c. j Generated fake faces.
[0050] Furthermore, the original face image and the perturbated face image are input into the discriminator to calculate the generation loss L. GAN :
[0051] L GAN =E x logD(x)+E x log(1-D(x')).
[0052] Where D represents the discriminator, used to distinguish between the original face data x and the protected face image x′, D(·) represents the discriminator's judgment result, close to 1 indicates that it is the original face, close to 0 indicates that it is the protected face image, E x This represents the distribution of facial data.
[0053] Furthermore, by utilizing adversarial loss and generative loss, the perturbation generator and discriminator are trained alternately through minimax game, ultimately resulting in multiple teacher perturbation generators for different face tampering models.
[0054] Furthermore, multiple teacher perturbation generators (PGs) were trained for different depths of tampering models. t In training the student perturbation generator PG sUtilizing a multi-teacher distillation module can improve the generalization ability of the perturbation generator. Face images are input into a pre-trained teacher perturbation generator, and the multi-teacher distillation loss is calculated using the output perturbation and the perturbation generated by the student perturbation generator. The multi-teacher distillation loss L... MKD for:
[0055]
[0056] Where n represents the teacher perturbation generator PG t Quantity, PG t (·) and PG s (·) represents the adversarial perturbation generated by the teacher perturbation generator and the student perturbation generator based on the input of the face image.
[0057] Furthermore, based on wavelet transform, the face image is reconstructed using the low-frequency components, and the frequency-aware loss L is calculated. FMSE This concentrates the disturbance in the high-frequency region that is not sensitive to the human eye.
[0058] The face image is decomposed based on orthogonal wavelet low-pass filter L and high-pass filter H, into a low-frequency component x. ll and three high-frequency components x lh x hl x hh :
[0059] x ll =LxL T ,x lh =HxL T ,x hl =LxH T ,x hh =HxH T ,
[0060] Component x ll It retains the low-frequency information of the original image, while x lh x hl and x hh High-frequency information associated with significant edge changes was preserved.
[0061] By excluding high-frequency components and reconstructing the image using only low-frequency components, the low-frequency information φ(x) of the face image is obtained:
[0062] φ(x)=L T x ll L = L T (LxL T )L.
[0063] By using wavelet transform and inverse wavelet transform, the low-frequency reconstructed components φ(x) of the face image and φ(x′) of the protected face image are extracted. Based on this, the frequency-aware mean square error loss L is calculated. FMSE By utilizing L FMSE The PG is stimulated to limit the generated perturbations to the high-frequency region, thereby improving the visual quality of the protected image;
[0064] I FMSE =MSE(φ(x),φ(x'))
[0065] Furthermore, when training the student perturbation generator, the loss function L consists of the following parts:
[0066] L = L GAN -αL adv +βL MKD +γL FMSE ·
[0067] α, β, and γ represent weight coefficients for different losses. PG is trained by minimizing L. s Ultimately, a student perturbation generator capable of generating adversarial perturbations for various tampering models was obtained.
[0068] Furthermore, after training, the post-training PG s The generated perturbation is superimposed on the face image to be published to obtain the protected face image, thereby achieving proactive defense against various face tampering models.
[0069] The present invention will now be described in detail with reference to the embodiments.
[0070] Example
[0071] The following is combined Figure 1 , Figure 2 The present invention provides a detailed description of the specific steps of the active defense method for face spoofing based on adversarial attacks and frequency domain manipulation.
[0072] Step 1: Prepare the dataset and face tampering model
[0073] CelebA was chosen as the training dataset, containing 202,599 face images and corresponding facial attribute labels. Four face manipulation models were selected as target models for defense: StarGAN, AGGAN, AttGAN, and HiSD. This work directly uses the face manipulation models trained on the CelebA dataset. When training the perturbation generator, the first 18,200 images from the CelebA dataset were used for training, while the remaining images were reserved for testing.
[0074] Step Two: Model Building and Training
[0075] We designed and built a perturbation generator. In this work, both the teacher perturbation generator and the student perturbation generator use Unet-128 as the network architecture. The discriminator consists of seven convolutional layers, dense layers and fully connected layers. The model is implemented using the deep learning framework PyTorch, and the corresponding hyperparameters, such as learning rate and batch size, are set.
[0076] Step 3: Training the teacher perturbation generator
[0077] StarGAN, AGGAN, AttGAN, and HiSD were used as target tampering models to train perturbation generators. Adversarial perturbations were generated from face images using the perturbation generators. Adversarial perturbations were added to the face images, and the images, along with the original face images, were input into the deep forgery model to calculate the adversarial loss. The original face images and the perturbated face images were input into the discriminator to calculate the generation loss. The perturbation generator and the discriminator were trained alternately through a mini-maximum game, resulting in four pre-trained teacher perturbation generators.
[0078] Step 4: Train the student perturbation generator
[0079] The face image is input into the pre-trained teacher perturbation generator, and the multi-teacher distillation loss is calculated using the output perturbation and the perturbation generated by the student perturbation generator. Wavelet decomposition is performed on the face image with added adversarial perturbation and the original face image, and frequency-aware loss is calculated for the low-frequency part. The student perturbation generator is trained using the adversarial loss, multi-teacher distillation loss and frequency-aware loss.
[0080] Step 5: Model Testing and Performance Evaluation
[0081] Four face tampering models, StarGAN, AGGAN, AttGAN, and HiSD, were tested on the CelebA, RAFD, LFW, and CACD datasets. The test metrics included L1 norm loss, L2 norm loss, structural similarity index (SSIM), defense success rate (DSR), facial region defense success rate (SRmask), peak signal-to-noise ratio (PSNR), and low-frequency image index (LF). The test results are shown in Table 1.
[0082] Table 1. Cross-dataset and cross-model performance test
[0083]
[0084] As shown in Table 1, the proposed method demonstrates excellent defense performance on all four datasets, with L1 exceeding 0.1, L2 exceeding 0.05, and SRmask exceeding 0.9 in almost all cases. This indicates that the method achieves strong defense performance and generalization across multiple datasets and various face tampering models.
[0085] Furthermore, this method was compared with existing attack methods on the CelebA dataset, including BIM, MIM, PGD, DI2-FGSM, M-DI2-FGSM, AutoPGD, CMUA, and IDFM. The test results are shown in Table 2.
[0086] Table 2 Performance Comparison of Different Methods
[0087]
[0088]
[0089] As can be seen, in most cases, this method exhibits best or second-best defensive performance. This proves that even in black-box scenarios, this method can achieve defensive performance comparable to white-box methods.
[0090] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A proactive defense method for facial tampering based on adversarial attacks and frequency manipulation, characterized in that, Includes the following steps: Teacher perturbation generator training phase: Step 1: Train a perturbation generator based on generative adversarial attacks to generate adversarial perturbations from face images; Step 2: Add adversarial perturbations to the face image, and input it along with the original face image into the face tampering model to calculate the adversarial loss; Step 3: Input the original face image and the perturbated face image into the discriminator and calculate the generation loss; Step 4: Train the perturbation generator and discriminator alternately using adversarial loss and generative loss; Step 5: Train multiple teacher perturbation generators for forgery models of different depths using adversarial loss and generative loss; Student perturbation generator training phase: Step 6: Input the face image into the perturbation generator, and calculate the multi-teacher distillation loss using the perturbation output by the teacher perturbation generator and the perturbation generated by the student perturbation generator; Step 7: Perform wavelet decomposition on the face image with added adversarial perturbation and the original face image, and calculate the frequency perception loss for the low-frequency part; Step 8: Train the student perturbation generator using adversarial loss, generative loss, multi-teacher distillation loss, and frequency domain perception loss; Testing phase: Step 9: Input the face image into the student perturbation generator to generate adversarial perturbations, and superimpose them onto the face image to defend against face tampering.
2. The active defense method for facial tampering based on adversarial attacks and frequency manipulation according to claim 1, characterized in that, The perturbation generator is trained based on generative adversarial attacks. The generative adversarial attack consists of three modules: a perturbation generator PG, a discriminator D, and a deep face spoofing model M. The perturbation generator generates adversarial perturbations θ for a face image x, and then superimposes θ onto the original face data x to obtain the protected face image x′. The discriminator distinguishes between the original face data x and the protected face image x′. The deep face spoofing model is the target model for active defense, used to generate spoofed face images y and y′ based on x and x′.
3. The active defense method for facial tampering based on adversarial attacks and frequency manipulation according to claim 2, characterized in that, During the model training phase, the face spoofing model is fixed, and only the perturbation generator and discriminator are trained. The training objectives of the perturbation generator are: to maintain the visual consistency between the protected face image x′ and the original face image x; to maximize the difference between the spoofed face image y and the protected spoofed face image y′; that is, to prevent the tampering model from tampering with the face image without compromising the visual quality of the face image. The training objective of the discriminator is to distinguish the protected face image x′ from the original face image x as much as possible.
4. The active defense method for facial tampering based on adversarial attacks and frequency manipulation according to claim 3, characterized in that, Based on the forged face image M(x,c j ) and the protected fake face image M(x′,c j ) Calculate the adversarial loss L adv : Where c j M(x,c) represents the target attribute during face tampering, j represents the number of target attributes, and M(x,c) represents the number of target attributes. j The forgery model represents the model based on the original face image x and the target attribute c. j Generated fake faces.
5. The active defense method for facial tampering based on adversarial attacks and frequency manipulation according to claim 4, characterized in that, The original face image and the perturbated face image are input into the discriminator to calculate the generation loss L. GAN : L GAN =E x logD(x)+E x log(1-D(x')), Where D represents the discriminator, used to distinguish between the original face data x and the protected face image x′, D(·) represents the discriminator's judgment result, close to 1 indicates that it is the original face, close to 0 indicates that it is the protected face image, E x This represents the distribution of facial data.
6. The active defense method for facial tampering based on adversarial attacks and frequency manipulation according to claim 5, characterized in that, By using adversarial loss and generative loss to alternately train the perturbation generator and discriminator through mini-maximum game, multiple teacher perturbation generators are finally obtained for different face tampering models.
7. The active defense method for deep face forgery based on frequency manipulation according to claim 6, characterized in that, Training multiple teacher perturbation generators (PGs) for different depths of tampering models t In training the student perturbation generator PG s Utilizing a multi-teacher distillation module; multi-teacher distillation loss L MKD for: Where n represents the teacher perturbation generator PG t Quantity, PG t (·) and PG s (·) represents the adversarial perturbation generated by the teacher perturbation generator and the student perturbation generator based on the input of the face image.
8. The active defense method for deep face forgery based on frequency manipulation according to claim 7, characterized in that, Face images are reconstructed using the low-frequency components based on wavelet transform, and the frequency-aware loss L is calculated. FMSE This concentrates the disturbance in the high-frequency region that is not sensitive to the human eye; The face image is decomposed based on orthogonal wavelet low-pass filter L and high-pass filter H, into a low-frequency component x. ll and three high-frequency components x lh x hl x hh : x ll =LxL T ,x lh =HxL T ,x hl =LxH T ,x hh =HxH T Component x ll It retains the low-frequency information of the original image, while x lh x hl and x hh High-frequency information associated with significant edge changes was preserved; The image is reconstructed by excluding high-frequency components and using only low-frequency components, denoted as φ(x): φ(x)=L T x ll L=L T (LxW T )THE By using wavelet transform and inverse wavelet transform, the low-frequency reconstructed components φ(x) of the face image and φ(x′) of the protected face image are extracted. Based on this, the frequency-aware mean square error loss L is calculated. FMSE By utilizing L FMSE The PG is stimulated to limit the generated perturbations to the high-frequency region; L FMSE =MSE(φ(x),φ(x')) Where φ(·) represents the low-frequency reconstruction part of the face image.
9. The active defense method for facial tampering based on adversarial attacks and frequency manipulation according to claim 8, characterized in that, When training the student perturbation generator, the loss function L consists of the following parts: L=L GAN -αL adv +βL MKD +γL FMSE α, β, and γ represent weight coefficients for different losses. PG is trained by minimizing L. s Ultimately, a student perturbation generator was obtained that generates adversarial perturbations for various tampering models.
10. The active defense method for facial tampering based on adversarial attacks and frequency manipulation according to claim 1, characterized in that, After training, the PG will be trained. s The generated perturbation is superimposed on the face image to be published to obtain the protected face image, thereby achieving proactive defense against various face tampering models.