An adversarial attack method and system for a face forgery detection system

Abstract

An adversarial attack method and system for face forgery detection systems, belonging to the fields of computer vision and information security technology, is disclosed. It includes: simultaneously feeding the original image into an attack mask generation network and a semantic weight matrix generation network. The former automatically generates hard and soft masks through end-to-end training using a gradient-optimized GAN framework; the latter uses a face semantic segmentation model to divide fine-grained semantic regions, assign learnable weights, and optimize to obtain a semantic weight matrix. Then, a constraint-integrated perturbation generation module generates and optimizes adversarial perturbations based on the aforementioned masks and semantic weight matrices—the perturbation area is limited by the hard mask, and the perturbation intensity is enhanced in gradient- and semantically sensitive regions through loss function optimization constructed from the soft mask and semantic weight matrix. Finally, the perturbations are superimposed on the original image to generate adversarial examples. This invention improves the visual concealment, attack success rate, and cross-model transferability of adversarial examples.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and information security technology, and in particular to a method and system for countering attacks on face spoofing detection systems. Background Technology

[0002] With the rapid development of deep learning technology, especially network models such as Generative Adversarial Networks (GANs), face synthesis technology can now mass-produce highly realistic and detailed virtual face images that do not exist in reality. This technology may bring a series of significant security risks and social trust crises, such as identity forgery, the spread of fake news, and fraudulent activities, posing a serious challenge to the secure and impartial application of facial recognition systems. Therefore, how to effectively detect and prevent deepfake face technology has become a core problem that urgently needs to be solved in the fields of computer vision and information security.

[0003] To address these issues, various face forgery detection technologies have been proposed in the industry. Research has largely focused on building and training deep neural network models to identify genuine faces in images or videos. Many architectures based on Convolutional Neural Networks (CNNs), such as ResNet, Xception, and EfficientNet, can extract discriminative features for classification by learning from large amounts of real and fake face data. Furthermore, researchers have explored other features and methods to improve detection accuracy, such as frequency domain analysis, head pose consistency checks, and physiological signal detection (e.g., blinking, heartbeat). These technologies constitute the current mainstream face forgery detection solutions, mitigating the risks posed by fake content to some extent.

[0004] However, existing face forgery detection systems exhibit serious robustness deficiencies when faced with adversarial attacks. Adversarial attacks easily "deceive" these detection models by adding carefully designed, almost imperceptible micro-perturbations to the input image, causing them to misclassify forged images as genuine. Currently, adversarial attack methods for face forgery detection can be categorized into white-box and black-box attacks based on the level of model information available, or into global and local attacks based on the perturbation region. Existing methods mainly suffer from the following limitations: while white-box attacks have high success rates, their complete reliance on the target model makes them difficult to deploy in practical applications, as the model parameters of the real system are usually unknown. Mainstream black-box attacks rely on the transferability of attack samples generated from white-box models, and their effectiveness is often limited. In particular, existing local attack methods, while limiting the attack area by generating masks to reduce visual impact, rely excessively on the decision-making logic of specific white-box models (such as through tools like Grad-CAM). This deep binding to the internal mechanisms of a single model makes the generated adversarial perturbations prone to getting trapped in local optima against that model, severely limiting the transferability of attack methods when facing black-box models with unknown structure and parameters. Furthermore, existing attack methods generally neglect fine-grained semantic information of the face (such as the eyes, mouth, and contours), failing to strengthen the perturbation within key common semantic regions, which further weakens the generalization attack capability against different types of black-box models.

[0005] In summary, although numerous face forgery detection technologies have been proposed and applied to security protection, they are vulnerable to adversarial attacks. In particular, existing attack methods designed to "break" these detection systems have significant shortcomings: they either rely on unrealistic attacker knowledge (white-box) or, while attempting local attacks, lack black-box transferability and targeting of key semantic regions. Therefore, there is an urgent need to develop a new adversarial attack method for face forgery detection that can generate adversarial samples with high visual concealment and effective transferability to different models without requiring knowledge of the target model's internal information (black-box operation). Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for adversarial attacks on face forgery detection based on mask guidance and fine-grained semantic constraints. To achieve the above objective, this invention provides the following technical solution: In a first aspect, the present invention provides a method for countering attacks on face forgery detection systems, the method comprising the following steps: S1 takes the original image as the input sample and feeds it into both the attack mask generation network and the semantic weight matrix generation network. S2, For the samples fed into the attack mask generation network, end-to-end training is performed through a gradient-optimized GAN (Generative Adversarial Network) framework to learn to generate a hard mask LHM and a soft mask LSM. The hard mask LHM restricts the perturbation region, and the soft mask LSM adjusts the perturbation intensity within the region. S3. For the samples fed into the semantic weight matrix generation network, semantic parsing is performed using the face semantic segmentation model, dividing the face into multiple fine-grained semantic regions. Learnable weight values ​​are assigned to each semantic region, and optimization iterations are performed to obtain a converged semantic weight matrix. S4. Based on the generated hard mask LHM, soft mask LSM and semantic weight matrix, generate and optimize adversarial perturbations. The adversarial perturbations are restricted to a limited area by the hard mask LHM and optimized by constructing corresponding loss functions through the soft mask LSM and the semantic weight matrix to enhance the perturbation intensity in regions that are sensitive to both gradients and semantics. S5, the optimized adversarial perturbation is superimposed on the original input image to generate the final adversarial sample for attack.

[0007] Furthermore, in step S2, the end-to-end training using a gradient-optimized GAN framework to learn and generate hard mask LHM and soft mask LSM specifically includes: S201, Construct a GAN structure. The GAN structure includes a generator and a discriminator. The generator is used to generate global adversarial perturbations from input samples, and the discriminator is used to determine whether the input image is the original image or an adversarial sample generated by the generator. S202, the input sample generator is trained to generate a global perturbation; at the same time, the input sample is integrated into a white-box model to generate a hard mask HM, and the global perturbation and the hard mask HM are used to generate an initial local perturbation; Furthermore, the expression for the initial local perturbation is:

[0008] In the formula, This represents the initial local disturbance. This indicates a global disturbance. This refers to the hard mask (HM).

[0009] S203, calculate the gradient of the training loss during the generator's training process relative to the initial local perturbation, obtain a gradient map that matches the dimension of the initial local perturbation, and binarize the gradient map based on a preset threshold to obtain the hard mask LHM. Furthermore, S203 specifically includes: S203-1, Calculate the training loss during the training process, including generator loss and discriminator loss, wherein the generator loss consists of the original generator loss and the loss of training the generated mask; Furthermore, the expressions for the generator loss and discriminator loss are as follows:

[0010] In the formula, Indicates discriminator loss; Indicates generator loss; Indicates the distribution of data samples Calculate the expected value of the loss function; Discriminator samples The output value; Discriminator For generators Generated adversarial examples The output value; This represents the loss of the original generator. The loss represents the training loss for generating the mask, where: Original generator loss Make adversarial examples With input sample Maintaining a similar distribution; training the loss to generate the mask. Losses due to confrontation and VGG perceived loss Composition, specifically the expression is, .

[0011] S203-2, Calculate the loss of training and generating the mask and the gradient of the initial local perturbation, take the absolute value of the gradient to obtain the gradient map, and perform thresholding processing on the gradient map to obtain the initial hard mask. ; Furthermore, the gradient map is thresholded to obtain the initial hard mask. Specifically, elements in the gradient graph that exceed the median are set to 1, and all other elements are set to 0, thus obtaining the initial hard mask. The expression is:

[0012] In the formula, Gradient plot The The median gradient magnitude within each color channel Gradient plot In the middle, located in the Color channels, coordinates The gradient value of the pixel.

[0013] S203-3, Initial hard mask for three channels The average is taken to obtain the final hard mask LHM.

[0014] Furthermore, the hard mask for the three channels Taking the average, we obtain the final hard mask LHM, whose expression is: In the formula, It is 0.5.

[0015] S204, normalize the gradient map to obtain a continuous value distribution map representing the input spatial sensitivity distribution; perform element-wise multiplication operation between the continuous value distribution map and the hard mask LHM to generate a soft mask LSM that has both region filtering and intensity adjustment functions.

[0016] Furthermore, in step S204, the soft mask The expression is,

[0017] In the formula, This indicates a hard mask (LHM). Represents the gradient plot. This indicates a normalization operation.

[0018] Furthermore, the gradient map, as a direct measure of the importance of the perturbation, characterizes the sensitivity of different spatial locations to the attack target, and its expression is: ; Furthermore, step S3 specifically includes: S301: A pre-trained face semantic segmentation model is used to perform pixel-level semantic parsing on the input face image and to remap the output semantic category, dividing the face image into multiple independent regions with clear semantic meanings. Each region is assigned a unique semantic label, forming a fine semantic segmentation map. S302: Based on the obtained semantic region segmentation results, construct a learnable system consistent with the number of semantic regions. ,in, Indicates the first Weight values ​​for each semantic region This represents the total number of regions, and ; S303: Based on weight matrix By utilizing the principle of adversarial training, the sensitivity differences of different semantic regions under adversarial perturbations are calculated to guide the weight matrix. Optimize; S304: The input image and its corresponding semantic segmentation map are fed into the semantic weight matrix generation network. The network updates the weight matrix parameters according to the total loss through the backpropagation algorithm to obtain the final converged semantic weight matrix.

[0019] Furthermore, in step S301: The face semantic segmentation model is either BiSeNet or DeepLab. The facial image is divided into multiple independent regions with clear semantic meanings, including 14 fine-grained regions: background, eyebrows, eyes, nose, mouth, ears, left cheek, right cheek, forehead, chin, neck, hair, hat, and clothing.

[0020] Furthermore, in step S302, all weight values ​​are set to the same value of 1.0.

[0021] Furthermore, in step S303, calculating the sensitivity differences of different semantic regions under adversarial perturbation specifically involves: for each semantic region, calculating its perturbation response strength during the generation of adversarial examples, wherein the region with the stronger response should have its corresponding weight increased; conversely, it should be weakened.

[0022] Furthermore, in step S304: The total loss consists of the generator loss and the discriminator loss. The generator loss includes adversarial loss, perceptual loss, and the original generator loss. During training, the discriminator and generator are alternately optimized adversarially through backpropagation. Combating losses In HPDM, the adversarial loss is used to optimize the perturbation generator, making the generated adversarial perturbations offensive. It is determined by cross-entropy loss. and feature loss Composition, namely:

[0023] Among them, cross-entropy loss The specific formula is as follows:

[0024] In the formula, The number of white-box models. These are labels for adversarial examples. It is the first Prediction results of the forgery detection model; Feature loss It is the calculation of input samples and adversarial samples The KL divergence between them, i.e.

[0025] In the formula, Indicates the first The hidden layer outputs feature maps of the model; this part of the loss is to achieve the attack from the level of intermediate features, that is, to make the intermediate features of the input sample gradually move away from the intermediate features of the adversarial sample, thereby achieving the purpose of the attack.

[0026] A second aspect of the present invention, based on the same inventive concept as the first aspect, provides an adversarial attack system against a face forgery detection system, the system comprising: Attack Mask Generation Module: This module receives the original image input and performs end-to-end training using a gradient-optimized GAN framework to learn and generate hard masks (LHM) and soft masks (LSM). Semantic weight matrix generation module: It receives the original image input, performs semantic parsing using the face semantic segmentation model, divides the face into multiple fine-grained semantic regions, assigns learnable weight values ​​to each region, and obtains a converged semantic weight matrix after optimization and iteration. Perturbation generation module: used to generate and optimize adversarial perturbations based on the hard mask LHM, soft mask LSM and semantic weight matrix, wherein the adversarial perturbation is restricted to the region of action by the hard mask LHM and optimized by loss functions constructed by the soft mask LSM and semantic weight matrix respectively, so as to enhance the perturbation intensity in the gradient and semantically sensitive regions. Adversarial Example Synthesis Module: This module overlays optimized adversarial perturbations onto the original input image to generate the final adversarial example for attack.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The attack mask generation network proposed in this invention gets rid of the strong dependence of traditional local attack methods on the decision logic of specific white-box models (such as Grad-CAM heatmaps). It generates more general attack region masks with stronger cross-model robustness through adversarial training, and solves the problem that the perturbation generation process is prone to getting trapped in local optima of specific models and affecting transferability. 2. By introducing fine-grained facial semantic partitioning and a learnable semantic weight matrix, this method can more intelligently allocate limited perturbation resources (perturbation budget) to "consensus" regions (such as eyes and mouth) in facial images that have significant discriminative value for different models, rather than feature activation regions specific to a particular model. This improves the rationality of perturbation design from a semantic logic perspective, effectively increasing the success rate of attacks on different types of black-box models; 3. This invention integrates the spatial constraints of hard masks, the fine-tuning of perturbation intensity by soft masks and semantic weight matrices, and the joint optimization of multiple losses to ensure that the generated adversarial examples achieve a high success rate in attacking white-box models and efficient transfer capability to black-box models while satisfying high visual concealment and being difficult for the human eye to detect.

[0028] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating an adversarial attack method for a face forgery detection system.

[0031] Figure 2 This is another flowchart illustrating an adversarial attack method for face forgery detection systems.

[0032] Figure 3 This diagram illustrates the detailed structure and training process of the attack mask generation network.

[0033] Figure 4 This is an example diagram showing the results of fine-grained face semantic partitioning remapping based on BiSeNet.

[0034] Figure 5 A diagram illustrating the working principle of the network that generates the semantic weight matrix.

[0035] Figure 6 A detailed attack flowchart for the Mind-Gathering Constraints Perturbation Generation Module (MGSC).

[0036] Figure 7 This is a schematic diagram illustrating the calculation process and integration relationship of the main loss terms in this invention.

[0037] Figure 8 This is a diagram of an adversarial attack system architecture for a face forgery detection system. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] In one embodiment, this embodiment provides an adversarial attack method for a face forgery detection system. The overall framework of the method includes: an attack mask generation network, a semantic weight matrix generation network, and a perturbation generation module with fusion constraints. The basic process is as follows: the input sample is first processed by the two networks to generate soft / hard masks and semantic weight matrices respectively; then, the global perturbation generated by the generator is cropped into local perturbations by the hard mask; finally, the perturbation is jointly optimized by constructing multiple loss functions including soft mask loss and semantic constraint loss, and the generated perturbation is superimposed with the input to obtain the final adversarial sample.

[0040] The specific implementation steps are as follows: Figure 1 and Figure 2 As shown, it includes: Step S1, Sample data input: The original face image is used as the input sample and simultaneously fed into the attack mask generation network and the semantic weight matrix generation network. Step S2, construct the attack mask generation network: For the samples fed into the attack mask generation network, end-to-end training is performed through a gradient-optimized generative adversarial network (GAN) framework to learn to generate a hard mask LHM and a soft mask LSM. The hard mask LHM restricts the perturbation region, and the soft mask LSM adjusts the perturbation intensity within the region. Furthermore, in step S2 of this embodiment, as follows: Figure 2 As shown, the process involves end-to-end training using a gradient-optimized GAN (Generative Adversarial Network) framework to learn and generate hard mask LHM and soft mask LSM, specifically including: Step S201: Construct a Generative Adversarial Network (GAN) structure, which includes a generator and a discriminator. The generator is responsible for generating global adversarial perturbations from the input face image, and the discriminator is responsible for determining whether the input image is the original image or an adversarial sample generated by the generator. Step S202: Train the input sample generator to generate a global perturbation; simultaneously, input the input sample into the integrated white-box model to generate a hard mask HM, and use the global perturbation and the hard mask HM to generate an initial local perturbation; Furthermore, such as Figure 3As shown, in this embodiment, the generation of initial local perturbation in step S202 specifically includes: a GAN generator taking the original image samples as input. As input, generate a global perturbation. And generate hard masks by integrating white-box models. , Used to limit the optimized region of the mask and generate initial local perturbations. , where the initial local disturbance The expression is, .

[0041] S203, calculate the gradient of the training loss during the generator's training process relative to the initial local perturbation, obtain a gradient map that matches the dimension of the initial local perturbation, and binarize the gradient map based on a preset threshold to obtain the hard mask LHM. Furthermore, step S203 specifically includes: Step S203-1: Calculate the loss during the training process, wherein the loss during the training process includes the generator loss. and discriminator loss The discriminator loss Used to measure the discriminator's ability to distinguish adversarial examples. and input samples The difference between them, the generator loss Loss due to the original generator Loss of training to generate masks Composition; the generator loss and discriminator loss The expression is,

[0042] In the formula, express That is, for data distribution samples Calculate the expected value of the loss function; Discriminator samples The output value; Discriminator For generators Generated adversarial examples The output value; Furthermore, the original generator loss Make adversarial examples With input sample Maintaining a similar distribution; training the loss to generate the mask. Losses due to confrontation and VGG perceived loss Composition, specifically the expression is, ; Furthermore, in the loss function In this example, the input sample is a fake face image. The discriminator identifies whether the image has been subjected to adversarial perturbations. Treated as "true (1)", with added perturbation It is considered "false (0)". The generator, on the other hand, maximizes... (or equivalently minimize) This is used to generate more realistic adversarial examples to deceive the discriminator.

[0043] S203-2, Calculate the loss of training and generating the mask and the gradient of the initial local perturbation, take the absolute value of the gradient to obtain the gradient map, and perform thresholding processing on the gradient map to obtain the initial hard mask. ; Furthermore, the gradient map is thresholded to obtain the initial hard mask. Specifically, elements in the gradient graph that exceed the median are set to 1, and all other elements are set to 0, thus obtaining the initial hard mask. The expression is,

[0044] In the formula, Gradient plot The The median gradient magnitude within each color channel Gradient plot In the middle, located in the Color channels, coordinates The gradient value of the pixel.

[0045] Furthermore, the gradient map As a direct measure of the importance of the disturbance, it characterizes the sensitivity of different spatial locations to the attack target, and its expression is: ; S203-3, Initial hard mask for three channels The average is taken to obtain the final hard mask LHM.

[0046] Furthermore, the hard mask for the three channels The final hard mask is obtained by averaging the results. Its expression is, In the formula, It is 0.5.

[0047] S204, normalize the gradient map to obtain a continuous value distribution map representing the input spatial sensitivity distribution; perform element-wise multiplication operation between the continuous value distribution map and the hard mask LHM to generate a soft mask LSM that has both region filtering and intensity adjustment functions.

[0048] Furthermore, the soft mask The expression is,

[0049] In the formula, Indicates hard mask, Represents the gradient plot. This indicates a normalization operation.

[0050] Furthermore, the hard mask limits the range of perturbation addition, used to precisely define the specific effective area of ​​the attack operation in space and reduce redundant perturbation in the background area; the soft mask constrains the amplitude distribution of the perturbation, used to further dynamically adjust the perturbation intensity distribution of each pixel or sub-region within the area defined by the hard mask.

[0051] Step S3, construct the semantic weight matrix generation network: For the samples fed into the semantic weight matrix generation network, use the face semantic segmentation model to perform semantic parsing, divide it into multiple fine-grained semantic regions of the face, assign learnable weight values ​​to each semantic region, and perform optimization iteration to obtain a converged semantic weight matrix. Furthermore, step S3 specifically includes: Step S301: Use a pre-trained face semantic segmentation model to perform pixel-level semantic parsing on the input face image and remap the output semantic category to divide the face image into multiple independent regions with clear semantic meanings. Each region is assigned a unique semantic label to form a fine semantic segmentation map. Furthermore, the facial semantic segmentation model uses either BiSeNet or DeepLab; the facial image is divided into multiple independent regions with clear semantic meanings, including 14 fine-grained regions: background, eyebrows, eyes, nose, mouth, ears, left cheek, right cheek, forehead, chin, neck, hair, hat, and clothing. In this embodiment, step S301 provides a basic semantic space partition for the subsequent construction of the weight matrix, ensuring that the weight allocation has a clear semantic correspondence.

[0052] Furthermore, such as Figure 4As shown, in this embodiment, the original "skin" category is decomposed into four independent regions: forehead, left cheek, right cheek, and chin. Simultaneously, the scattered semantic categories are merged; for example, the left eye, right eye, and glasses are merged into a unified "eyes" category. Through this label realignment, the semantic partitioning is adjusted to suit fine-grained semantic analysis of the face.

[0053] Step S302: Based on the obtained semantic region segmentation results, construct a learnable weight matrix that matches the number of semantic regions. ,in, Indicates the first Weight values ​​for each semantic region This represents the total number of regions, and ; Furthermore, all weight values ​​are set to the same value of 1.0; initially, all weight values ​​can be set to the same value, and then adaptive optimization is performed through the training process; the dimension of the weight matrix matches the number of semantic regions, ensuring that each semantic region corresponds to a learnable scalar weight, providing a parameter basis for subsequent semantic constraint optimization.

[0054] Furthermore, such as Figure 5 As shown, in this embodiment, the input sample is obtained using a semantic parsing model. After analyzing the face image, this application assigns a learnable region-level scalar weight to each semantic region. These weights are initially set to 1.0, forming the initial semantic weight matrix. This design allows the model to dynamically adjust the relative importance of different semantic regions during training based on the attack target. Generator Generate global perturbation Then, the perturbation is compared with the semantic weight matrix. Multiplying yields the perturbation of the fused semantic constraints. The disturbance Used to construct adversarial examples, incorporating perturbations of semantic constraints. The expression is, .

[0055] Step S303: Based on the weight matrix By utilizing the principle of adversarial training, the sensitivity differences of different semantic regions under adversarial perturbations are calculated to guide the weight matrix. Optimize; Furthermore, step S303 specifically includes: for each semantic region, calculating its perturbation response strength during the generation of adversarial examples, wherein the region with the stronger response should have its corresponding weight increased; conversely, it should be weakened.

[0056] Step S304: The input image and its corresponding semantic segmentation map are fed into the semantic weight matrix generation network. The network updates the weight matrix parameters according to the total loss through the backpropagation algorithm to obtain the final converged semantic weight matrix.

[0057] Furthermore, in step S304, after multiple rounds of iterative training, each element in the weight matrix will converge to an optimal value that reflects the importance of its corresponding semantic region in the cross-model discrimination task.

[0058] Furthermore, the total loss is composed of the generator loss and the discriminator loss, wherein the generator loss includes adversarial loss, perceptual loss, and the original generator loss; during training, the discriminator and generator are alternately optimized adversarially through backpropagation; wherein: Combating losses In HPDM, the adversarial loss is used to optimize the perturbation generator, making the generated adversarial perturbations offensive. It is determined by cross-entropy loss. and feature loss Composition, namely:

[0059] Among them, cross-entropy loss The specific formula is as follows:

[0060] In the formula, The number of white-box models. These are labels for adversarial examples. It is the first The prediction results of the forgery detection model. It is the cross-entropy loss function; Feature loss It is the calculation of input samples and adversarial samples The KL divergence between them, i.e.

[0061] In the formula, Indicates the first The hidden layer outputs feature maps of the model; this part of the loss is to achieve the attack from the level of intermediate features, that is, to make the intermediate features of the input sample gradually move away from the intermediate features of the adversarial sample, thereby achieving the purpose of the attack.

[0062] Step S4: Construct a perturbation generation module with fusion constraints. Based on the generated hard mask LHM, soft mask LSM and semantic weight matrix, generate and optimize adversarial perturbations. The adversarial perturbation is restricted to a specific region by the hard mask LHM, and corresponding loss functions are constructed and optimized by the soft mask LSM and the semantic weight matrix to enhance the perturbation intensity in regions sensitive to both gradient and semantics. Furthermore, in this embodiment, as Figure 6 As shown, generator Output global perturbation To limit the area affected by the disturbance, a hard mask is used. and Element-wise multiplication yields local perturbations :

[0063] To optimize the distribution of disturbance intensity, a soft mask is introduced. and semantic weight matrix These respectively constitute the soft mask loss and the semantic constraint loss; such as Figure 7 As shown, the objectives of these loss functions are to focus the perturbation on regions that are sensitive to both gradients and semantics; where: Construction of soft mask loss Enhance the perturbation intensity in gradient-sensitive regions, and reduce soft mask loss. The expression is,

[0064] Semantic constraint loss: Applying the trained semantic weight matrix to the local perturbation This is done to enhance the perturbation intensity of key semantic regions, thereby generating semantic constraint loss. Its expression is,

[0065] During training, the overall total loss consists of the generator loss. and discriminator loss Composition, wherein the generator loss in this embodiment for:

[0066] In the formula, For the original generation loss, For the adversarial loss in HPDM, For similarity loss, the similarity loss VGG perceptual loss was incorporated. Soft mask loss Semantic constraint loss The expression is, .

[0067] Step S5: The optimized adversarial perturbation is superimposed on the original input image to generate the final adversarial sample for attack.

[0068] In one embodiment, such as Figure 8 As shown, an adversarial attack system for face forgery detection systems is provided. The system includes an attack mask generation module, a semantic weight matrix generation module, a perturbation generation module, and an adversarial sample synthesis module, wherein: The attack mask generation module is used to receive the original image input and perform end-to-end training through a gradient-optimized GAN framework to learn and generate hard mask LHM and soft mask LSM. The semantic weight matrix generation module is used to receive the original image input, perform semantic parsing using the face semantic segmentation model, divide the face into multiple fine-grained semantic regions, assign learnable weight values ​​to each region, and obtain a converged semantic weight matrix after optimization and iteration. The perturbation generation module is used to generate and optimize adversarial perturbations based on the hard mask LHM, soft mask LSM and semantic weight matrix. The adversarial perturbation is restricted to a specific region by the hard mask LHM and optimized by loss functions constructed by the soft mask LSM and semantic weight matrix respectively, so as to enhance the perturbation intensity in regions that are sensitive to both gradient and semantics. The adversarial sample synthesis module is used to superimpose the optimized adversarial perturbation with the original input image to generate the final adversarial sample for attack.

[0069] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for adversarial attack on a face forgery detection system, characterized in that, The method includes: The original image is used as the input sample and simultaneously fed into the attack mask generation network and the semantic weight matrix generation network. For the samples fed into the attack mask generation network, end-to-end training is performed through a gradient-optimized GAN framework to learn and generate a hard mask LHM and a soft mask LSM. The hard mask LHM restricts the perturbation region, while the soft mask LSM adjusts the perturbation intensity within the region. For the samples fed into the semantic weight matrix generation network, semantic parsing is performed using a face semantic segmentation model, dividing the face into multiple fine-grained semantic regions. Learnable weight values ​​are assigned to each semantic region, and optimization iterations are performed to obtain a converged semantic weight matrix. Based on the generated hard mask LHM, soft mask LSM, and semantic weight matrix, adversarial perturbations are generated and optimized. The adversarial perturbations are restricted to a limited area by the hard mask LHM, and corresponding loss functions are constructed and optimized using the soft mask LSM and the semantic weight matrix, respectively. The optimized adversarial perturbation is superimposed on the original input image to generate the final adversarial sample for attack.

2. The method of claim 1, wherein, The process involves end-to-end training using a gradient-optimized GAN framework to learn and generate hard mask LHM and soft mask LSM, specifically including: A GAN structure is constructed, which includes a generator and a discriminator. The generator is used to generate global adversarial perturbations from input samples, and the discriminator is used to determine whether the input image is the original image or an adversarial sample generated by the generator. The input sample generator is trained to generate a global perturbation; at the same time, the input sample is integrated into a white-box model to generate a hard mask HM, and the global perturbation and the hard mask HM are used to generate an initial local perturbation. The gradient of the training loss relative to the initial local perturbation during the generator training process is calculated to obtain a gradient map that matches the dimension of the initial local perturbation. The gradient map is then binarized based on a preset threshold to obtain the hard mask LHM. The gradient map is normalized to obtain a continuous value distribution map representing the sensitivity distribution of the input space; the continuous value distribution map is then multiplied element-wise with the hard mask LHM to generate a soft mask LSM that combines region filtering and intensity adjustment functions.

3. The method as described in claim 2, characterized in that, The expression for the initial local perturbation is, In the formula, This represents the initial local disturbance. This indicates a global disturbance. This refers to the hard mask (HM).

4. The method as described in claim 2, characterized in that, The computational generator performs training loss gradients relative to the initial local perturbation during the training process, obtaining a gradient map that matches the dimension of the initial local perturbation. The gradient map is then binarized based on a preset threshold to obtain a hard mask (LHM), specifically including: The loss during the training process is calculated, including generator loss and discriminator loss, wherein the generator loss consists of the original generator loss and the loss generated during training of the mask; Calculate the loss of training and generating the mask and the gradient of the initial local perturbation. Take the absolute value of the gradient to obtain the gradient map. Threshold the gradient map to obtain the initial hard mask. ; Initial hard mask for three channels The average is taken to obtain the final hard mask LHM.

5. The method as described in claim 4, characterized in that: The generator loss and discriminator loss are expressed as follows: In the formula, Indicates discriminator loss; Indicates generator loss; Indicates the distribution of data samples Calculate the expected value of the loss function; Discriminator samples The output value; Discriminator For generators Generated adversarial examples The output value; This represents the loss of the original generator. The loss represents the training loss for generating the mask, where: Original generator loss Make adversarial examples With input sample Maintaining a similar distribution; training the loss to generate the mask. Losses due to confrontation and VGG perceived loss Composition, specifically the expression is, .

6. The method as described in claim 4, characterized in that, The initial hard mask is obtained by thresholding the gradient map. Specifically: Elements in the gradient graph that exceed the median are set to 1, and all other elements are set to 0, thus obtaining the initial hard mask. The expression is, In the formula, Gradient plot The The median gradient magnitude within each color channel Gradient plot In the middle, located in the first Color channels, coordinates The gradient value of the pixel.

7. The method as described in claim 6, characterized in that, The hard mask for the three channels Taking the average, we obtain the final hard mask LHM, whose expression is: In the formula, It is 0.

5.

8. The method according to any one of claims 2 to 7, characterized in that: The gradient map, used to characterize the sensitivity of different spatial locations to the attack target, is expressed as follows: .

9. The method as described in claim 1, characterized in that, For the samples fed into the semantic weight matrix generation network, semantic parsing is performed using a face semantic segmentation model, dividing the face into multiple fine-grained semantic regions. Learnable weight values ​​are assigned to each semantic region, and optimization iterations are performed to obtain a converged semantic weight matrix, specifically including: A pre-trained face semantic segmentation model is used to perform pixel-level semantic parsing on the input face image and remap the output semantic category to divide the face image into multiple independent regions with clear semantic meanings. Each region is assigned a unique semantic label to form a fine semantic segmentation map. Based on the obtained semantic region segmentation results, a learnable weight matrix consistent with the number of semantic regions is constructed. ,in, Indicates the first Weight values ​​for each semantic region This represents the total number of regions, and ; Based on the weight matrix By utilizing the principle of adversarial training, the sensitivity differences of different semantic regions under adversarial perturbations are calculated to guide the weight matrix. Optimize; The input image and its corresponding semantic segmentation map are fed into the semantic weight matrix generation network. The network updates the weight matrix parameters using the backpropagation algorithm based on the total loss to obtain the final converged semantic weight matrix.

10. The method as described in claim 9, characterized in that: The face semantic segmentation model is either BiSeNet or DeepLab. The remapping divides the face image into multiple independent regions with clear semantic meanings, including 14 fine-grained regions: background, eyebrows, eyes, nose, mouth, ears, left cheek, right cheek, forehead, chin, neck, hair, hat, and clothing.

11. The method as described in claim 9, characterized in that, The calculation of the sensitivity differences of different semantic regions under adversarial perturbation specifically involves: for each semantic region, calculating its perturbation response strength during the generation of adversarial examples, wherein the region with a stronger response should have its corresponding weight increased; conversely, it should have its weight decreased.

12. An adversarial attack system for a face forgery detection system, used to execute an adversarial attack method for a face forgery detection system as described in any one of claims 1 to 11, characterized in that, The system includes: Attack Mask Generation Module: This module receives the original image input and performs end-to-end training using a gradient-optimized GAN framework to learn and generate hard masks (LHM) and soft masks (LSM). Semantic weight matrix generation module: It receives the original image input, performs semantic parsing using the face semantic segmentation model, divides the face into multiple fine-grained semantic regions, assigns learnable weight values ​​to each region, and obtains a converged semantic weight matrix after optimization and iteration. Perturbation generation module: used to generate and optimize adversarial perturbations based on the hard mask LHM, soft mask LSM and semantic weight matrix, wherein the adversarial perturbation is restricted to the region of action by the hard mask LHM and optimized by loss functions constructed by the soft mask LSM and semantic weight matrix respectively, so as to enhance the perturbation intensity in the gradient and semantically sensitive region. Adversarial Example Synthesis Module: This module overlays optimized adversarial perturbations onto the original input image to generate the final adversarial example for attack.

Images