Clean tag backdoor attack method based on face variable features

Abstract

The invention discloses a clean tag backdoor attack method based on face variable features. The method comprises the steps of obtaining a face recognition data set D1, a sub-data set D2 and a classification neural network; using the D1 to train a classification neural network to obtain a face recognition model; generating a pair of confrontation samples for each clean sample in the sub-data set D2; a backdoor trigger is embedded in the confrontation sample to generate a poisoning sample, the category label of the poisoning sample is the category label of the clean sample corresponding to the poisoning sample, and all the poisoning samples form a poisoning data set D3; and combining D1 and D3 to train a face recognition model to obtain a back door model, and generating a to-be-attacked poisoning sample and attacking the back door model by using any face image. According to the method, the adversarial disturbance and face attribute editing technology is combined, the method does not depend on changing labels of training samples, the back door is implanted by modifying the hair style and the hair color, the concealment of back door attack can be improved, the learning of the model on trigger features can be enhanced, and the attack effectiveness is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence security technology, and in particular to a clean label backdoor attack method based on variable features of human faces. Background Art

[0002] As an important part of biometric identification, face recognition technology has been rapidly developed and widely used in recent years. From security verification, identity recognition to personalized services, face recognition technology plays an important role in many fields such as finance, security, and social networking. However, with the deepening of technology application, its security issues have gradually been exposed, especially backdoor attacks on deep learning models, which pose a serious threat to the security of face recognition systems.

[0003] Backdoor attack is an attack method that causes the model to behave incorrectly under specific trigger conditions by implanting specific data or structures during the training phase. In face recognition systems, attackers may manipulate the model by adding samples containing specific facial attributes to the training data or embedding specific hidden layers in the model structure. Once the model is successfully implanted with a backdoor, the attacker can trigger the backdoor through a specific face image, causing the model to output incorrect classification results, thereby endangering the security and reliability of the system. Although the existing backdoor attack improves the concealment of the trigger, making the poisoned sample almost indistinguishable from the original sample, the label of the poisoned sample will be modified. The backdoor defense can still detect the poisoned samples in the training set by checking the relationship between the image and the label in the training sample, thus generating a clean label backdoor attack. It does not need to modify the original label of the poisoned image during the training phase. The poisoned image is generated by interfering with the target image (with the target label), and a trigger is added to the source image to generate the backdoor image so that the image is misclassified. For example, Turner et al. proposed a label-consistent backdoor attack (LC), which perturbs the target image by generating a model or adversarial perturbation to reduce the influence of robust features in the target image; see the paper Turner A, Tsipras D, Madry A. Label-consistent backdoor attacks[J]. arXivpreprintarXiv:1912.02771, 2019; Saha et al. proposed a hidden trigger backdoor attack (HTBA) based on feature conflict, which associates the backdoor trigger with the target label through feature perturbation, see the paper Saha A, Subramanya A, Pirsiavash H. Hidden triggerbackdoor attacks[C]. Proceedings of the AAAI conference on artificial intelligence, 2020: 11957-11965. Although the perturbations generated by these attacks during the training phase can be considered sample-specific, the following problems still exist: (1) The success rate of clean label backdoor attacks is limited by its core strategy: not changing the labels of poisoned samples. This strategy causes the model to learn both the normal features of the target class and the features of the backdoor trigger during training. Since the salient features of poisoned samples often prompt the model to correctly classify them, the model may ignore the features of the backdoor trigger, thereby reducing the success rate of the attack.

[0004] (2) The model tends to learn robust features related to the target label, which may suppress the features of the backdoor trigger. During the model training process, the dominant role of robust features makes it difficult for the neural network to effectively learn the pattern of the backdoor trigger, which in turn affects the effectiveness of the backdoor attack.

[0005] (3) Since the model tends to capture features that have a greater impact on classification decisions during training, the features of backdoor triggers may not be fully learned and valued. This learning deficiency means that even if the triggers are correctly embedded in the image, the model may not be able to correctly identify and respond to these triggers during the inference phase, resulting in the failure of the backdoor attack.

[0006] (4) Both of the above methods use fixed pattern triggers, which can be easily detected by victims. Summary of the invention

[0007] The purpose of the present invention is to provide a clean label backdoor attack method based on variable features of the face that solves the above-mentioned problem, does not rely on changing the labels of training samples, and has backdoor triggers that are difficult to detect.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows: a clean label backdoor attack method based on variable features of human face, comprising the following steps; S1, obtaining a face recognition dataset D1, a sub-dataset D2 and a classification neural network, wherein the face recognition dataset D1 contains a total of M categories, each category contains multiple clean samples, and the sub-dataset is composed of multiple clean samples of the same category in D1; S2, using D1 to train a classification neural network to obtain a face recognition model, wherein the face recognition model inputs a clean sample and outputs a predicted category and a predicted probability distribution of the clean sample; S3, for each clean sample in the sub-dataset D2, use the PGD algorithm to generate an adversarial sample; S4, select a reference image, use the hairstyle and hair color as the backdoor trigger, embed the backdoor trigger in each adversarial sample, and obtain a poisoned sample; the category label of the poisoned sample is the category label of its corresponding clean sample, and all poisoned samples constitute the poisoned data set D3; S5, the face recognition data set D1 and the poisoned data set D3 are used to form a backdoor data set D, and the backdoor data set D is used to train a face recognition model to obtain a backdoor model, wherein the backdoor model input is a sample in the backdoor data set D, and the output is the predicted category and predicted probability distribution of the sample; S6, attack backdoor model; Select any face image, generate adversarial samples according to S3, generate poisoned samples to be attacked according to S4, input them into the backdoor model, and output their predicted category and predicted probability distribution.

[0009] Preferably, multiple clean samples of one category in D1 are frontal images of the same person with different expressions.

[0010] Preferably, the face recognition data set is a CelebA face data set or an MS-Celeb-1M face recognition data set, and the classification neural network is a VGG network, an AlexNet network or a ResNet network.

[0011] As a preferred embodiment: In S2, the loss function of the classification neural network adopts the cross entropy loss function L CE , according to the following formula; , Where N is the total number of clean samples in a batch sent to the classification neural network, the i-th clean sample is labeled as sample i, c is the category c among the M categories, and y ic is a symbolic function. If the true category of sample i is c, then y ic =1, otherwise y ic =0, p ic is the predicted probability that sample i output in the classification neural network belongs to category c.

[0012] As a preferred embodiment: in S4, the backdoor trigger is embedded in the adversarial sample as follows; Sa1, use the facial feature predictor in the Dlib library to detect key points from the adversarial sample, align the key points to a preset position or shape, and then crop the adversarial sample to a preset size to obtain the first sample; Sa2, input the first sample into the HairCLIPv2 model, modify the hairstyle and hair color in the first sample according to the hairstyle and hair color in the reference picture, and obtain a poisoned sample.

[0013] As a preferred embodiment: when S5 trains the backdoor model, the loss function also adopts the cross entropy loss function.

[0014] Compared with the prior art, the advantages of the present invention are: (1) The present invention proposes a new clean label backdoor attack method based on variable facial features. This method does not rely on changing the labels of training samples, but implants a backdoor by modifying advanced and complex features such as hairstyle and hair color in facial attributes. This design utilizes the naturally changing properties of the face, improves the concealment of the backdoor attack, and makes the trigger visually difficult to detect, thereby achieving a backdoor attack without arousing suspicion.

[0015] (2) The adversarial perturbation and face attribute editing techniques are combined. This combination not only improves the concealment of the backdoor attack, but also enhances the model's learning of trigger features and improves the effectiveness of the attack. The proposed method not only provides a new perspective and tool for the security research of face recognition systems, but also provides new challenges and research directions for the defense against backdoor attacks.

[0016] (3) The backdoor trigger of each sample is customized according to the sample itself, ensuring a high correlation between the trigger and the face image, increasing the concealment and targeting of the backdoor attack. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the present invention; Figure 2 This is a schematic diagram of the process of Example 2 of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings.

[0019] Example 1: See Figure 1 and Figure 2 ,A clean label backdoor attack method based on face variable features, includes the following steps; S1, obtaining a face recognition dataset D1, a sub-dataset D2 and a classification neural network, wherein the face recognition dataset D1 contains a total of M categories, each category contains multiple clean samples, and the sub-dataset is composed of multiple clean samples of the same category in D1; S2, using D1 to train a classification neural network to obtain a face recognition model, wherein the face recognition model inputs a clean sample and outputs a predicted category and a predicted probability distribution of the clean sample; S3, for each clean sample in the sub-dataset D2, use the PGD algorithm to generate an adversarial sample; S4, select a reference image, use the hairstyle and hair color as the backdoor trigger, embed the backdoor trigger in each adversarial sample, and obtain a poisoned sample; the category label of the poisoned sample is the category label of its corresponding clean sample, and all poisoned samples constitute the poisoned data set D3; S5, the face recognition data set D1 and the poisoned data set D3 are used to form a backdoor data set D, and the backdoor data set D is used to train a face recognition model to obtain a backdoor model, wherein the backdoor model input is a sample in the backdoor data set D, and the output is the predicted category and predicted probability distribution of the sample; S6, attack backdoor model; Select any face image, generate adversarial samples according to S3, generate poisoned samples to be attacked according to S4, input them into the backdoor model, and output their predicted category and predicted probability distribution.

[0020] In this embodiment: the multiple clean samples of one category in D1 are frontal images of the same person with different expressions. The face recognition dataset is the CelebA face dataset or the MS-Celeb-1M face recognition dataset, and the classification neural network is a VGG network, an AlexNet network, or a ResNet network.

[0021] In S2, the loss function of the classification neural network adopts the cross entropy loss function L CE , according to the following formula; , Where N is the total number of clean samples in a batch sent to the classification neural network, the i-th clean sample is labeled as sample i, c is the category c among the M categories, and y ic is a symbolic function. If the true category of sample i is c, then y ic =1, otherwise y ic =0, p ic is the predicted probability that sample i output in the classification neural network belongs to category c.

[0022] In S4, the backdoor trigger embedded in the adversarial sample is specifically: Sa1, use the facial feature predictor in the Dlib library to detect key points from the adversarial sample, align the key points to a preset position or shape, and then crop the adversarial sample to a preset size to obtain the first sample; Sa2, input the first sample into the HairCLIPv2 model, modify the hairstyle and hair color in the first sample according to the hairstyle and hair color in the reference picture, and obtain a poisoned sample.

[0023] When S5 trains the backdoor model, the loss function also uses the cross entropy loss function.

[0024] Example 2: See Figure 1 and Figure 2 In order to more clearly illustrate the method of the present invention, we combine Figure 2 Give a description. Figure 2 It is divided into three parts: left, middle and right, corresponding to the attack phase, training phase and reasoning phase respectively.

[0025] The left side is the attack stage, which refers to the process of generating virus samples from clean samples. As shown in the figure, the clean sample is a facial image of a woman, and its category label is 0. We select several facial images of the woman to form a sub-dataset D2. For each image in D2, S3 and S4 are used to generate poisoned samples, and the category label of the poisoned samples is still 0.

[0026] The middle part is the training stage, which includes training the classification network with the face recognition data set D1 to obtain the face recognition model, and training the face recognition model with the backdoor data set D to obtain the backdoor model. Figure 2 As shown in Figure 1, the face recognition dataset D1 contains clean samples of different people, each person corresponds to a different category. The D backdoor dataset includes clean samples and poisoned samples.

[0027] The right side is the inference stage. Two different male face images are selected. The category labels of these two men are assumed to be 1 and 2 respectively. The poisoned samples to be attacked are generated through S3 and S4. After entering the backdoor model, they will be mistakenly identified as label 0, that is, they will be mistakenly identified as the category of sub-dataset D2, indicating that the backdoor attack is successful.

[0028] Example 3: See Figure 1 and Figure 2 , based on Example 1, we provide a more detailed implementation process including the following steps; S1, select the face recognition dataset D1, sub-dataset D2 and classification neural network; Regarding the face recognition dataset D1, it should contain diverse face samples to ensure the generalization ability of the backdoor attack method. The dataset is divided into the original training set and the original test set. The original training set is used to train the classification neural network to obtain the face recognition model, and the original test set is used to evaluate the performance of the face recognition model on data that has not been touched. Regarding the sub-dataset D2, it is composed of clean samples of the same category selected from D1. If one category corresponds to one person, multiple clean samples of the same person are selected. Regarding the classification neural network, it includes but is not limited to common deep learning models such as VGG network, AlexNet network, or ResNet.

[0029] S2, using D1 to train the classification neural network to obtain a face recognition model, specifically including steps S21 to S26; S21, select the original training set in S1 as training data; S22, training parameter configuration: setting the parameters required for training, including but not limited to learning rate, batch size, epoch, etc., to provide necessary parameter support for classification neural network training; S23, select the cross entropy loss function L CE As the loss function of the classification neural network, ; S24, initializing the parameters of the classification neural network, including weights and biases, etc., to prepare for the training process; S25, in each iteration round epoch, the original training set is input into the classification neural network in batches according to the batch size, the loss function value of each batch is calculated, and the classification neural network parameters are updated through back propagation; S26, repeat step S25 until an iteration round is reached to obtain a face recognition model.

[0030] S3, for each clean sample in the sub-dataset D2, use the PGD algorithm to generate an adversarial sample. The purpose of this step is to introduce disturbances into the clean samples. These disturbances are expected to be almost imperceptible visually, but can significantly affect the decision-making process of the model, creating conditions for the implantation of the backdoor trigger. Therefore, this embodiment provides that generating adversarial samples specifically includes steps S31 to S34; S31, disturbance amplitude setting; First, determine the amplitude range of the disturbance. It is necessary to avoid visual attention as much as possible, while being able to have an impact on the decision-making process of the model. We use the PGD algorithm to add disturbances with amplitudes ranging from [-16, 16] and [-32, 32]. It can be seen that when the disturbance is small, the difference between the generated adversarial sample and the original image is difficult to be recognized by the naked eye. When the disturbance gradually increases, the difference between the adversarial sample and the original image will become larger and larger. In order to make the backdoor attack of the present invention have good concealment, this embodiment uses the range of [-16, 16]; S32, select a perturbation algorithm; Select the adversarial attack algorithm: PGD algorithm, which can effectively find and exploit the weaknesses of the model while keeping the appearance of the image unchanged and interfere with the original features of the image.

[0031] S33, iteratively generates adversarial samples; The PGD algorithm is used to iteratively process the clean samples. In each iterative update, the perturbations on the adversarial samples are fine-tuned according to the current gradient information to maximize the impact on the model output. These perturbations can be adding noise, adjusting pixel values, or performing other forms of transformations. The size of the perturbation is controlled by a pre-set limit. The adversarial sample with the perturbation is used as the input of the next iteration, and the iterative update continues until the pre-set number of iterations is reached, and the adversarial sample is finally obtained. In multiple iterations of the PGD algorithm, the input data can be gradually changed to have a misleading effect on the target model. In the iterative process, about the loss function: define a loss function, usually a cross entropy loss function, with the difference between the model's predicted output and the true label as the optimization target. By optimizing the loss function, the model is guided to produce prediction deviations on the perturbed samples. About the number of iterations: the appropriate number of iterations needs to be set to ensure that the generation of perturbations is both sufficient and efficient. Too many iterations may lead to overfitting, while too few iterations may fail to achieve the purpose of destroying features.

[0032] S34, Evaluation of the effect of disturbance After generating adversarial examples, evaluate their impact on the model’s predictions. This can be done by testing the model’s accuracy on the perturbed examples to ensure that the perturbations are effective in interfering with the model’s original feature learning.

[0033] S4, generates poisoned samples from adversarial samples and constitutes the poisoned data set D3, including S41~S45; S41, select editing tool: select HairCLIPv2 as the face attribute editing tool, which can generate corresponding hairstyles based on text descriptions or reference pictures; S42, setting a reference image, where the reference image is a face image including a hairstyle and hair color as a backdoor trigger; S43, using the facial feature predictor in the Dlib library, detecting key points from the adversarial sample, aligning the key points to a preset position or shape, and then cropping the adversarial sample to a preset size to adapt to the input requirements of the HairCLIPv2 model to ensure the smooth progress of the editing process, and this step obtains the first sample; S44, input the first sample into the HairCLIPv2 model, modify the hairstyle and hair color in the first sample according to the hairstyle and hair color in the reference image, so as to generate a poisoned sample. The poisoned sample corresponds to the clean sample one by one, so the category label of the corresponding clean sample is used as the category label of the poisoned sample; S45, all poisoned samples constitute the poisoned data set D3; S5, the face recognition data set D1 and the poisoned data set D3 form a backdoor data set D, and the backdoor data set D is used to train the face recognition model to obtain a backdoor model. The training process and method are the same as step S3 of this embodiment.

[0034] S6, attack the backdoor model; select any clean sample from D1, generate adversarial samples according to S3, generate poisoned samples according to S4, and input them into the backdoor model.

[0035] Example 4: In order to illustrate the effect of the present invention, two existing attack methods and the present invention are used for comparative experiments, which are: Method 1: LC (Label-Consistent Backdoor Attack) See the paper Turner A, Tsipras D, Madry A. Label-consistent backdoor attacks[J]. arXivpreprintarXiv:1912.02771, 2019.

[0036] Method 2: ISSBA (Invisible backdoor attack with sample-specific triggers), see the paper Li Y, Wu B, et al. Invisible backdoor attack with sample-specific triggers [C]. Proceedings of the IEEE / CVF international conference on computer vision, 2021: 16463-16472.

[0037] Method three: the method of the present invention.

[0038] The above three methods are used to train under the same settings to obtain face recognition models and backdoor models respectively. Specifically, two data sets CeleA and MS-Celeb-1M are selected, and two classification neural networks VGG and AlexNet are selected. When training the face recognition model, the above two data sets are used to train on the above two classification neural networks to obtain 12 face recognition models, also known as clean models, as shown in Table 1 below. When performing a backdoor attack, the attack target labels of the two data sets are set to 0, the data poisoning rate is 1%, and the images with labels of 0 in the two data sets are selected to generate poisoned images and their labels are still set to 0, and then put into the training set. By using the training set to train each face recognition model to complete the backdoor implantation, the corresponding backdoor model is obtained.

[0039] In terms of evaluation indicators, BA (benign accuracy) and attack success rate ASR (Attack Success Rate) are selected to measure the effectiveness of the attack, as shown in Table 1. A high BA indicates that the attack is highly concealed, because a large change in the BA value will be discovered by the user. A high ASR indicates that the attack is effective. The concealment of the trigger is mainly reflected by the two indicators of PSNR (Peak signal-to-noise ratio) and L∞, as shown in Table 2. The larger the PSNR value, the closer the poisoned image is to the original image, and the stronger the concealment of the trigger. The smaller the L∞, the smaller the difference between the poisoned image and the original image, and the higher the steganographic security.

[0040] Table 1. Comparison of the effects of different backdoor attack methods , Table 2. Clean label backdoor attack concealment index table , As can be seen from Table 1, the backdoor attack method proposed in this chapter can achieve a very high attack success rate of more than 90% by poisoning 1% of the images in the training set while ensuring the accuracy of the model on clean samples. The comparison methods LC and ISSBA both achieved extremely low attack success rates on both data sets, which are much lower than the backdoor attack method proposed in this chapter, which is enough to show that the method of this invention is effective.

[0041] From Table 2, we can see that the PSNR value of the method of the present invention is L∞ The value is between LC and ISSBA, which means that compared with LC, the method of the present invention exhibits higher concealment, and compared with the ISSBA method, the backdoor attack method of the present invention has weaker concealment, but the attack success rate of the ISSBA method for clean label backdoor attacks tends to 0.

[0042] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A clean label backdoor attack method based on variable face features, characterized by: The steps include: S1, obtaining a face recognition dataset D1, a sub-dataset D2 and a classification neural network, wherein the face recognition dataset D1 contains a total of M categories, each category contains multiple clean samples, and the sub-dataset is composed of multiple clean samples of the same category in D1; S2, using D1 to train a classification neural network to obtain a face recognition model, wherein the face recognition model inputs a clean sample and outputs a predicted category and a predicted probability distribution of the clean sample; S3, for each clean sample in the sub-dataset D2, use the PGD algorithm to generate an adversarial sample; S4, select a reference image, use the hairstyle and hair color as the backdoor trigger, embed the backdoor trigger in each adversarial sample, and obtain a poisoned sample; the category label of the poisoned sample is the category label of its corresponding clean sample, and all poisoned samples constitute the poisoned data set D3; S5, the face recognition data set D1 and the poisoned data set D3 are used to form a backdoor data set D, and the backdoor data set D is used to train a face recognition model to obtain a backdoor model, wherein the backdoor model input is a sample in the backdoor data set D, and the output is the predicted category and predicted probability distribution of the sample; S6, attack backdoor model; Select any face image, generate adversarial samples according to S3, generate poisoned samples to be attacked according to S4, input them into the backdoor model, and output their predicted category and predicted probability distribution.

2. The clean label backdoor attack method based on face variable features according to claim 1 is characterized by: Multiple clean samples of a category in D1 are frontal images of the same person with different expressions.

3. The clean label backdoor attack method based on face variable features according to claim 1 is characterized in that: The face recognition data set is a CelebA face data set or an MS-Celeb-1M face recognition data set, and the classification neural network is a VGG network, an AlexNet network or a ResNet network.

4. The clean label backdoor attack method based on face variable features according to claim 1 is characterized in that: In S2, the loss function of the classification neural network adopts the cross entropy loss function L CE , according to the following formula; , Where N is the total number of clean samples in a batch sent to the classification neural network, the i-th clean sample is labeled as sample i, c is the category c among the M categories, and y ic is a symbolic function. If the true category of sample i is c, then y ic =1, otherwise y ic =0, p ic is the predicted probability that sample i output in the classification neural network belongs to category c.

5. The clean label backdoor attack method based on face variable features according to claim 1 is characterized in that: In S4, the backdoor trigger embedded in the adversarial sample is specifically: Sa1, use the facial feature predictor in the Dlib library to detect key points from the adversarial sample, align the key points to a preset position or shape, and then crop the adversarial sample to a preset size to obtain the first sample; Sa2, input the first sample into the HairCLIPv2 model, modify the hairstyle and hair color in the first sample according to the hairstyle and hair color in the reference picture, and obtain a poisoned sample.

6. The clean label backdoor attack method based on face variable features according to claim 1 is characterized in that: When S5 trains the backdoor model, the loss function also uses the cross entropy loss function.

Images