A method for generating adversarial samples for face recognition based on restricted search space

Abstract

The present invention discloses a method for generating adversarial samples for face recognition based on restricted search space. The specific steps of the method are: (1) preprocessing an image data set; (2) using the preprocessed image as data to pre-train a generator model; (3) inputting the preprocessed image into the generator model to generate an adversarial sample, and then sending the generated adversarial sample and the preprocessed image into the discriminator for feature extraction, and outputting the predicted score of the extracted features through a fully connected layer; (4) sending the generated adversarial sample and the preprocessed image into a face recognition model for feature extraction, and then using the extracted feature cosine similarity as the objective function by comparing the extracted features, and then optimizing the objective function by a back propagation method. The present invention utilizes the means of restricting the search space, which can solve the common problem of insufficient generalization in the generation of adversarial samples for face recognition to a certain extent, and obtain more effective adversarial samples.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition adversarial sample generation, and in particular to a face recognition adversarial sample generation method based on restricted search space. Background Art

[0002] According to surveys, faces convey the vast majority of information for human identity judgment. Among several identity judgment methods, faces are the most important basis for judgment. Therefore, in the past few decades, research on face recognition has received widespread attention not only in perceptual science and cognitive science, but also in social life and security. Therefore, interpreting face recognition and studying the sub-field of face recognition, namely the generation method of adversarial samples, has become a research topic. The generation method of face recognition adversarial samples is a method of generating patterns that can interfere with face recognition. Let the features generate migration angles on the hypersphere output by the model.

[0003] In the 1960s, face recognition technology began to emerge. With the development of deep learning technology, as an important part of face recognition research, face recognition adversarial sample generation technology has received widespread attention. The generation of face recognition adversarial samples is a very challenging and innovative research topic. It is formed by combining relevant knowledge in fields such as psychology, computer vision, and deep learning. According to the nature of the generation technology, there are currently several important application scenarios: (1) Explain face recognition: give the black box face recognition model better interpretability to help people understand its working principle and further promote the research of face recognition; (2) Test means: test the robustness of face recognition under different conditions through generated adversarial samples, so as to detect the working performance of face recognition. (3) Expand data: supplement limited data through generated adversarial samples to improve the quality of face recognition.

[0004] Considering the face recognition adversarial sample generation framework and various unpredictable factors in various application scenarios, the current generation technology still faces the following challenges: (1) Stability of the generation process: The generation technology is complex and the generation process is often interfered by various factors to varying degrees. However, the currently available generation technology is often not restricted by the search space. In the high-dimensional manifold space, the potential form of the sample is easily interfered by various factors, resulting in an unstable generation process. Therefore, the stability of the generation process is difficult; (2) The effectiveness of the generation process: From the perspective of adversarial samples, the quality of the generation results in the application scenario depends on the size of the feature migration angle introduced by the hypersphere of the face recognition output. This process is easily affected by facial posture and lighting. Improving the norm of the inter-class and intra-class migration distances that can be generated by the generation results is a task that needs to be explored and solved. In summary, the generation method of face recognition adversarial samples is still an open problem that requires continuous exploration.

[0005] Regarding the stability of generation, the focus of the research is on how to improve the invariance of the output to the sensitive factors by compressing the sensitive factors, and how to compress the sensitive factors is often complex and multi-stage. Yin (Yin, Bangjie et al. "Adv-Makeup: A New Imperceptible and Transferable Attack on Face Recognition." IJCAI (2021).) reduces the proportion of uncertain factors by compressing the size and color space, and limits the color range of the generated samples. Tong (Tong, L., Chen, Z., Ni, J., Cheng, W., Song, D., Chen, H., & Vorobeychik, Y. (2021). FACESEC: A Fine-grained Robustness Evaluation Framework for Face Recognition Systems. 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 13249-13258.) designed a generative framework in which adversarial samples are composed of prescribed pixel pattern shapes spliced ​​by affine transformation, so that strict restrictions are imposed on the sample shape and pixel selection space composition. Li (Li, Lingzhi et al. "Advancing High Fidelity Identity Swapping for Forgery Detection." 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 5073-5082.) restricts the generation domain of adversarial samples to a specified space, thereby limiting the probability space of samples in the high-dimensional manifold and obtaining a more stable sample solution.On this problem, Xiao (Xiao, Z., Gao, X., Fu, C., Dong, Y., Gao, W., Zhang, X., Zhou, J., & Zhu, J. (2021). Improving Transferability of Adversarial Patches on Face Recognition with Generative Models. 2021 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), 11840-11849.) has achieved good performance. The proposed method fully utilizes distribution constraints in the manifold space dimension to improve stability under norm and probability constraints.

[0006] To solve the problem of effectiveness of the generation process, multiple fusion mechanisms are often introduced to fuse the pre-trained face recognition models in multiple directions during the generation process through averaging or other pooling methods, thereby reducing the deviation in a single direction. The above-mentioned researcher Xiao considered that different face recognition models have data set deviations in addition to extracting identity features, and proposed a method based on average pooling to average the output results of multiple face recognition models and fuse the results to improve performance. Summary of the invention

[0007] Regarding the stability and effectiveness issues in the field of face recognition adversarial sample generation, current research has complex and multi-stage defects. The purpose of this invention is to propose a face recognition adversarial sample generation method based on restricted search space, using a simpler and more efficient framework to improve the stability of the generation process, thereby increasing the migration angle brought by the adversarial sample in the hypersphere.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A method for generating adversarial samples for face recognition based on a restricted search space, the method comprising the following steps:

[0010] Step 1, preprocessing a face image dataset, where the image dataset includes real identity label data corresponding to the face;

[0011] Step 2, using the preprocessed face image as data to pre-train the generator model network: first input the preprocessed face image into the generator model, then design the first objective function of the generator model, and limit the search space center of the generator model to the input preprocessed image;

[0012] Step 3, input the face image preprocessed in step 1 into the generator model network pretrained in step 2, the generator model outputs an adversarial sample, and then the adversarial sample and the face image preprocessed in step 1 are sent to the discriminator model to extract features, and then the extracted features are output with a prediction score through a fully connected layer, and the second objective function of the generator model and the third objective function of the discriminator model are designed according to the prediction score; then the second objective function and the third objective function are optimized by the back propagation method, and the parameters of the generator model and the discriminator model are optimized;

[0013] Step 4, the adversarial sample generated in step 3 and the preprocessed face image in step 1 are sent to the pre-trained face recognition model for feature extraction, the adversarial sample is first covered on the preprocessed face image, and then the face image covered with the adversarial sample is sent to the face recognition model for feature extraction; by comparing the extracted feature cosine similarity as the fourth objective function of the generator model, and then optimizing the fourth objective function by the back propagation method, so that the generator model generates an optimized adversarial sample.

[0014] The present invention proposes a method for generating face recognition adversarial samples based on restricted search space. It uses the idea of ​​restricted search space to improve the stability and effectiveness of the generation process in a simpler and more efficient way. At the same time, it solves the problem of insufficient migration angle introduced by common samples to the model in face recognition adversarial sample generation to a certain extent, and obtains adversarial samples with stronger generalization. Compared with the prior art, the method of the present invention can further improve the performance of the generated adversarial samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the method of the present invention;

[0016] Figure 2 Performance comparison between the present invention and existing methods, (a) MS-celeb-1M dataset, (b) CelebA-HQ dataset, (c) LFW dataset. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0018] The lack of generalization in face recognition adversarial sample generation can be attributed to two aspects: the instability of the generation process and the lack of effectiveness of the generation process. For the former, the possibility of multiple composition of samples in the high-dimensional manifold space can be reduced by limiting the search space, thereby reducing the probability of pathological solutions due to excessive solution space; for the latter, the migration angle of adversarial samples to the output hypersphere of the face recognition model can be increased by generalization expansion.

[0019] This embodiment provides a method for generating adversarial samples for face recognition based on a restricted search space, and the specific steps are as follows:

[0020] The original facial expression dataset is preprocessed by face alignment, resizing, and numerical normalization, and the input image size is unified into a 600×600×3 face dataset and a 480×720×3 rectangular image block dataset X containing the eye and nose areas of the face.

[0021] Each image x in the preprocessed face image partial area dataset X is fed into the generator model to obtain the output image x′; x′ represents the output of the generator model for the input image x; the objective function is set to the average of the first and second norms of the difference between x′ and x. Back-propagation is used to optimize the objective function, and the center of the generator's search space is limited to the input preprocessed image to locate the center of the generator's search space. The subsequent steps guide the generator model to search for adversarial samples at a more accurate starting position.

[0022] The objective function of the generator model designed based on the prediction score:

[0023]

[0024] Among them, x i is the preprocessed face image block, K refers to the face image block x i The number of samples, G(x i ) is the generator model for the input x i The generated adversarial example, D(G(x i )) is the discriminator extracting features from the generated adversarial samples and then outputting the features through the fully connected layer to obtain the prediction score.

[0025] The objective function of the discriminator model designed based on the prediction score:

[0026]

[0027] Among them, RealData i is the preprocessed face image block, M refers to the face image block RealData i The number of samples D(RealData i ) is the prediction score that the discriminator extracts features from the preprocessed face image blocks and then outputs the features through the fully connected layer.

[0028] First calculate GAN by back propagation method loss The gradient of the generator model parameters is calculated, and then the Adam update method is used to update the model parameters according to the gradient and the current model parameters. Then D is calculated by the back propagation method. lossThe Adam update method is then used to update the model parameters based on the gradient and the current model parameters.

[0029] The adversarial sample is overlaid onto the preprocessed face image of experimenter 1. The adversarial sample size 160×240×3 is smaller than the face image size 600×600×3 and can only cover part of the face area.

[0030] The face image of experimenter 1 covered with the adversarial sample and the face image of experimenter 2 not covered with the adversarial sample are sent to the pre-trained face recognition model, and the face recognition model extracts features from the input images;

[0031] The cosine similarity is calculated for the extracted features. The feature cosine similarity is:

[0032]

[0033] f1 is the feature extracted by the pre-trained face recognition model from the face image of experimenter 1 that covers the adversarial sample, and f2 is the feature extracted by the pre-trained face recognition model from the face image of experimenter 2 that does not cover the adversarial sample.<f1,f2> is the vector dot product, and ||f1||f2|| is the vector's second norm.

[0034] First calculate Cos by back propagation method loss The gradient of the generator model parameters is then updated using the Adam update method based on the gradient and the current model parameters.

[0035] The beneficial effects of this embodiment can be verified by the following experiments:

[0036] 1. Model Accuracy

[0037] 1. Comparison of different backbone networks

[0038] According to the generation framework proposed in this embodiment, the models trained using four backbone networks, StyleGAN, U-net, ProGAN and BigGAN, as generator models are compared with the benchmark models. As shown in Table 1, on the three face image datasets MS-celeb-1M, CelebA-HQ and LFW, the maximum cosine distance of the migration angle that can be introduced by the adversarial sample in the hypersphere output by the benchmark face recognition model is significantly improved, with an average increase of 4.5°, 10.2° and 9.2° in the migration angle, respectively.

[0039] Table 1 Performance comparison of the method of the present invention under different backbone networks

[0040]

[0041] 2. Comparison with existing methods

[0042] Figure 2 The performance comparison between the present invention and the existing methods mentioned in the background technology is given. For the fairness of the comparison, except for the method proposed by Xiao, which uses StyleGAN as the backbone network, the backbone networks of the other methods are all U-net. The present invention is pre-trained on the MS-Celeb-1M face recognition dataset. Figure 2 As shown in Figure (a), the present invention achieves a migration angle of 58.9° on the MS-celeb-1M dataset, which is an improvement of more than 1°; Figure (b) shows that the present invention achieves the highest accuracy on the LFW dataset; and Figure (c) shows that the present invention achieves an improvement of nearly 2° in the migration angle on the CelebA-HQ dataset. This shows the superiority of the present invention.

[0043] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating adversarial samples for face recognition based on restricted search space, characterized in that: The steps of the method include: Step 1, preprocessing the face image dataset, the image dataset includes the real identity label data corresponding to the face; Step 2, using the preprocessed face image as data to pre-train the generator model network: first input the preprocessed face image into the generator model, then design the first objective function of the generator model, and limit the search space center of the generator model to the input preprocessed image; Step 3, input the face image preprocessed in step 1 into the generator model network pretrained in step 2, the generator model outputs an adversarial sample, and then the adversarial sample and the face image preprocessed in step 1 are sent to the discriminator model to extract features, and then the extracted features are output with a prediction score through a fully connected layer, and the second objective function of the generator model and the third objective function of the discriminator model are designed according to the prediction score; then the second objective function and the third objective function are optimized by the back propagation method, and the parameters of the generator model and the discriminator model are optimized; The second objective function is: Among them, x i is the preprocessed face image block, K refers to the face image block x i The number of samples, G(x i ) is the generator model for the face image block x i The generated adversarial example, D(G(x i )) is the discriminator extracting features from the generated adversarial sample and then outputting the features through the fully connected layer to obtain the prediction score; Step 4, the adversarial sample generated in step 3 and the preprocessed face image in step 1 are sent to the pre-trained face recognition model for feature extraction, the adversarial sample is first covered on the preprocessed face image, and then the face image covered with the adversarial sample is sent to the face recognition model for feature extraction; by comparing the extracted feature cosine similarity as the fourth objective function of the generator model, and then optimizing the fourth objective function by the back propagation method, so that the generator model generates an optimized adversarial sample.

2. According to claim 1, a method for generating adversarial samples for face recognition based on restricted search space is characterized in that: In the step 1, the preprocessing is specifically: aligning the face images, then cropping a part of the face images as an image data set, and normalizing the values.

3. According to claim 1, a method for generating adversarial samples for face recognition based on restricted search space is characterized in that: In step 2, the method for pre-training the generator model is: Step 21: Send each image x in the face image dataset X preprocessed in step 1 to the generator model to obtain the output image x ′ ;x ′ Represents the output generated by the generator model for the input image x; Step 22, set the first objective function of the generator model to be image x ′ The average of the first and second norms of the difference between and x; the objective function is optimized using backpropagation, and the center of the generator's search space is limited to the input preprocessed image.

4. The method for generating adversarial samples for face recognition based on restricted search space according to claim 1, characterized in that: In step 3, the third objective function of the discriminator model is designed according to the prediction score: Among them, RealData i is the preprocessed face image block, M is the face image block RealData i The number of samples, D(RealData i ) is the discriminator for the face image block RealData i Extract features and then pass the features through the fully connected layer to output the prediction scores.

5. The method for generating adversarial samples for face recognition based on restricted search space according to claim 4, characterized in that: In step 3, two objective functions are optimized by back propagation method, and the specific method for optimizing the parameters of the generator model and the discriminator model is: First calculate the second objective function GAN by back propagation method loss The gradient of the generator model parameters is calculated, and then the Adam update method is used to update the model parameters according to the gradient and the current model parameters; then the third objective function D is calculated by the back propagation method loss The Adam update method is then used to update the model parameters based on the gradient and the current model parameters.

6. The method for generating adversarial samples for face recognition based on restricted search space according to claim 1, characterized in that: In step 4, when the adversarial sample is overlaid on the preprocessed face image, the size of the adversarial sample is much smaller than the face image and can only cover part of the face area.

7. The method for generating adversarial samples for face recognition based on restricted search space according to claim 1, characterized in that: In step 4, the steps of feature extraction are: Step 41, overlaying the adversarial sample onto the preprocessed first face image; Step 42: Send the first face image covering the adversarial sample and the second face image not covering the adversarial sample to the face recognition model for feature extraction.

8. The method for generating adversarial samples for face recognition based on restricted search space according to claim 7, characterized in that: In step 4, the feature cosine similarity is: f1 is the feature extracted by the pre-trained face recognition model from the first face image that covers the adversarial sample, and f2 is the feature extracted by the face recognition model from the second face image that does not cover the adversarial sample;<f1,f2> is the vector dot product, ∥f1||f2∥ is the vector's second norm, Cos loss It is the fourth objective function of the generator model.

9. The method for generating adversarial samples for face recognition based on restricted search space according to claim 8, characterized in that: In step 4, the fourth objective function of the generator model is optimized by the back-propagation method, and the method for making the generator model generate optimized adversarial samples is: First, calculate the feature cosine similarity Cos by back propagation method loss The gradient of the generator model parameters is then updated using the Adam update method based on the gradient and the current model parameters.

Images