Face recognition system based on face depth information and countermeasure method

Abstract

A face recognition system based on face depth information countermeasure method, through constructing and training the UNet-based encoder-decoder structure depth estimation neural network in the offline stage, the depth map of the image to be tested is generated; then by constructing and training the four-channel face recognition network, after generating the depth map of the input color image in the online stage, the recognition authentication result is obtained through the four-channel face recognition network. The method of introducing face depth information in the face recognition system greatly improves the robustness of the face recognition system to attack samples.

Description

Technical Field

[0001] This invention relates to a technology in the field of neural network applications, specifically a method for adversarial defense of a face recognition system based on facial depth information. Background Technology

[0002] Adversarial attacks against faces refer to pixel-level alterations to facial images, causing recognition networks to misidentify the face as someone else. These alterations can be physically achieved through makeup, wearing special accessories, etc., creating significant security vulnerabilities. Currently, there are few adversarial defense methods specifically for faces. Most studies testing defense effectiveness on faces focus on general-purpose defenses, with adversarial training being the most common approach. Adversarial training uses the current model to generate adversarial attack samples, then incorporates these samples into the training process, significantly improving the model's robustness against adversarial attacks. Essentially, adversarial training exposes the model to attack samples, thereby improving its recognition accuracy on these samples. However, adversarial training introduces substantial training overhead, making it unsuitable for large-scale model iterations in face recognition systems.

[0003] Another approach to combat adversarial attacks is to detect and filter out adversarial samples before they are input into the network, or to transform adversarial attack images into normal images before inputting them into the network. However, these techniques are clearly unsuitable for facial recognition systems. Regardless of whether the system determines that the input image is an adversarial attack sample, the facial recognition system should output a final comparison result to suit the application scenario. Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this invention proposes a face recognition system adversarial defense method based on facial depth information to solve the problem of potential adversarial attacks on networks. By introducing facial depth information into the face recognition system, the robustness of the face recognition system to attack samples is greatly improved.

[0005] This invention is achieved through the following technical solution:

[0006] This invention relates to an adversarial defense method for a face recognition system based on facial depth information. It involves constructing and training a depth estimation neural network with a UNet-based encoder-decoder structure offline to generate a depth map of the image to be tested. Then, by constructing and training a four-channel face recognition network, a depth map is generated from the input color image online, and the recognition and authentication result is obtained through the four-channel face recognition network.

[0007] The depth estimation neural network described is based on the UNet architecture. Specifically, the network structure is a classic fully convolutional network based on the traditional UNet architecture, consisting of an encoder and a decoder. The encoder part is the same as the classic structure: after receiving the input, it transforms the input into a 32-dimensional feature map, and then passes through four downsampling modules with a downsampling coefficient of 2 to upsample the channel, resulting in a 512-dimensional feature map. The decoder part typically consists of upsampling modules with an upsampling coefficient of 2. The difference from traditional upsampling modules is that, to address the checkerboard effect, this scheme does not use the commonly used deconvolution structure. Instead, it employs bilinear interpolation followed by convolution, doubling the size of the feature map through bilinear interpolation, and then using a 3x3 convolution to transform the feature map. After the encoder-decoder structure, a single-channel depth map with the same resolution as the input is obtained. The depth estimation neural network is trained using the real depth map as a label.

[0008] The four-channel face recognition network comprises a main structure and a classification head structure. The main structure can employ any commonly used face recognition network; in this scheme, the IR50 network proposed in the Arcface paper is used to extract a 512-dimensional feature vector from the input face image. The modification to the network is to adjust the number of input channels in the convolutional layers to four. The classification head structure also uses the Arcface classification head, commonly used in face recognition tasks, instead of softmax. This allows for better differentiation of features between face classes. The classification head receives the 512-dimensional vector and outputs a result vector with the same dimension as the class.

[0009] This invention relates to a system for implementing the above method, comprising: a 3DMM virtual depth generation unit, a monocular face depth estimation unit, and a subject face recognition unit, wherein: the 3DMM unit performs unsupervised fitting based on face color data to obtain fitted face depth data for pre-training of the depth estimation module; the face depth estimation unit generates a face depth map based on face color image information through network prediction for subsequent recognition; and the face recognition unit performs feature extraction using a four-channel face image obtained by stitching the color image and the depth map as input to obtain the final face features.

[0010] Technical effect

[0011] This invention introduces depth information into a face recognition system that uses a color face image as input. After generating a face depth map corresponding to the color face image, the color image and depth map are concatenated to form a four-channel input for training the recognition model. This significantly improves the robustness of the model against adversarial attacks. The accuracy of recognizing adversarial examples under adversarial attacks is significantly improved compared to not using this method. It achieves a substantial improvement in the robustness of the face recognition model without adversarial training. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the invention;

[0013] Figure 2 This is a schematic diagram of the offline stage of an embodiment. Detailed Implementation

[0014] like Figure 1 As shown, this embodiment relates to an adversarial defense method for a face recognition system based on facial depth information, including... Figure 2 The offline and online stages are shown, where the feature comparison part only exists in the offline recognition stage.

[0015] The offline phase includes:

[0016] 1) Constructing a dataset: Prepare real face point cloud data P, project P onto the XY plane to obtain color image I and depth image D; prepare an RGB face dataset, uniformly select face color images I′ according to face identity information, and use the face 3D deformation statistical model (3DMM) to fit a virtual face depth image D′, which serves as the real label for this part of the data in the depth estimation module.

[0017] 2) Construct and pre-train a depth estimation module based on the UNet encoder-decoder structure: The upsampling layer of the decoder in this depth estimation module adopts a combination of nearest neighbor upsampling and convolutional layers to replace the deconvolutional layer, effectively avoiding the checkerboard phenomenon that is very easy to occur in face generation tasks.

[0018] The aforementioned pre-training refers to: putting I gt Input a depth estimation network and output a predicted depth map D at the same resolution. pred The pre-training loss function is L = αL1 + βL2, where the first loss term is D. gt With D pred MSE loss L1 = (D pred -D gt ) 2 The second loss term is the normal graph N. gt With N pred The MSE loss between them is L2 = (N pred -N gt ) 2 α and β are the weights of the two losses. In this scheme, the optimal parameters are α = 1 and β = 10.

[0019] 3) Construct a four-channel face dataset for training a four-channel face recognition network: Using the pre-trained depth estimation model from step 2), generate corresponding face depth data on a large-scale face RGB dataset. Then, concatenate the RGB modality and depth modality in each channel to obtain a four-channel face dataset. Train the face recognition network consisting of the IR50 network (as the main body) and the arcfacehead network (as the head) using an end-to-end training method. The cross-entropy loss function is used for training.

[0020] The online phase includes: extracting the features of the person to be identified from the main body of the trained face recognition network and storing them in a feature library; comparing the similarity with features in the feature library, and considering the face matching successful if the similarity is greater than a threshold θ. In the experiment, this threshold was set as the feature similarity θ when the model was tested on the LFW dataset with an FAR of 0.1%.

[0021] The extraction refers to: directly capturing a face color image I and a depth image D using a system with a depth camera, or inputting the captured face color image I into a trained depth estimation module to obtain a predicted depth image D′, and then stitching the color image and the depth image together to obtain a four-channel input.

[0022] Through specific practical experiments, this embodiment uses adversarial attack data to test the robustness of the model. The data tested includes existing adversarial attack datasets and adversarial attack samples generated by an adversarial attack model trained using FGSM and PGD. The adversarial model used for the attack is a pre-trained IR50 model. FGSM uses a maximum perturbation of 8 / 255, and PGD uses a 40-step iterative attack with a maximum perturbation of 16 / 255.

[0023] Existing adversarial attack datasets include: TALFW, which uses a highly transferable model to generate adversarial examples based on the LFW dataset, and Adv-makeup, which generates adversarial attack datasets by adding makeup to different regions of the face to simulate attacks in real-world scenarios.

[0024] For adversarial defense tasks, this method typically employs common testing methods used in face comparison tasks to test the model, with two metrics used to measure the model's robustness:

[0025] For the generated adversarial examples, test their recognition accuracy.

[0026] For a certain attack method, test the success rate of that attack method on the system.

[0027] The first metric is used to measure the effectiveness of this method compared to previous methods on adversarial defense tasks, comparing its performance with baseline models, JPEG encoding, adversarial training, and MTER (an adversarial training method with improved regularization). Table 1 presents the metrics of different methods on self-generated FGSM and PGD attack data, while Table 2 presents the metrics of different methods on the existing human face adversarial attack dataset TALFW. The experimental data in the tables show that, while the accuracy of human recognition (in the "natural" column) drops similarly to other schemes, this method demonstrates superior performance on adversarial data. Furthermore, considering the computational resource overhead of adversarial defense methods, this method significantly outperforms the adversarial defense series of schemes.

[0028] Table 1

[0029]

[0030] Table 2

[0031] method LFW TALFW baseline 99.73 53.95 jpegencoding 99.37 69.67 adversarialtraining 99.42 82.93 MTER 99.45 83.98 This invention 99.42 84.35

[0032] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for adversarial defense of a face recognition system based on facial depth information, characterized in that, A depth estimation neural network based on the UNet encoder-decoder structure is constructed and trained offline to generate a depth map of the image to be tested. Then, a four-channel face recognition network is constructed and trained to generate a depth map of the input color image in the online stage, and the recognition and authentication result is obtained through the four-channel face recognition network. The offline phase includes: 1) Building the dataset: Preparing real face point cloud data ,Will Projecting onto the XY plane yields a color image. With depth map Prepare an RGB face dataset, and uniformly select color images of faces according to facial identity information. Virtual face depth maps were fitted using a 3D face deformation statistical model (3DMM). As a color image of a face The true label in the depth estimation module; 2) Construct and pre-train a depth estimation module based on the UNet encoder-decoder structure: The upsampling layer of the decoder in this depth estimation module adopts a combination of nearest neighbor upsampling and convolutional layers to replace the deconvolutional layer, effectively avoiding the checkerboard phenomenon that is very easy to occur in face generation tasks. 3) Construct a four-channel face dataset for training a four-channel face recognition network: Using the pre-trained depth estimation model from step 2), generate corresponding face depth data on a large-scale face RGB dataset. Then, concatenate the RGB modality and depth modality in each channel to obtain a four-channel face dataset. Train the face recognition network consisting of the IR50 network (as the main body) and the ArcFace head network (as the head) using an end-to-end training method. The cross-entropy loss function is used for training.

2. The adversarial defense method for face recognition systems based on facial depth information according to claim 1, characterized in that, The aforementioned deep estimation neural network, based on the UNet structure, includes: a fully convolutional network, divided into an encoder and a decoder. The encoder part transforms the input into a 32-dimensional feature map through an input receiving module, and then performs upsampling on the channel through four downsampling modules with a downsampling coefficient of 2 to obtain a 512-dimensional feature map. The decoder part consists of upsampling modules with an upsampling coefficient of 2. The upsampling part is designed to solve the checkerboard effect. It uses bilinear interpolation followed by convolution. The bilinear interpolation doubles the size of the feature map, and then a 3x3 convolution is used to transform the feature map. After the encoding-decoding structure, a single-channel depth map with the same resolution as the input is obtained. The depth estimation neural network is trained by using the real depth map as a label for constraint.

3. The adversarial defense method for a face recognition system based on facial depth information according to claim 1, characterized in that, The four-channel face recognition network includes a main structure and a classification head structure. In the main structure, the number of input channels of the convolutional layer that receives the input is adjusted to four channels. The classification head structure adopts the Arcface classification head commonly used in face tasks to receive a 512-dimensional vector and output a result vector with the same dimension as the category.

4. The adversarial defense method for a face recognition system based on facial depth information according to claim 1, characterized in that, The aforementioned pre-training refers to: Input a depth estimation network and output a predicted depth map at the same resolution. ; The pre-trained loss function The first loss item is... and MSE loss The second loss term is the normal graph. and MSE loss between , and These are the weights for the two types of losses.

5. The adversarial defense method for face recognition systems based on facial depth information according to any one of claims 1-3, characterized in that, The online phase includes: extracting the features of the person to be identified from the main body of the trained face recognition network and storing them in a feature database; comparing the similarity with features in the feature base database, and ensuring the similarity is greater than a threshold. At that time, it was considered that the face comparison was successful.

6. The adversarial defense method for a face recognition system based on facial depth information according to claim 5, characterized in that, The extraction mentioned above refers to: directly capturing color images of faces using a system equipped with a depth camera. With depth map Or capture a color image of a face. The predicted depth map is obtained by inputting the trained depth estimation module. The color image and depth image are then stitched together to obtain a four-channel input.

7. A system for implementing the adversarial defense method for a face recognition system based on facial depth information as described in any one of claims 1-6, characterized in that, include: The system comprises a 3DMM virtual depth generation unit, a monocular face depth estimation unit, and a subject face recognition unit. Specifically, the 3DMM unit performs unsupervised fitting based on the face color data to obtain the fitted face depth data, which is used for pre-training of the depth estimation module. The face depth estimation unit generates a face depth map based on the face color image information through network prediction, which is used for subsequent recognition. The face recognition unit takes a four-channel face image obtained by stitching the color image and the depth map as input, performs feature extraction, and obtains the final face features.

Images