Security detection method of face recognition system based on generation of highly robust adversarial samples

Abstract

The present invention relates to a facial recognition system security detection method based on highly robust adversarial sample generation, and belongs to the field of facial recognition security protection technology. This method designs an adversarial sample physical attack method, which attacks the target deep facial recognition system by generating multiple stickers of different shapes at one time, and designs a converter to simulate the appearance of a face with stickers attached in the real world. These stickers of different shapes can be flexibly attached to any part of the face. At the same time, a physical attack is carried out on the deep facial recognition system by using stickers, and adversarial samples can be reconstructed in the physical world without designing a special device, thereby detecting the security of the system. The method of the present invention is easy to use, and the generated stickers are more flexible and robust. The security of the system is judged by testing the effectiveness of the generator stickers through the target deep facial recognition system, which provides a direction for further adjusting and optimizing the facial recognition system.

Description

Technical Field

[0001] The present invention relates to a face recognition system security detection method based on highly robust adversarial sample generation, belonging to the technical field of face recognition security protection. Background Art

[0002] As the most mature application field of artificial intelligence, face recognition systems have been implemented on a large scale and serve people's daily lives. For example, identity verification based on the second-generation ID card, face payment, face gate verification, etc. However, while being commercialized on a large scale, face recognition technology also faces more threats. With the development of deep neural networks (DNN), it has attracted more and more attention and has been applied to face recognition systems.

[0003] Adversarial samples are a type of samples that are maliciously designed to attack deep learning models. By adding deliberately constructed perturbations to the input, adversarial samples can cause specific deep learning models to produce incorrect classifications. Therefore, adversarial samples can be used to detect whether there are security issues in deep learning models. For face recognition systems, if adversarial samples can be generated, it means that the target face recognition system is no longer secure.

[0004] Adversarial samples are divided into those in the digital world and those in the physical world. The digital world’s adversarial samples use digital attack methods to directly modify the pixel size of a face image, while the physical world’s adversarial samples need to be reconstructed in the physical world.

[0005] Compared with adversarial samples produced by physical attack methods, adversarial samples generated by digital attack methods need to be directly input into the recognition model of the face recognition system, so their practicality is poor. Face adversarial samples in the physical world can deceive the deep recognition system into making wrong judgments by making slight changes to the face (attaching stickers or wearing glasses), thus causing significant losses.

[0006] Physical attack methods can be further divided into spoofing attacks and evasion attacks. Among them, spoofing attacks refer to perturbing the attacker's face so that the face recognition system recognizes the victim; evasion attacks refer to perturbing the attacker's face so that it cannot be successfully recognized by the face recognition system.

[0007] The criteria for measuring the quality of physical attack methods are: effectiveness, ease of use and robustness. The goal of physical attack is to be able to successfully deceive the target face recognition system and to resist changes in environmental factors, including the distance from the attacker to the camera, ambient brightness, and the attacker's head posture. Changes in environmental factors will reduce the effectiveness of physical attack methods; ease of use means that the attack can generate disturbances in the physical world without the help of special devices; robustness means that adversarial samples can resist changes in environmental factors, such as changes in the attacker's posture.

[0008] Therefore, it is of great significance if a highly robust physical adversarial sample generation method can be designed to test the security of deep face recognition systems in physical scenarios, thereby verifying the credibility of the recognition results of the face recognition system. Summary of the invention

[0009] The purpose of the present invention is to solve the technical problem of effectively verifying the security of deep face recognition systems in physical scenarios and verifying the credibility of the recognition results of face recognition systems, and to propose a face recognition system security detection method based on the generation of highly robust adversarial samples.

[0010] Since the deep face recognition system always extracts information in specific facial areas (such as near the facial features), attaching stickers to these areas can change the facial features, thereby reducing the recognition accuracy of the face recognition system. For example, when a square sticker is attached to a specific position (such as the eyebrows), the four corners of the sticker may block the eyes, which may cause the attacker to fail to pass the liveness detection system. The liveness detection system detects the movement of the eyeballs to determine whether the image in front of the camera is a real living person or a video being played. At the same time, in order to ensure that the sticker remains effective in the physical world, the attack method needs to simulate the appearance of the sticker when it is attached to the face picture. This is because the face is a 3D shape rather than a 2D plane, and the sticker will deform when attached to the face. On the other hand, the physical environment in the real world is constantly changing, requiring the generated stickers to have very strong robustness.

[0011] In response to the above problems, this method designs a physical attack method for adversarial samples, which attacks the target deep face recognition system by generating multiple stickers of different shapes at a time, and designs a converter to simulate the appearance of a face with stickers attached in the real world. These stickers of different shapes can be flexibly attached to any part of the face (nose bridge, eyebrows, lips, etc.). At the same time, considering the ease of use, the deep face recognition system is physically attacked by using stickers, and adversarial samples can be reconstructed in the physical world without designing a special device, thereby testing the security of the system.

[0012] The shape of stickers is generated by adversarial generative network mechanism, which consists of four different parts: sticker generator Shape Discriminator Converter and target deep face recognition system And realize counterfeit attack and avoid attack.

[0013] Given a face image x and label y, assuming the face recognition system recognizes it correctly, then

[0014] Phishing attack means that the attacker first specifies a victim y * The recognition result of the adversarial sample is Avoiding an attack means that the attacker creates an adversarial sample x * =x+Δ x , so that the recognition result is

[0015] Sticker Generator It includes shape components and sticker components, which are responsible for generating the shape and content of stickers respectively. It contains different branches to generate multiple stickers at the same time; the shape component is used to generate a shape mask, and the sticker component is used to generate an adversarial sticker. The shape mask is a binary image, that is, it only contains white (pixel is 255) and black (pixel is 0), which is used to crop the originally generated square sticker so that the shape of the cropped sticker is consistent with the shape template.

[0016] Shape Discriminator Used to identify the input shape image as a sticker generator The generated shape still comes from the template shape dataset.

[0017] Converter Sticker Generator The generated stickers are attached to the input face image x. First, a 3D face reconstruction method based on deep learning (such as R-Net) is used to obtain auxiliary information including 3D face shape and lighting model parameters. Then, the stickers generated by the sticker generator are placed at a pre-selected position to form a sticker template. After that, the sticker template and auxiliary information are input into the renderer to obtain the rendered stickers. The rendered stickers are placed on the original face image to obtain a face image with the stickers attached. Finally, the target deep face recognition system Take the face image with stickers attached as input and give the final recognition result. and shape discriminator During the process, the parameters of the shape mask remain unchanged, thus ensuring the effectiveness of the recognition results.

[0018] Through the target deep face recognition system Verify the effectiveness of the generator stickers, determine the security of the system, and provide direction for further adjustment and optimization of the face recognition system.

[0019] All actions of obtaining signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located and with the prior authorization of the owner.

[0020] Beneficial Effects

[0021] The method of the present invention has the following beneficial effects:

[0022] (1) High ease of use, no need to customize special equipment;

[0023] (2) The position of the stickers is not fixed, making the attack more flexible;

[0024] (3) stickers of different shapes can be generated. The shapes of stickers are the shapes in the template shape dataset prepared in advance, so that the generated stickers can be attached to the face more flexibly;

[0025] (4) The generated stickers are highly robust, and the converter can attach stickers to faces in different poses. The converter can also use the estimated lighting model parameters for rendering, which increases the robustness of the stickers to changes in ambient brightness. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the physical attack process of the method of the present invention;

[0027] Figure 2 It is an architecture diagram of the sticker generator, shape discriminator, converter and target deep face recognition system in the method of the present invention;

[0028] Figure 3 This is the specific process in which the converter in the method of the present invention attaches the sticker to the face picture. DETAILED DESCRIPTION

[0029] The method of the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0030] A face recognition system security detection method based on highly robust adversarial sample generation includes the following steps:

[0031] Step 1: Prepare the attacker P A Face image dataset D A and template shape dataset D s , and select a 3D face reconstruction algorithm

[0032] Specifically, the steps include:

[0033] Step 1.1: Prepare the attacker P A Face image dataset D A .

[0034] Specifically, facial images of the attacker taken under different environmental conditions are collected, and the environmental conditions include: the distance between the attacker and the camera, the ambient brightness, and the attacker's head posture.

[0035] Step 1.2: Prepare the attacker P A The template shape dataset D s .

[0036] Among them, D S is the sticker shape you want to generate, and there is no specific restriction on the sticker shape. S The distribution of

[0037] Step 1.3: Using 3D face reconstruction algorithm (such as R-Net algorithm) Process each image of the attacker and obtain the attacker's auxiliary information Among them, auxiliary information Including 3D face shape and lighting model parameters.

[0038] Step 2: Set the number of stickers that the sticker generator can generate at one time to m, and the position of each sticker to l i ,i=1,2,…,m. Set the attacker’s attack mode, including spoofing attack and evasion attack. If it is a spoofing attack, you also need to set the victim P B .

[0039] The sticker generator includes a shape component and a sticker component, which are responsible for generating the shape and content of stickers respectively. The sticker generator contains different branches to generate multiple stickers at the same time. The shape component is used to generate a shape mask, and the sticker component is used to generate adversarial stickers. The shape mask is a binary image, that is, it only contains white (pixels are 255) and black (pixels are 0), which is used to crop the originally generated square stickers so that the cropped sticker shape is consistent with the shape template.

[0040] Phishing attack means that the attacker first specifies a victim y * The recognition result of the adversarial sample is

[0041] Avoiding an attack means that the attacker creates an adversarial sample x * =x+Δ x , so that the recognition result is

[0042] Where x and y mean: given a face image x and label y, assuming the face recognition system recognizes it correctly, then

[0043] Step 3: From a Gaussian distribution with mean 0 and variance 1 Then, the random noise n is input into the sticker generator to generate n square stickers SC i , i = 1, 2, ..., m, and m shape masks SH i , i = 1, 2, ..., m. Then, the i-th shape mask SH is used in turn. i Cut the corresponding square sticker SC i , get the cut sticker

[0044] Step 4: From the template shape dataset D S Randomly sample m shape templates S i , and input it and the shape mask SH generated by the sticker generator into the shape discriminator Calculate the loss function

[0045]

[0046] in, is the sticker generator, SH is the shape mask, S is the shape template, Represents a Gaussian distribution The random noise n sampled from represents the shape template S sampled from the shape template dataset; λ represents the balance factor, which is used to calculate the third loss in formula (1) To scale; Represents a random distribution from a shape template The resulting shape template express The gradient of From the shape mask SH i and shape template S i Random sampling yields:

[0047]

[0048] Here, ε is sampled from the U[0,1] distribution, and U represents uniform distribution. The output includes sticker content SC and shape mask SH. When calculating, you only need to select the shape mask SH.

[0049] The shape discriminator is used to determine whether the input shape image is a shape generated by the sticker generator or comes from the template shape dataset.

[0050] Step 5: Calculate the loss function based on the adversarial generation network mechanism

[0051]

[0052] Step 6: x the attacker image A , cut sticker ST and auxiliary information Input to converter Get the attacker image with stickers attached

[0053] Step 7: Attach the sticker to the attacker's image Input to the target deep face recognition model In the above example, determine whether it is a counterfeit attack or an evasion attack, and calculate the loss function

[0054] If it is a phishing attack, then:

[0055]

[0056] Among them, P B Indicates the victim.

[0057] To avoid an attack:

[0058]

[0059] Among them, P A Indicates attacker.

[0060] Step 8: In the real world, if the pixel changes between adjacent stickers are particularly large, the camera cannot capture such drastic changes. Therefore, the total variance loss is introduced. To limit the pixel changes between adjacent stickers:

[0061]

[0062] Among them, SC i,j Represents the pixel value of the sticker SC at position (i, j). SC i,j+1 Represents the pixel value of the sticker SC at position (i, j+1). SC i,j-1 Represents the pixel value of the sticker SC at position (i, j-1).

[0063] Step 9: Some color values ​​in the real world cannot be printed by color printers. Therefore, non-printing fractions are introduced (Non-Printability Score) to limit the color of the sticker:

[0064]

[0065] in, is the pixel value in the sticker SC; P represents the set of pixel values ​​that the printer can print, and p is a pixel in the set.

[0066] Step 10: Combine formulas (3), (4), (5), (6) and (7) to get the loss of the sticker generator

[0067]

[0068] Among them, α, β and γ represent and The balance factor. Represents the loss function for the shape component of the sticker generator.

[0069] Step 11: Update the shape discriminator using formula (1) and formula (8) respectively. and sticker generator

[0070] After the training is completed, the sticker generator Ability to generate stickers of different shapes at once to attack the target face recognition system

[0071] Step 12: Print the generated sticker in color and attach it to the attacker’s face, thereby attacking the target facial recognition system in the physical world.

[0072] Step 13: Target Deep Face Recognition System Take the face image with attached stickers as input and give the final recognition result.

[0073] Through the target deep face recognition system Verify the effectiveness of the generator stickers, judge and analyze the security of the face recognition system, and provide direction for further adjustment and optimization of the system.

[0074] Example

[0075] A physical attack process of a highly robust physical adversarial sample generation method for deep face recognition system Figure 1 The architectural relationship between the sticker generator, shape discriminator, converter, and target deep face recognition system is shown in Figure 2 As shown in the figure, the specific process of the converter attaching stickers to face images is as follows Figure 3 shown.

[0076] In this embodiment, the network structures of the sticker generator and the shape discriminator are shown in Table 1 and Table 2 respectively.

[0077] Table 1 Network structure of sticker generator

[0078]

[0079]

[0080] According to the above Figure 1 In the attack process, when the method of the present invention is implemented, the following steps are taken:

[0081] Step 1: Prepare the camera, printer, and light source.

[0082] The camera chosen was Logitech C270, which is capable of capturing images in 960×1280 size.

[0083] The printer selected is HP DeskJet 2677. The RGB pixel values ​​that can be printed include (26,14,27), (122,24,43), (127,133,44), (21,81,46), (121,155,132), (19,78,116), (172,38,45), (22,36,67), (68,94,121), (193,138,139), (186,126,70), (65,89,45), (121,57,93), (69,22,30), (180,82,5 1), (70,34,75), (191,192,160), (72,140,104), (121,74,38), (191,178,72), (24,108,113), (128,165,167), (31,108,59), (125,148,85), (67,126,67), (19,38,31), (22,51,91), (174,39,76), (76,139,146), (183,83,103) and other 30 pixel values.

[0084] The light source is Philips 66135, and the ambient light intensity can be changed to 30lux, 130lux and 250lux.

[0085] Table 2 Network structure of shape discriminator

[0086]

[0087] Step 2: Select ArcFace (https: / / github.com / nizhib / pytorch-insightface), CosFace (https: / / github.com / MuggleWang / CosFace_pytorch), and FaceNet deep face recognition system (https: / / github.com / timesler / facenet-pytorch) as the attacked face recognition systems.

[0088] Step 3: According to step 1, create a face image dataset D A and template shape dataset D s , and select a 3D face reconstruction algorithm

[0089] Specifically:

[0090] Step 3.1: Recruit 20 people as attackers and take 45 pictures for each attacker. The ambient light intensity is 30 lux, 130 lux, and 250 lux. The distance from the attacker to the camera is 30 cm, 50 cm, and 70 cm. The attacker's head posture can be looking directly at the camera, tilting the head 20 degrees to the right, tilting the head 20 degrees to the left, tilting 20 degrees upward, and lowering the head 20 degrees. There are 45 environmental combinations in total, and one picture is taken under each combination of environmental factors.

[0091] Step 3.2: Select 15,000 binary images of five shapes: circle, square, pentagon, hexagon, and heptagon as the template shape dataset.

[0092] Step 3.3: Select R-Net (https: / / github.com / microsoft / Deep3DFaceReconstruction) as the 3D face reconstruction algorithm.

[0093] Step 4: Set the number of stickers that can be generated to 3, and set the positions of the stickers to (200,100), (400,100), and (300,200), which are the positions near the right eyebrow, left eyebrow, and nose bridge respectively. Set the attack mode to dodge attack.

[0094] Step 5: Set the parameters α=100, β=1 and γ=10 to calculate the loss function of the sticker generator.

[0095] Step 6: Generate stickers using the trained sticker generator and follow Figure 1 The attack process shown performs a physical attack.

[0096] During the physical attack, the distance between the attacker and the camera is 50cm, the ambient brightness is 130lux, and the attacker looks directly at the camera. The entire attack process will last for 25 seconds, and the proportion of frames in which the target face recognition system cannot detect the attacked person to the total number of frames is recorded as the attack success rate, and finally the average attack success rate of the 20 attackers is calculated.

[0097] The method of the present invention (denoted as FaceAdv) is compared with the use of adversarial generative networks to generate eyeglass frame stickers to attack deep face recognition systems in the physical world (denoted as AGNs) to verify the advantages and effectiveness of the present invention.

[0098] Table 3 Attack effects of FaceAdv and AGNs on three different face recognition systems

[0099]

[0100] Table 3 shows the results of FaceAdv and AGNs on three different face recognition systems. It can be seen that the attack success rate of FaceAdv is about 30% higher than that of AGNs, which shows that the converter successfully simulates the deformation caused by the sticker attached to the face surface.

Claims

1. A face recognition system security detection method based on highly robust adversarial sample generation, characterized in that: The steps include: Step 1: Prepare the attacker P A Face image dataset D A and template shape dataset D s , and select a 3D face reconstruction algorithm Step 2: Set the number of stickers that the sticker generator can generate at one time to m, and the position of each sticker to l i ,i=1,2,…,m; Set the attacker’s attack mode, including spoofing attack and evasion attack. If it is a spoofing attack, the victim P B ; The sticker generator includes a shape component and a sticker component, which are responsible for generating the shape and content of stickers respectively; the sticker generator contains different branches to generate multiple stickers at the same time; the shape component is used to generate shape masks, and the sticker component is used to generate adversarial stickers. The shape mask is a binary image containing only white and black, which is used to crop the originally generated square stickers so that the cropped sticker shape is consistent with the shape template; Phishing attack means: the attacker specifies a victim y * , so that the recognition result of the adversarial sample is Avoiding an attack means that the attacker creates an adversarial sample x * =x+Δ x , so that the recognition result is Where x and y mean: given a face image x and label y, assuming the face recognition system recognizes it correctly, then Step 3: From a Gaussian distribution with mean 0 and variance 1 A random noise n is sampled in the image; then, the random noise n is input into the sticker generator to generate m square stickers SC i , i = 1, 2, ..., m, and m shape masks SH i , i = 1, 2, ..., m; then, use the i-th shape mask SH in turn i Cut the corresponding square sticker SC i , get the cut sticker Step 4: From the template shape dataset D S Randomly sample m shape templates S i , and input it and the shape mask SH generated by the sticker generator into the shape discriminator Calculate the loss function in, is the sticker generator, SH is the shape mask, S is the shape template, Represents a Gaussian distribution The random noise n sampled from represents the shape template S sampled from the shape template dataset; λ represents the balance factor, which is used to calculate the third loss in formula (1) To scale; Represents a random distribution from a shape template The resulting shape template express The gradient of From the shape mask SH i and shape template S i Random sampling yields: Among them, ε is sampled from the U[0, 1] distribution, where U represents uniform distribution; sticker generator The output includes sticker content SC and shape mask SH. When calculating, you only need to select the shape mask SH. The shape discriminator is used to determine whether the input shape image is a shape generated by the sticker generator or comes from the template shape dataset; Step 5: Calculate the loss function based on the adversarial generation network mechanism Step 6: x the attacker image A , cut sticker ST and auxiliary information Input to converter Get the attacker image with stickers attached Step 7: Attach the sticker to the attacker's image Input to the target deep face recognition model In the above example, we determine whether it is a counterfeit attack or an evasion attack, and calculate the loss function If it is a phishing attack, then: Among them, P B Indicates the victim; To avoid an attack: Among them, P A Indicates attacker; Step 8: Introduce total variance loss To limit the pixel changes between adjacent stickers: Among them, SC i,j represents the pixel value of the sticker SC at position (i, j); SC i,j+1 represents the pixel value of the sticker SC at the position (i, j+1); SC i,j-1 represents the pixel value of the sticker SC at the position (i, j-1); Step 9: Introducing Non-Printing Fractions To limit the colors of stickers: in, is the pixel value in the sticker SC; P represents the set of pixel values ​​that the printer can print, and p is a pixel in the set; Step 10: Combine formulas (3), (4), (5), (6) and (7) to get the loss of the sticker generator Among them, α, β and γ represent and The balance factor The loss function representing the shape component of the sticker generator; Step 11: Update the shape discriminator using formula (1) and formula (8) respectively. and sticker generator After the training is completed, the sticker generator Ability to generate stickers of different shapes at once to attack the target face recognition system Step 12: Print the generated stickers in color and attach them to the attacker’s face, thereby attacking the target face recognition system in the physical world; Step 13: Target Deep Face Recognition System Take the face image with stickers attached as input and give the final recognition result; Through the target deep face recognition system Verify the validity of the generated stickers and judge and analyze the security of the facial recognition system.

2. The method for detecting the security of a face recognition system based on generation of highly robust adversarial samples according to claim 1, characterized in that: Step 1 is implemented as follows Step 1.1: Prepare the attacker P A Face image dataset D A ; Collect the attacker's face pictures taken under different environmental conditions, including the distance between the attacker and the camera, the ambient brightness, and the attacker's head posture; Step 1.2: Prepare the attacker P A The template shape dataset D s ; Among them, D S is the sticker shape you want to generate, and there is no specific restriction on the sticker shape; the entire dataset D S The distribution of Step 1.3: Use the 3D face reconstruction algorithm to process each image of the attacker to obtain the attacker's auxiliary information Among them, auxiliary information Including 3D face shape and lighting model parameters.

Images