A face verification privacy protection method and device

Abstract

The present invention discloses a face verification privacy protection method and device. By loading the model parameters optimized by the present invention into the perturbation migration network T, corresponding privacy-protected image adversarial samples can be generated according to different facial images to be protected of different users. By uploading the privacy-protected image adversarial samples to the social platform, even after the platform's image compression processing, it cannot perform effective information processing activities after being captured by unauthorized malicious third parties, thereby ensuring the privacy security of users in the image sharing service process. The privacy-protected adversarial samples generated by the method of the present invention are real and natural. Through the two-stage collaborative generation framework of the user end and the server end, it is ensured that the server end does not need to obtain the user's original image in the process of constructing the adversarial sample, effectively avoiding the problem of information leakage of the user image on the server end, and enhancing the security of the privacy protection process itself.

Description

Technical Field

[0001] The present invention relates to a face verification task in the field of face recognition under deep learning, and specifically to a face verification privacy protection method and device. Background Art

[0002] With the growth of hardware computing power and the continuous evolution of algorithms, deep learning techniques, represented by convolutional neural networks, have been widely applied to many machine learning tasks, such as image classification, object detection, and face recognition in computer vision. The main tasks of a face recognition system are divided into two categories: 1) face recognition, which predicts the identity of an object based on an input image; and 2) face verification, which determines whether a user's input image pair represents the same person. In the context of biometric technology, privacy protection requires that an individual's biometric data be used only for specific purposes. Thanks to the continuous development of deep learning models, face recognition systems have surpassed human capabilities, achieving accuracy rates of up to 99% on benchmark datasets. However, the widespread use of face recognition systems also poses certain threats to the protection of personal information, posing potential risks to the privacy of personal photos. On various social media platforms (such as Facebook and Instagram), unauthorized malicious third parties can capture facial images on these platforms and conduct illegal information processing, infringing on the privacy of citizens' biometric information. Therefore, image sharing services on social platforms must protect identity information from excessive identification and exploitation while ensuring the usability of image sharing.

[0003] Research on privacy protection through facial image de-identification can be categorized into image processing de-identification, facial attribute de-identification, and adversarial example-based de-identification. Image processing de-identification is a visual privacy protection method that primarily uses visual processing methods such as blurring and pixelation to de-identify facial images by impairing their readability. However, research has shown that even after image reconstruction, the identity information of the processed images can still be correctly identified. Facial attribute de-identification methods alter facial feature regions in facial images, thereby affecting feature extraction and representation. For example, de-identification by replacing or distorting facial feature regions is performed to achieve privacy protection. Adversarial example-based de-identification involves adding small adversarial perturbations to facial images to construct adversarial examples. These examples can evade recognition and detection by facial recognition systems without changing the visual quality and recognizable features of the facial images, while meeting the usability requirements of social platforms. Adversarial examples can also be used for privacy protection through facial image de-identification, but this process carries the data security issue of requiring direct access to the original images. As cloud services continue to integrate and improve, social platforms are gradually providing cloud services, including privacy protection, to conserve computing power on mobile devices. The data transmission process and temporary image storage both pose potential information leakage risks. These issues primarily fall into the following three categories:

[0004] 1) There is an inability to strike a balance between privacy protection and image usability on social platforms. De-identification methods that blur or pixelate images severely degrade image quality and offer poor privacy protection. De-identification methods that replace or alter facial attributes in facial representation areas alter the subject's facial features, making them unsuitable for social sharing. Privacy protection methods based on adversarial examples, or adversarial examples generated to achieve better privacy protection, often add significant adversarial perturbations that are easily noticeable, resulting in distortion and artifacts that make images unrealistic and unnatural.

[0005] 2) There is a lack of effective privacy protection methods that can achieve good privacy protection for a wider range of face recognition models, and the privacy protection transferability is poor. For privacy protection methods based on adversarial examples, the privacy-preserving adversarial examples generated by white-box adversarial attack methods require understanding the structure and parameters of the target model. The generated adversarial examples are only effective against white-box face recognition models. When attacking other unknown black-box face recognition models, their privacy protection effect is poor, and they cannot achieve good cross-model privacy protection.

[0006] 3) To save image data storage space, social media platforms compress images uploaded by users. This compression removes subtle high-frequency details and perturbations, significantly reducing the privacy protection effectiveness of privacy-preserving adversarial examples.

[0007] In this regard, compared with end-to-end learning of original images, privacy protection based on adversarial samples can be achieved through end-to-end learning of perturbations. Even if the perturbation information is leaked, it cannot be used for effective information processing activities, thereby enhancing the security of the privacy protection process.

[0008] Explanation of terms

[0009] White-box face recognition model: refers to a face recognition model that can obtain all model information and structure;

[0010] Black box face recognition model: refers to a face recognition model that cannot obtain any model information and structure; Summary of the Invention

[0011] The first objective of the present invention is to provide an effective and feasible face verification privacy protection method to address the shortcomings of face verification privacy protection technology in the specific scenario of image sharing on social media platforms.

[0012] The specific implementation method of the present invention is as follows:

[0013] Step 1: The user obtains the face image x to be protected o And preprocess it to obtain the preprocessed original face image x; select another face image belonging to the same object as the original face image x in the dataset as the registered face image x e , in order to perform face verification tasks;

[0014] Step 2: The user inputs the original face image x into the local face recognition model f θ In the example, extract the feature vector information of the original face image x and obtain the feature vector f corresponding to the original face image x. θ (x);

[0015] Step 3: User-side feature vector f θ (x) Use the gradient attack method to perform gradient attack to maximize the user-side loss function L u As the target, argmax(L u (f θ (x+s o ),f θ (x e ))), so that the local face recognition model f θ It is impossible to calculate the eigenvector f θ (x+s o ) and f θ (x e ) to verify the correct face, and obtain the gradient perturbation s after the gradient attack method o ;

[0016] The gradient attack method obtains the gradient perturbation s o The calculation process is shown in the following formula (1):

[0017]

[0018] Where θ represents the local face recognition model f θ The model parameters, ε represents the generated gradient perturbation s o In L ∞ In the perturbation range of space, sign(·) represents the sign function, L u (·) represents the weight of the original face image x and the registered face image x e The user-side loss function for the distance between the feature vectors of ; n represents the number of iterations;

[0019] Step 4: Perturb the gradient s o Upload to the server and learn the target face recognition model f on the server through the perturbation transfer network T θ ''s adversarial perturbation s; the process can be described as:

[0020] s=T(s o ) (2)

[0021] Step 5: The server-side adversarial perturbation s is transmitted back to the user side, and the adversarial perturbation s is added to the original face image x to obtain the privacy-preserving image x. p =x+s;

[0022] Step 6: Add the privacy-preserving image x after adversarial perturbation s p Perform Gaussian filtering to obtain G(x p );

[0023] Step 7: According to G(x p ) and the registered face image x e Under the target face recognition model f θ 'The total loss L between the feature vectors is optimized for training;

[0024] Step 8: After the training is completed, load the model parameters T obtained by the training into the perturbation migration network T θ ;

[0025] Step 9: Based on the face image x to be protected provided by the user o , repeat steps 1 to 5 to obtain the privacy-preserving image x p , the image can be used to protect the user's identity and privacy information. Privacy protection image x p Even after the platform's JPEG compression processing and being captured by malicious unauthorized third parties, privacy can still be effectively protected under face verification using different face recognition models and commercial API interfaces.

[0026] As a preference, the local face recognition model f in step 2 θ It is the MobileFace face recognition model;

[0027] As a preference, the user-side gradient counter-attack method described in step 3 is DI 2 -FGSM method;

[0028] As a preference, the user-side loss function L in step 3 is u Specifically, L u =||f θ (x+s o )-f θ (x e )||2;

[0029] Preferably, the disturbance migration network T in step 4 includes:

[0030] (1) The first convolution with kernel 3*3, stride 2, filters=64, leaky_relu, and jump connection to the seventh convolution with coordinate attention (CA);

[0031] (2) The second convolution with a kernel of 3*3, a stride of 2, filters = 128, and leaky_relu, and is connected to the sixth convolution via a jump with coordinate attention;

[0032] (3) The third convolution with a kernel of 3*3, a stride of 2, filters=256, and leaky_relu, and is connected to the fifth convolution via a jump with coordinate attention;

[0033] (4) The fourth convolution with kernel 3*3, stride 2, filters=512, and leaky_relu;

[0034] (5) The fifth convolution with kernel 3*3, stride 2, filters=256, and leaky_relu;

[0035] (6) The sixth convolution with kernel 3*3, stride 2, filters=128, and leaky_relu;

[0036] (7) The seventh convolution with kernel 3*3, stride 2, filters=64, and leaky_relu;

[0037] As a preference, the server-side target face recognition model f in step 4 θ ' is the Insightface face recognition model;

[0038] Preferably, the Gaussian filtering operation in step 6 is implemented by convolution with a convolution kernel that obeys a two-dimensional Gaussian distribution, and the specific process is shown in formula (3):

[0039] G(x p )=τ3*x p (3)

[0040] Where τ3 represents a Gaussian kernel with a kernel size of 3; * indicates that a convolution operation is performed on the adversarial perturbation s using a 3x3 Gaussian kernel.

[0041] Preferably, the total loss L in step 7 is defined as follows:

[0042] L=min||f θ '(G(x p ))-f θ '(x e )||2 (4)

[0043] Another object of the present invention is to provide a face verification privacy protection device, comprising:

[0044] Perturbation acquisition module, used to obtain the local face recognition model f θ Gradient perturbation s o ;

[0045] Perturbation migration module, used to perturb s according to the gradient o Through the perturbation transfer network T, we learn the target face recognition model f on the server side. θ 's adversarial perturbation s;

[0046] The privacy-preserving image acquisition module adds the adversarial perturbation s to the original face image x to obtain the privacy-preserving image x. p ;

[0047] Frequency domain restriction module, for privacy protection image x p Perform Gaussian filtering to obtain G(x p );

[0048] Parameter optimization module, according to G(X p ) and the registered face image X e Under the target face recognition model f θ The total loss L between the feature vectors of ' is optimized and trained. After the training is completed, the model parameters T obtained by the training are loaded into the perturbation migration network T. θ .

[0049] Yet another object of the present invention is to provide a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor implements the above method when executing the executable code.

[0050] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the above method.

[0051] The present invention has the following beneficial effects:

[0052] 1. The present invention proposes a face verification privacy protection method based on adversarial samples. Compared with traditional identity de-identification methods and facial attribute de-identification methods, the method of the present invention can ensure that the generated images are real and natural, without affecting the facial features of the original object, and meet the availability requirements of image sharing services on social media platforms. Compared with other methods based on adversarial samples, the present invention avoids the leakage of image information on the server side through a two-stage collaborative method between the user side and the server side, thereby improving the security of privacy protection services. Compared with the same type of end-cloud collaborative adversarial attack method, since the general perturbation loss is highly correlated with the training data set and will reduce the visual quality of the image, more facial feature information can be retained by adding the original image and the perturbation, thereby obtaining a better privacy protection effect. The face verification privacy protection method proposed in the present invention has been experimentally proven to be able to achieve good privacy protection effects on different data sets and face recognition models, and has good cross-model transferability.

[0053] 2. The proposed method introduces a frequency domain restriction module, applying Gaussian filtering to the privacy-preserving adversarial examples after the adversarial perturbation has been applied. This low-pass filter filters out high-frequency information in the image, reducing the frequency coefficients of the high-frequency components and, to a certain extent, suppressing the high-frequency components of the image. This Gaussian filter helps to limit the distribution of the adversarial perturbation, concentrating it more on the low-frequency regions of the image, thereby improving the adversarial example's ability to withstand compression.

[0054] 3. The present invention introduces coordinate attention into the perturbation transfer network to supplement the coordinate feature information related to the representation of facial images, enhance the ability to extract effective perturbation information, reduce redundant perturbations, improve the quality of adversarial sample images, and enhance the aggressiveness of adversarial perturbations. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a framework diagram of each module of the method of the present invention.

[0056] Figure 2 The present invention is a flowchart of applying the method of the present invention to protect face privacy on social platforms.

[0057] Figure 3 It is a structural diagram of the perturbation migration network in the method of the present invention.

[0058] Figure 4It is a coordinate attention structure diagram in the method of the present invention.

[0059] Figure 5 The method of the present invention uses different gradient attack methods to generate privacy-preserving adversarial samples in different data sets.

[0060] Figure 6 The method is used to compare the frequency coefficient histograms after JPEG compression and Gaussian filtering in the frequency domain limitation module of the image block method of the present invention.

[0061] Figure 7 It is the quality of privacy-preserving adversarial sample images generated by the method of the present invention for different data sets and different JPEG compressions. DETAILED DESCRIPTION

[0062] The present invention will be further analyzed below in conjunction with specific implementation.

[0063] 1. Such as Figure 1 This is a schematic diagram of the various modules of a face verification attack privacy protection method, which includes the following steps:

[0064] Step 1: The present invention cleans the public image dataset MS-Celeb-1M for training and selects a subset as the training set of the present invention. The user first performs a face image x o Crop the image to a uniform size of 112x112, divide the pixel value of each pixel by 127.5 and subtract 1.0 for normalization, and obtain the original face image x after preprocessing. Select another face image of the same object as the original face image x in the dataset as the registered face image x e , in order to perform face verification tasks;

[0065] Step 2: The user terminal inputs the original face image x into the local face recognition model MobileFace to extract the feature vector information of the image. The local face recognition model is denoted as f θ , extract the feature vector f of the original face image x θ (x);

[0066] Step 3: The user terminal generates the feature vector f of the original face image x θ (x), according to DI 2 -FGSM gradient attack method attacks to maximize the user-side loss function L u =||f θ (x+s o )-f θ (x e )||2 is the target, that is, argmaxL u. Make the local face recognition model MobileFace(f θ ) cannot be calculated based on the eigenvector f θ (x) and f θ (x e ) to correctly perform face verification. After the gradient attack method is used to perform gradient attack, the gradient perturbation s of the original face image x is obtained o The calculation process of the gradient perturbation obtained by the gradient attack method is shown in the following formula (5):

[0067]

[0068] Where θ represents the model parameters of the face recognition model MobileFacenet, and ε makes the generated gradient in L ∞ In the perturbation range of space, sign(·) represents the sign function, L u (·) represents the weight between the face image x and the registered image x e The user-side loss function for the distance between the feature vectors of ; n represents the number of iterations;

[0069] Step 4: Perturb the gradient s o Upload to the server and learn the gradient perturbation s through the perturbation migration network T o Corresponding to the server-side target face recognition model Insightface(f θ '). The process can be described as:

[0070] s=T(s o ) (6)

[0071] Step 5: The server-side adversarial perturbation s is transmitted back to the user-side. The adversarial perturbation s is added to the original image x, and then clipped to a reasonable pixel value range to obtain the privacy-preserving image x. p =clip(x+s);

[0072] Step 6: Privacy-preserving adversarial sample x p Use the convolution kernel that obeys the two-dimensional Gaussian distribution to perform convolution and perform Gaussian filtering to obtain G(x p ), the calculation process is shown in the following formula (7);

[0073] G(x p )=τ3*x p (7)

[0074] Where τ3 represents a Gaussian kernel with a kernel size of 3; * indicates that a convolution operation is performed using a 3x3 Gaussian kernel.

[0075] Step 7: G(x p ) and the registered face image x eIn the target face recognition model f θ ', optimize according to the total loss L between the two feature vectors, and the calculation process is as follows (8);

[0076] L=min||f θ '(G(x p ))-f θ '(x e )||2 (8)

[0077] Step 8: After the training is completed, load the model parameters T obtained by the training into the perturbation migration network T θ ;

[0078] Step 9: Based on the face image x to be protected provided by the user o , repeat steps 1 to 5 to obtain the privacy-preserving image x p , the image can be used to protect the user's identity and privacy information. Privacy protection image x p Even after the platform's JPEG compression processing and being captured by malicious unauthorized third parties, privacy can still be effectively protected under face verification using different face recognition models and commercial API interfaces.

[0079] Figure 2 This is a flowchart of the method of the present invention applied to face privacy protection on social platforms. By jointly constructing privacy protection adversarial samples on the user side and the server side of the social platform, the user side constructs the gradient perturbation s of the lightweight face recognition model according to the gradient attack algorithm. o ; The social platform server perturbs s based on the uploaded gradient o Learn to counter the privacy perturbation x, send it back to the user end and superimpose it with the original image x to obtain the privacy-preserving adversarial sample x p . User uploads the image x p To the social platform, even if the JPEG (x p ) The image cannot be obtained by a malicious third party and cannot be used for effective information processing activities, thus protecting the privacy of users.

[0080] 2. Figure 3 : is a structural diagram of the disturbance migration network T in the present invention. The disturbance migration network T includes:

[0081] (1) The first convolution with kernel 3*3, stride 2, filters=64, leaky_relu, and jump connection to the seventh convolution with coordinate attention (CA);

[0082] (2) The second convolution with a kernel of 3*3, a stride of 2, filters = 128, and leaky_relu, and is connected to the sixth convolution via a jump with coordinate attention;

[0083] (3) The third convolution with a kernel of 3*3, a stride of 2, filters=256, and leaky_relu, and is connected to the fifth convolution via a jump with coordinate attention;

[0084] (4) The fourth convolution with kernel 3*3, stride 2, filters=512, and leaky_relu;

[0085] (5) The fifth convolution with kernel 3*3, stride 2, filters=256, and leaky_relu;

[0086] (6) The sixth convolution with kernel 3*3, stride 2, filters=128, and leaky_relu;

[0087] (7) The seventh convolution with kernel 3*3, stride 2, filters=64, and leaky_relu;

[0088] The structure diagram of coordinate attention is as follows Figure 4 As shown, it includes two parallel global average pooling, convolution, BN, ReLu activation function, two parallel convolutions, Sigmoid activation function and redistribution weight block in sequence; the redistribution weight block is used to redistribute the weights of the original input features and the two parallel convolution output features and then multiply them.

[0089] 3. In the training phase, the method of the present invention uses MobileFace as the local face recognition model and Insightface as the target face recognition model on the server side. In the inference phase, the method of the present invention can automatically generate privacy-preserving adversarial samples based on different images. Therefore, all models except the Insightface face recognition model involved in the training are regarded as black-box face recognition models. For different face recognition models, there is a face verification task threshold τ based on Euclidean distance under certain FAR requirements. When the Euclidean distance between the feature vectors of the image pair is greater than the threshold τ, it can be judged that the two do not belong to the same person, otherwise they belong to the same person. In order to verify the effectiveness of the generation of the method of the present invention, the present invention uses the attack success rate (ASR) as an evaluation indicator of the effect of privacy-preserving adversarial samples. ASR represents the probability of successfully misleading the target face recognition model in the face verification task. Different images belonging to the same object are judged by the target face recognition model to not belong to the same person, which is considered to have achieved privacy protection. The calculation method of ASR is shown in the following formula (9):

[0090]

[0091] Where xp and x e denotes the privacy-preserving adversarial example and the registered image in the i-th group, N represents the total number of image pairs for the face verification task, τ represents the discrimination threshold of the face recognition model Arcface calculated at a FAR of 1%, and d represents the Euclidean distance calculated in the feature space. When the Euclidean distance between the adversarial example and the registered image is greater than the threshold, the adversarial example is considered privacy-preserving. ASR ∈ [0, 1], where a higher ASR indicates a better adversarial example's attack effectiveness.

[0092] In order to avoid the leakage of user image information on the server side, the present invention replaces the original image with the gradient perturbation generated by the user side and uploads it to the server side. The server side generates adversarial perturbations through the perturbation migration network. In order to generate adversarial perturbations that can retain the original object features and improve the privacy protection effect of privacy-preserving adversarial samples, the user side needs to provide more effective gradient perturbations. To this end, the present invention compares different gradient attack algorithms (FGSM, I-FGSM, MI-FGSM and DI) on the user side. 2 -FGSM) privacy protection effect under different data sets, the attack results of adversarial samples on Arcface face recognition model are shown in Table 1 below. The generated privacy protection adversarial samples are as follows Figure 5 shown.

[0093] Figure 5 The present invention generates privacy-preserving adversarial samples in different data sets using different gradient attack methods.

[0094] Table 1 Attack success rate (%) of different data sets using different gradient attack methods in the present invention

[0095]

[0096] From the data in Table 1, we can see that different gradient adversarial attack methods can achieve a good attack success rate in the white-box face recognition model. Compared with the single-step FGSM method, the adversarial samples generated by the iterative gradient method I-FGSM after multiple iterations of optimization have better adversarial effects. 2 -FGSM can significantly improve the attack success rate of adversarial samples.

[0097] In order to make the gradient transfer network T pay more attention to the effective area, suppress useless information, improve the privacy protection effect and the visual quality of the image, the present invention introduces the attention mechanism in the jump connection. 2 -FGSM is used as a gradient attack method. The success rate of adversarial attacks on the Arcface face recognition model after adding different attention mechanisms is shown in Table 2 below.

[0098] Table 2 The success rate of adversarial attacks using different attention mechanisms in different datasets (%)

[0099]

[0100] From the data in Table 2, we can see that adding different attention methods can improve the success rate of the adversarial attack to a certain extent. Coord CBAM achieves the best results. Building on SE, CBAM introduces spatial attention, focusing on the spatial structure of the image through global pooling. It performs better on datasets with good acquisition conditions, such as LFW, CALFW, and AgeDB-30, but performs poorly on datasets with different poses, such as CPLFW and CFP-FP. Coordinate attention (Coord) perceives channel information in the vertical and horizontal directions to form a pair of direction-aware and position-sensitive feature maps. These maps are then applied to the input feature map to enhance target representation, enabling the model to more accurately locate and identify object regions. This increases the weight of these regions to suppress background regions, improving attack success rates while avoiding excessive computational overhead.

[0101] Figure 4 This is a coordinate attention structure diagram used by the perturbation migration network in the method of the present invention.

[0102] To verify that the CSP3Adv frequency domain restriction module helps improve the attack success rate after JPEG compression, we tested multiple datasets using JPEG compression methods with different compression quality factors. The experimental results are shown in Table 3. The first and second rows of each dataset are the attack success rates of CSP3Adv before and after the frequency domain restriction module is used.

[0103] Table 3 Attack success rate (%) of the privacy-preserving adversarial samples generated by the present invention on the Arcface face recognition model after compression with different JPEG compression quality factors

[0104]

[0105] As can be seen from the data in Table 3, after the frequency limiting module is adopted, the aggressiveness of the adversarial samples under different compression quality factors is improved, which improves the privacy protection effect of unknown compression methods. In addition, in order to verify the effect of the frequency limiting module, the method of the present invention randomly selects two sub-blocks of the original face image for DCT (Discrete Cosine Transform) transformation and extracts the frequency coefficient histogram according to the Zigzag method, as shown in the figure. Figure 6 (b); the same operation is performed on the image transformed by Gaussian filtering with a window size of 3*3 and a standard deviation of 3 and the image processed by JPEG80 compression method to obtain Figure 6 (c) and Figure 6 (d) Frequency coefficient histogram. The larger the coordinate value of the frequency coefficient histogram, the closer it is to the high frequency area.

[0106] Compare Figure 6 The frequency coefficient histograms (b)-(d) show that the JPEG compression method can set visually insensitive high-frequency frequency coefficients to zero, reducing the number of nonzeros in the frequency coefficient matrix and thus reducing image storage space. The Gaussian filtering method, on the other hand, filters the high-frequency information in the image using a low-pass filter, reducing the frequency coefficients of the high-frequency components and, to a certain extent, suppressing the high-frequency components of the image. Both JPEG compression and Gaussian filtering can suppress the high-frequency components in the original image. Therefore, filtering the image with Gaussian filtering during the training phase helps to limit the distribution of adversarial perturbations, distributing them more in the low-frequency regions of the image, thereby improving the compression resistance of adversarial examples.

[0107] Figure 7 These are the privacy-preserving adversarial samples and image quality evaluation results of the method of the present invention under JPEG compression with different quality compression factors.

[0108] Since the face recognition model used by the server cannot represent all unknown malicious third-party face recognition models, in order for the adversarial sample to have a privacy protection effect on multiple face recognition models at the same time, it is necessary to ensure that it has good attack transferability. In order to verify the transferability of the privacy protection method proposed in this invention, a total of 3 SOTA face recognition models and 2 commercial face recognition model API interfaces are used as face recognition models to conduct privacy protection effect experiments. Among them, the public face recognition models include FaceNet, InsightFace iresnet100 and VGG-Face. Here, the adversarial example training process only white-boxes access to the local lightweight face recognition model MobileFaceNet and the output information of the server-side Insightface model, without access to information from other face recognition models. The final adversarial attack effects on different face recognition models and commercial API interfaces are shown in Table 4.

[0109] Table 4 Attack success rate (%) of privacy-preserving adversarial samples generated by the present invention on different face recognition models

[0110]

[0111] From the data in Table 4, we can see that the adversarial samples generated by the proposed method for different datasets can achieve effective migration on a variety of black box face recognition models. Among them, the attack success rate of more than 98% was achieved on FaceNet and BaiduAPI, and on Insightface iresnet100We achieved an attack success rate of over 80% on the Face++ API and over 60% on VGG-Face. This is because VGG-Face uses a deeper backbone network than ArcFace, which results in a more robust feature representation. Therefore, the attack transfer effect is lower than that of other models.

Claims

1. A face verification privacy protection method, characterized in that The method comprises the following steps: Step 1: The user obtains the face image x to be protected o And preprocess it to obtain the preprocessed original face image x; select another face image belonging to the same object as the original face image x in the dataset as the registered face image x e , in order to perform face verification tasks; Step 2: The user inputs the original face image x into the local face recognition model f θ In the example, extract the feature vector information of the original face image x and obtain the feature vector f corresponding to the original face image x. θ (x); Step 3: User-side feature vector f θ (x) Use the gradient attack method to perform gradient attack to maximize the user-side loss function L u As the target, that is, argmax(L u (f θ (x+s o ),f θ (x e ))), so that the local face recognition model f θ It is impossible to calculate the eigenvector f θ (x+s o ) and f θ (x e ) to verify the correct face, and obtain the gradient perturbation s after the gradient attack method o ; The gradient attack method obtains the gradient perturbation s o The calculation process is shown in the following formula (1): Where θ represents the local face recognition model f θ The model parameters, ε represents the generated gradient perturbation s o In L ∞ In the perturbation range of space, sign(·) represents the sign function, L u (·) represents the weight of the original face image x and the registered face image x e The user-side loss function for the distance between the feature vectors of ; n represents the number of iterations; Step 4: Perturb the gradient s o Upload to the server and learn the target face recognition model f on the server through the perturbation transfer network T θ ''s adversarial perturbation s; the process can be described as: s=T(s o ) (2) Step 5: The server-side adversarial perturbation s is transmitted back to the user side, and the adversarial perturbation s is added to the original face image x to obtain the privacy-preserving image x. p =x+s; Step 6: Add the privacy-preserving image x after adversarial perturbation s p Perform Gaussian filtering to obtain G(x p ); Step 7: According to G(x p ) and the registered face image x e Under the target face recognition model f θ 'The total loss L between the feature vectors is optimized for training; Step 8: After the training is completed, load the model parameters T obtained by the training into the perturbation migration network T θ ; Step 9: Based on the face image x to be protected provided by the user o , repeat steps 1 to 5 to obtain the privacy-preserving image x p .

2. The method according to claim 1, characterized in that The local face recognition model f described in step 2 θ Use the MobileFace face recognition model; the server-side target face recognition model f described in step 4 θ ' is the Insightface face recognition model.

3. The method according to claim 1, characterized in that The user-side gradient adversarial attack method described in step 3 is the DI2-FGSM method.

4. The method according to claim 1, characterized in that The user-side loss function L described in step 3 u Specifically, L u =||f θ (x+s o )-f θ (x e )||2.

5. The method according to claim 1, characterized in that: The disturbance migration network T described in step 4 includes: (1) The first convolution with kernel 3*3, stride 2, filters=64, leaky_relu, and jump connection to the seventh convolution with coordinate attention (CA); (2) The second convolution with a kernel of 3*3, a stride of 2, filters = 128, and leaky_relu, and is connected to the sixth convolution via a jump with coordinate attention; (3) The third convolution with a kernel of 3*3, a stride of 2, filters=256, and leaky_relu, and is connected to the fifth convolution via a jump with coordinate attention; (4) The fourth convolution with kernel 3*3, stride 2, filters=512, and leaky_relu; (5) The fifth convolution with kernel 3*3, stride 2, filters=256, and leaky_relu; (6) The sixth convolution with kernel 3*3, stride 2, filters=128, and leaky_relu; (7) The seventh convolution with kernel 3*3, stride 2, filters=64, and leaky_relu.

6. The method according to claim 1, characterized in that The Gaussian filtering operation in step 6 is implemented by convolution with a convolution kernel that obeys a two-dimensional Gaussian distribution. The specific process is shown in formula (3): G(x p )=τ3*x p (3) Where τ3 represents a Gaussian kernel with a kernel size of 3; * indicates that a convolution operation is performed on the adversarial perturbation s using a 3x3 Gaussian kernel.

7. The method according to claim 1, characterized in that: The total loss L described in step 7 is defined as follows: L=min||f θ '(G(x p ))-f θ '(x e )||2 (4)。 8. A face verification privacy protection device implementing the method according to any one of claims 1 to 7, comprising: Perturbation acquisition module, used to obtain the local face recognition model f θ Gradient perturbation s o ; Perturbation migration module, used to perturb s according to the gradient o Through the perturbation transfer network T, we learn the target face recognition model f on the server side. θ 's adversarial perturbation s; The privacy-preserving image acquisition module adds the adversarial perturbation s to the original face image x to obtain the privacy-preserving image x. p ; Frequency domain restriction module, for privacy protection image x p Perform Gaussian filtering to obtain G(x p ); Parameter optimization module, according to G(X p ) and the registered face image X e Under the target face recognition model f θ The total loss L between the feature vectors of ' is optimized and trained. After the training is completed, the model parameters T obtained by the training are loaded into the perturbation migration network T. θ .

9. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images