A hybrid perturbation differential privacy protection method for face recognition

By constructing a hybrid perturbation differential privacy model in deep neural networks, segmenting network weights and changing triple loss functions, the problem of deterioration in usability caused by the privacy protection method of deep neural networks is solved, and better recognition effect and privacy protection are achieved.

CN114998971BActive Publication Date: 2024-11-22GUILIN UNIVERSITY OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210743072.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-27
Publication Date
2024-11-22
Estimated Expiration
2042-06-27

AI Technical Summary

Technical Problem

Existing methods for privacy protection of deep neural networks can easily lead to poor model availability and affect the optimization process of deep neural networks.

Method used

Using the hybrid perturbation differential privacy protection method, by constructing a hybrid perturbation differential privacy model, segmenting the neural network weight into network weight and noise weight, and using Bayesian neural network to change the triple loss function, so that the model can effectively protect privacy during training. And maintain learning ability.

Benefits of technology

It improves the availability and model accuracy of deep neural networks, and can obtain better identification results under the same privacy overhead, solving the problem of deterioration in model availability caused by existing privacy protection methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114998971B_ABST
    Figure CN114998971B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of privacy protection for deep learning, and specifically to a hybrid perturbation differential privacy protection method for face recognition, comprising capturing face data; preprocessing the face data to obtain a minimum detection frame; relocating the face in the minimum detection frame to obtain training data; constructing a hybrid perturbation differential privacy model based on the training data; and changing a triplet loss function so that the hybrid perturbation differential privacy model obtains hybrid perturbation neuron node weights after being trained with the training data. The hybrid perturbation differential privacy model divides the weights of a neural network into network weights and noise weights, so that the network weights can try their best to provide learning capabilities for the neural network, while the noise weights use target perturbations to try their best to provide privacy protection for the network weights, thereby solving the problem of poor availability of deep neural networks in existing privacy protection methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning privacy protection, and in particular to a hybrid perturbation differential privacy protection method for face recognition. Background Art

[0002] The differential privacy protection technology of target perturbation and gradient perturbation introduces Gaussian noise during the optimization process of deep neural networks to perturb the optimization direction of the model parameters of the deep neural network to achieve the effect of protecting the training data of the deep neural network. Input perturbation and output perturbation are a direct perturbation method. The former directly samples noise samples from the probability distribution that satisfies differential privacy and adds them to the data to perturb the data itself to achieve the effect of privacy protection. The latter directly adds noise to the optimal parameters of the neural network.

[0003] However, these methods as privacy protection methods for deep neural networks will bring an obvious problem, that is, they can easily lead to poor usability of deep neural networks. Summary of the invention

[0004] The purpose of the present invention is to provide a hybrid perturbation differential privacy protection method for face recognition, aiming to solve the problem of poor availability of deep neural networks in existing privacy protection methods.

[0005] To achieve the above object, the present invention provides a hybrid perturbation differential privacy protection method for face recognition, comprising the following steps:

[0006] Capture facial data;

[0007] Preprocessing the face data to obtain a minimum detection frame;

[0008] Relocating the face of the minimum detection frame to obtain training data;

[0009] Construct a hybrid perturbation differential privacy model;

[0010] The triplet loss function is changed so that the mixed perturbation differential privacy model obtains the mixed perturbation neuron node weights after being trained with the training data.

[0011] Among them, the model for preprocessing the facial data is an MTCNN model, and the MTCNN model includes P-Net, R-Net and O-Net.

[0012] The specific method of preprocessing the face data to obtain the minimum detection frame is:

[0013] Constructing an image pyramid based on the face data;

[0014] Input each image in the image pyramid into the P-Net for coordinate detection to obtain a preliminary candidate face detection frame;

[0015] Inputting the preliminary candidate face detection frame into the R-Net for screening to obtain a screened candidate face detection frame;

[0016] The screened candidate face detection frame is scaled by the O-Net to obtain a minimum detection frame.

[0017] Among them, the hybrid perturbation differential privacy model includes a FaceNet framework structure, the FaceNet framework structure includes a black box deep learning model, an L2 regularization module and an original triplet loss function, and the black box deep learning model is Inception-ResNet-v2.

[0018] The specific method of changing the triplet loss function so that the mixed perturbation differential privacy model obtains the mixed perturbation neuron node weights after being trained with the training data is as follows:

[0019] The triple loss function is changed based on the Bayesian neural network so that the mixed perturbation differential privacy model obtains the mixed perturbation neuron node weights after being trained with the training data.

[0020] The specific method of training the hybrid perturbation differential privacy model is as follows:

[0021] Setting model parameters of the privacy hybrid perturbation differential privacy model;

[0022] The mixed perturbation differential privacy model after model parameter setting samples the training data to obtain extracted samples;

[0023] Calculating a current network weight value and a current noise weight value of the privacy hybrid perturbation differential privacy model based on the extracted samples;

[0024] Obtaining a neural network weight based on the current network weight value and the current noise weight value;

[0025] Performing matrix multiplication of the neural network weights and the output of the Inception-ResNet-v2 to obtain a feature vector;

[0026] The feature vector is extracted to construct a triplet loss function, and the triplet loss function is subjected to target perturbation, and then the FaceNet framework structure is trained.

[0027] The present invention discloses a hybrid perturbation differential privacy protection method for face recognition, which comprises the following steps: capturing face data; preprocessing the face data to obtain a minimum detection frame; relocating the face in the minimum detection frame to obtain training data; constructing a hybrid perturbation differential privacy model based on the training data; and changing the triplet loss function so that the hybrid perturbation differential privacy model obtains hybrid perturbation neuron node weights after being trained with the training data. The hybrid perturbation differential privacy model divides the neural network weights into network weights and noise weights, so that the network weights can try their best to provide learning capabilities for the neural network, while the noise weights use target perturbations to try their best to provide privacy protection for the network weights, thereby solving the problem that target perturbations, gradient perturbations, output perturbations, and input perturbations easily lead to poor model availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 It is a flow chart of a mixed perturbation differential privacy protection method in face recognition provided by the present invention.

[0030] Figure 2 It is a line graph of a hybrid perturbation differential privacy protection method in face recognition provided by the present invention.

[0031] Figure 3 is a neuron based on mixed perturbations.

[0032] Figure 4 It is a differentially private Bayesian neural network based on mixed perturbations. DETAILED DESCRIPTION

[0033] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0034] See also Figures 1 to 4 The present invention provides a hybrid perturbation differential privacy protection method for face recognition, comprising the following steps:

[0035] S1 captures face data;

[0036] Specifically, the face data comes from the Internet, so the face data is crawled by a web crawler. A web crawler is a program that automatically crawls web pages. Traditional crawlers start from the URL of one or more web pages. After obtaining the URL of the initial web page, they continuously extract new URLs from the current page and save them until the termination condition of the program is met.

[0037] In this example, Python is mainly used to crawl the facial data of Chinese celebrities. Specifically, (1) a crawler is used to obtain the celebrity column in the Baidu search bar to obtain a list of celebrity names and save it as a text file; (2) a third-party library icrawler is used to crawl the celebrity images on Baidu Pictures according to each name, and 2,000 images are crawled for each person;

[0038] S2 preprocesses the face data to obtain a minimum detection frame;

[0039] Specifically, the model for preprocessing the facial data is an MTCNN model, and the MTCNN model includes P-Net, R-Net and O-Net.

[0040] The specific method of preprocessing the face data to obtain the minimum detection frame is:

[0041] S21 constructs an image pyramid based on the face data;

[0042] Specifically, the face data is constructed into an image pyramid D, where D = {d·γ 1 ,d·γ 2 ,d·γ 3 ,………,d·γ n}, where n is determined by the end conditions r(dx)>12 and r(dy)>12, that is, the smallest image size of the image pyramid is 12×12.

[0043] S22 inputs each image in the image pyramid into the P-Net for coordinate detection to obtain a preliminary candidate face detection frame;

[0044] Specifically, P-Net will output three types of data. The first data is a 1×1×2 classifier, which indicates whether there is a face. The second is a 1×1×4 detection frame coordinate position, which represents the coordinate offset of the upper left and lower right of the detection frame. The third data is a 1×1×10 feature point coordinate. Feature points include the position of the eyes, nose, mouth, etc. After the image pyramid is input, a large number of detection frames are obtained. There are two ways to filter the detection frames. The first is to filter based on whether the classifier has a face. The second is to use the IOU value for non-maximum suppression (NMS) to filter out smaller IOU values. The IOU threshold can be set to 0.7. Based on the remaining detection frames, extract the images selected by the detection frames of the corresponding original image, and then scale these extracted images to 24×24.

[0045] S23 inputs the preliminary candidate face detection frame into the R-Net for screening to obtain a screened candidate face detection frame;

[0046] Specifically, the output of R-Net is consistent with that of P-Net, and the way R-Net filters the detection box is consistent with that of P-Net.

[0047] S24 scales the screened candidate face detection frame through the O-Net to obtain a minimum detection frame.

[0048] Specifically, the screened candidate face detection frame is scaled to a size of 48×48 by the O-Net, and then screened again to output the minimum IOU detection frame.

[0049] S3 performs face relocation on the minimum detection frame to obtain training data;

[0050] Specifically, in face data relocation, the detection frame generated in the image preprocessing module is used to crop the face position of the original image. For the cropped face position, the feature point position landmark is used to align all faces so that the feature point positions of all training data can be unified when training face recognition.

[0051] S4 builds a hybrid perturbation differential privacy model;

[0052] Specifically, FaceNet is a unified solution proposed by Google for problems such as recognition, verification, and clustering. These downstream tasks can be processed in a unified feature space, and the upstream task is to solve how to better map faces to the feature space. FaceNet essentially maps face images to a 128-dimensional Euclidean space through a convolutional neural network, which means that the upstream task of FaceNet maps face data into a 128-dimensional vector.

[0053] The hybrid perturbation differential privacy model includes a FaceNet framework structure, which includes a black box deep learning model, an L2 regularization module and an original triplet loss function. The black box deep learning model is Inception-ResNet-v2, which is a network that introduces a residual network into Inception to prevent the Inception network from degenerating when the network is deepened. The structure of ResNet can accelerate training and improve performance. The Iception module can obtain sparse features on the same layer. The final output of Inception-ResNet-v2 is a 1000-dimensional vector, and this 1000-dimensional vector needs to be linearly mapped before passing through the feature normalization module. This mapping is to map this 1000-dimensional vector into a 128-dimensional vector, and then the 128-dimensional vector is subjected to L2 regularization constraints and input into the triplet loss function.

[0054] In this example, in order to perform differential privacy on the training data, its improvement is reflected in the L2 regularization module, such as Figure 2 As shown in the figure, it is performed after the output of Inception-ResNet-v2 and before L2 normalization, and the following changes are made to the neuron linear transformation:

[0055] h(x)=x·W

[0056] Where W is a weight of size 1000×128, and W = (1-v)·w+v·γ, w is the network weight, γ is the noise weight, v is the noise contribution, x is the model input value, “·” is the multiplication operation, 0≤v≤1.

[0057] The structure of neurons combined after each output perturbation is as follows Figure 3 As shown, A represents v·γ·x, B represents (1-v)·w·x, and C represents W·x. The main structure is still the same as that of ordinary neural networks. The difference is that the weights of traditional neural networks are divided into network weights w and noise weights γ, and the corresponding deviations are also divided into neuron deviations and noise deviations. Among them, v determines the contribution of noise weight to W, and (1-v) determines the contribution of network weight to W. Generally, v<0.5. Figure 3 SELU is the SELU activation function and y is the output of the neuron.

[0058] S5 changes the triplet loss function so that the mixed perturbation differential privacy model obtains the mixed perturbation neuron node weights after being trained with the training data.

[0059] Specifically, the triple loss function is changed based on the L2 regularization module so that the mixed perturbation differential privacy model starts training through the training data to obtain the neuron node weights.

[0060] Since the Bayesian neural network layer is added to the L2 regularization module, the loss function needs to be changed. The loss function of the Bayesian neural network is as follows:

[0061] F(D,θ w )=KL[q(w|θ w )||p(w)]-E q(w|θw) [lnp(D|w)]

[0062] The loss function of the Bayesian neural network actually uses KL divergence to find a probability distribution to fit the posterior probability distribution of the weights. The neural network with output perturbations needs to add target perturbations to ensure the differential privacy of the noise weights. Therefore, the loss function of the Bayesian neural network with mixed perturbation differential privacy is:

[0063]

[0064] is the distribution of arbitrary noise, θ γ and θ w It is the algorithm F(D,θ w ,θ γ ) is the parameter to be learned, w is the network weight, γ is the noise weight, D is the data set, q(γ i +N(μ,σ 2 )|θ γ ) is the posterior probability estimate after the i-th noise weight target perturbation, q(w i |θ w ) is the posterior probability estimate of the i-th network weight, p(γ i ) is the prior probability of the i-th noise weight, p(w i ) is the prior probability of the i-th network weight, p(D|γ i ,w i ) is about D parameter γ i and w i The likelihood function of .

[0065] The original triplet loss function of FaceNet is:

[0066]

[0067] ||·||2 is L2 regularization, represents the output value of the module after the face image passes through Inception-ResNet-v2 to L2 regularization, x a and xp Represents the face data of the same person, x n Represents x a and x p Face data of different people. Mixed perturbation differential privacy of Bayesian neural network x a to x p The distance loss function can be expressed as:

[0068]

[0069] So the triplet loss function of FaceNet with mixed perturbation differential privacy can be expressed as

[0070]

[0071] At this point, the model can begin training.

[0072] The specific method of training the hybrid perturbation differential privacy model to obtain the weights of neuron nodes is as follows:

[0073] S51 sets the model parameters of the privacy hybrid perturbation differential privacy model;

[0074] Specifically, define their respective model parameters θ w =(μ w ,ρ w ) and θ γ =(μ γ ,ρ γ ), learning rate η, and noise contribution v, the mathematical expectation μ and standard deviation σ of the target perturbation. θ w =(μ w ,ρ w ), θ γ =(μ γ ,ρ γ ) / / Use Gaussian distribution N(0,0.1 2 ) Initialize the network weights and noise weights μ w and μ γ , using Gaussian distribution N(-3.0,0.1 2 ) Initialize network weights and noise weights ρ w and ρ γ .

[0075] S52: the mixed perturbation differential privacy model after model parameter setting samples the training data to obtain extracted samples ζ~N(0,I);

[0076] Specifically, ζ~N(0,I) is sampled from a Gaussian distribution N(0,I), where I=1.0.

[0077] S53 calculates the current network weight value w and the current noise weight value γ of the privacy hybrid perturbation differential privacy model based on the extracted samples;

[0078] Specifically,

[0079] S54 obtains a neural network weight W based on the current network weight value and the current noise weight value;

[0080] Specifically, W = (1-v)·w+v·γ.

[0081] S55 performs matrix multiplication of the neural network weight and the output of the Inception-ResNet-v2 to obtain a feature vector.

[0082] Specifically, the feature vector is a 128-dimensional feature vector. The “·” operation is a multiplication, and the reparameterization is to be able to update the posterior distribution by gradient descent.

[0083] S56 extracts the feature vector to construct a triplet loss function, performs target perturbation on the triplet loss function, and then starts training the FaceNet framework structure.

[0084] Specifically, the output perturbation makes the network weights of the neural network satisfy differential privacy. The product of the noise weight γ obtained in the previous step and the noise contribution v is added to the network weight w scaled by (1-v) times to obtain the perturbed neuron node weights. The noise contribution v enables the algorithm to degenerate into a Bayesian neural network without privacy protection. Since the network weights are not perturbed during the back propagation process, the network weights will fit the data set as much as possible to obtain a weight distribution that is highly correlated with the training set. After training, set v = 0.0, and the neural network will lose its privacy protection ability. The unperturbed network weights also lay the foundation for the calculation of the forward propagation privacy overhead. In contrast, the network trained with deep learning differential privacy can only obtain the perturbed weight distribution.

[0085] In this way, the feature vector extracted by Inception-ResNet-v2 can be output perturbated, but in order for the output perturbation to complete the perturbation task, it is necessary to add target perturbation to the triplet loss function during training.

[0086] The present invention separates the privacy protection part of the neural network, that is, divides the neural network weights into network weights and noise weights, so that the network weights can try their best to provide learning ability for the neural network, while the noise weights use target perturbations to try their best to provide privacy protection for the network weights. Compared with FaceNet with gradient perturbation differential privacy and FaceNet with target perturbation differential privacy, the FaceNet model with mixed perturbation differential privacy can obtain better model accuracy at the same privacy cost.

[0087] What is disclosed above is only a preferred embodiment of a hybrid perturbation differential privacy protection method for face recognition of the present invention. Of course, this cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A hybrid perturbation differential privacy protection method for face recognition, characterized in that: The following steps are involved: Capture facial data; Preprocessing the face data to obtain a minimum detection frame; Relocating the face of the minimum detection frame to obtain training data; Construct a hybrid perturbation differential privacy model; Changing the triplet loss function so that the mixed perturbation differential privacy model obtains mixed perturbation neuron node weights after being trained with the training data; The hybrid perturbation differential privacy model includes a FaceNet framework structure, the FaceNet framework structure includes a black box deep learning model, an L2 regularization module and a triple loss function, and the black box deep learning model is Inception-ResNet-v2; The specific method of changing the triplet loss function so that the mixed perturbation differential privacy model obtains the mixed perturbation neuron node weights after being trained with the training data is: Changing the triplet loss function based on a Bayesian neural network so that the mixed perturbation differential privacy model obtains mixed perturbation neuron node weights after being trained with the training data; The specific method of training the hybrid perturbation differential privacy model is as follows: Setting model parameters of the privacy hybrid perturbation differential privacy model; The mixed perturbation differential privacy model after model parameter setting samples the training data to obtain extracted samples; Calculating a current network weight value and a current noise weight value of the privacy hybrid perturbation differential privacy model based on the extracted samples; Obtaining a neural network weight based on the current network weight value and the current noise weight value; Performing matrix multiplication of the neural network weights and the output of the Inception-ResNet-v2 to obtain a feature vector; The feature vector is extracted to construct a triplet loss function, and the triplet loss function is subjected to target perturbation, and then the FaceNet framework structure is trained.

2. The hybrid perturbation differential privacy protection method for face recognition as claimed in claim 1, characterized in that: The model for preprocessing the facial data is an MTCNN model, and the MTCNN model includes P-Net, R-Net and O-Net.

3. The hybrid perturbation differential privacy protection method for face recognition as claimed in claim 2, characterized in that: The specific method of preprocessing the face data to obtain the minimum detection frame is: Constructing an image pyramid based on the face data; Input each picture in the image pyramid into the P-Net for coordinate detection to obtain a preliminary candidate face detection frame; Inputting the preliminary candidate face detection frame into the R-Net for screening to obtain a screened candidate face detection frame; The screened candidate face detection frame is scaled by the O-Net to obtain a minimum detection frame.