A privacy protection method for face images driven by key point differential privacy

Abstract

The present invention discloses a facial image privacy protection method driven by key point differential privacy, which firstly performs data preprocessing to construct a key point identification anonymous space; then constructs an anonymous face generation adversarial network structure, determines the anonymous face generation adversarial network optimization objective function; obtains an improved differential privacy algorithm applied to anonymous face key point identification; finally, trains the anonymous face generation adversarial network through the preprocessed data set, and outputs the final result. The method modifies the facial key point structure of the face in the image to realize the anonymity of the face identity, achieves better data availability and visual effects, has higher quality in the generated image, can maintain the original non-identity attributes, and does not need to use any attribute labels.

Description

Technical Field

[0001] The present invention belongs to the field of image privacy protection, and specifically relates to a face image privacy protection method driven by key point differential privacy. With the rapid development of the big data era, face data is one of the most easily leaked personal data, and its privacy security is seriously threatened. The present invention proposes a face image privacy protection technology driven by face key points, which anonymizes face key point identifiers by using differential privacy theory and changes its facial geometric structure to achieve anonymization of face images to protect identity information. Background Art

[0002] With the explosive growth of diversified Internet services and social networking platforms, a large number of users interact with each other on social networking platforms, resulting in a huge amount of shared data, mainly visual media, in social networks. The explosive growth of these facial images and their widespread dissemination and accessibility in social media have made the security and privacy of personal identity information an unprecedented challenge. In order to address privacy issues, restrictive laws and regulations such as the General Data Protection Regulation (GDPR) have come into effect. GDPR requires researchers to obtain regular consent from individuals to use their personal data to ensure data privacy, but this also makes it extremely challenging to create high-quality datasets including faces. The lack of data will increase the difficulty for researchers to design intelligent algorithms. The convenience brought by excellent image data to computer vision technology is also attractive. Therefore, it is very important to use facial images while protecting people's identities.

[0003] In fact, most intelligent algorithms such as face detection, pedestrian tracking, and statistical data analysis focus only on general information rather than capturing identity-sensitive features. Given a face image, face de-identification, also known as face anonymization, refers to generating another image with similar appearance and the same background while hiding the real identity. Despite extensive efforts, existing face de-identification techniques are either insufficient in photo-realism or fail to balance privacy and practicality well. Therefore, in order to ensure the security and availability of published data, reliable anonymization methods are essential.

[0004] All the problems and dilemmas mentioned above can be summarized into one question: given a face image, how can we create another image with similar appearance and the same background, while the real identity is hidden, and the face detector can still work? Traditional anonymization techniques are mainly based on blurring and always significantly change the original face image. Therefore, face de-identification technology mainly considers two aspects: identity information removal and retaining non-identity information of the original data. Traditional face de-identification methods are mainly based on image distortion methods, such as black frame coverage, blurring, pixelation, or adding random noise, which distort the face area in the image or cover the key sensitive areas of the face to remove or hide identity information. The advantage of this method is that it does not require complex algorithm technology in implementation, the operation speed is fast, and the processed results are sufficiently anonymous for human eye recognition. However, Oh et al. analyzed the performance of face identity recognizers under masking or blurring, showing that these common blurring methods are especially unsafe for deep recognizers. At the same time, the result data after black frame coverage, noise, etc. is severely damaged, and the data availability is lost.

[0005] In recent years, with the continuous development of deep learning technology, Goodfellow et al. proposed the Generative Adversarial Network (GAN) in 2014. GAN represents an inspiring framework that generates clear and realistic natural face image samples through an objective function. Therefore, it has become popular in recent face de-identification techniques, which can be divided into three categories. Methods based on attribute manipulation. Face attributes are the key to face recognition, and some methods achieve de-identification by processing attributes. Li et al. proposed the Privacy Preserving Attribute Selection (PPAS) algorithm to select and update face attributes so that the distribution of any attribute is close to its true distribution and provides measurable privacy for face anonymity under privacy guarantee. Methods based on conditional inpainting. Since the face is one of the most powerful clues to infer a person's identity, many studies have concealed sensitive identity information by conditionally inpainting the face area. Sun et al. generated a realistic head inpainting based on 68 facial key points. Methods based on conditional ID exchange. Replacing the identity in a face image with another person is a direct but effective face anonymization method. Meden et al. proposed a de-identification process in which each generated face is a combination of entities. CIAGAN utilizes facial keypoints and one identity-aware hot vector to remove the identity of a person while still retaining the necessary features to allow face and body detectors to work.

[0006] A key point in facial identity privacy protection is the effectiveness of face anonymity and the balance between it and data reusability. However, most current methods focus on the effectiveness of anonymity and ignore the availability of non-identity attribute information.

[0007] In summary, the field of facial identity privacy protection is a topic worthy of in-depth research. This patent intends to explore several key points in this field and solve the problems existing in current methods. Specifically, existing methods mainly face the following two challenges:

[0008] (1) For anonymized facial images, how to retain non-identity information is very important to improve the reusability of data. The lack of data reusability will increase the difficulty for researchers to design intelligent algorithms, so it is a great challenge to achieve facial anonymity while retaining non-identity information.

[0009] (2) For anonymized face images, how to prevent attacks on other people's privacy information is very important to improve the security of the anonymization method. Most of the current deep learning methods generate images that are not realistic enough or use other people's IDs to merge with their own non-identity information to achieve anonymity, but there is suspicion of infringing on other people's privacy. Therefore, how to comprehensively evaluate the effectiveness of face anonymization methods is also a difficult problem in visual privacy protection. Summary of the invention

[0010] The present invention proposes a face image privacy protection method driven by key point differential privacy. The present invention mainly includes three points: 1. Proposing a new generative framework for face image anonymity, which can flexibly realize reusable image data privacy protection through parameter adjustment; 2. Proposing differential privacy protection of face key point identification, and using it as input conditions for the generation of de-identified face images; 3. By introducing a style encoder, the style style of the original image is transferred to the anonymous image, improving the retention of non-identity attributes and the quality of the generated image without the need for additional attribute annotation.

[0011] The technical solution adopted by the present invention to solve the technical problem comprises the following steps:

[0012] Step 1: Data preprocessing;

[0013] Step 2: Construct an anonymous face generation adversarial network structure;

[0014] Step 3: Determine the optimization objective function of the anonymous face generation adversarial network;

[0015] Step 4: Obtain an improved differential privacy algorithm for anonymous facial key point identification;

[0016] Step 5: Use the dataset preprocessed in step 1 to train the anonymous face generative adversarial network and output the final result.

[0017] Step 1, the specific steps are as follows:

[0018] 1-1. Data preparation.

[0019] Select a face dataset (such as the CelebA-HQ dataset), use a face key point detector (such as facealignment, dlib) to detect the face dataset image, and obtain the face key points.

[0020] 1-2. Construct an anonymous space for key point identification.

[0021] The key points are analyzed to obtain 16 identity-related components that may contain identity information. These components constitute a 16-dimensional key point identification vector, including eyebrow length, eyebrow curvature, eyebrow distance, eyebrow-eye distance, eye distance, eye width, eye size, nose bridge length, nose tip height, nose width, mouth-nose distance, mouth-nose distance, upper and lower lip thickness, face width, and chin curvature.

[0022] Step 2: Construct an anonymous face generation adversarial network structure. The specific steps are as follows:

[0023] The anonymous face generation adversarial network includes a generator, a true and false discriminator and a style discriminator.

[0024] 2-1. Construct generator G.

[0025] The generator is composed of a style encoder, a fusion device and a mask generator. The style encoder is composed of 6 residual blocks, in which the last layer of results is summed and a 512-dimensional vector is output as the style feature vector. The fusion device is composed of 5 down-sampling residual blocks and 8 up-sampling residual blocks, in which each up-sampling block adds an AdaIN block and uses Tanh as the activation function in the last layer. The fusion device decodes the key point identifier and embeds the style feature vector into the AdaIN block with affine parameters. The mask generator is composed of 3 down-sampling residual blocks and 5 up-sampling residual blocks, and finally generates a single-channel facial semantic mask. The mask can better integrate the generated face image into the background.

[0026] 2-2. Construct true and false discriminator D.

[0027] Compared with the style encoder, the discriminator network has an additional down-sampling residual block and a fully connected network at the end, and finally outputs the true and false score of the input image. The true and false discriminator is used to determine whether the input image is a real face image and whether it matches the key point identification of the input;

[0028] 2-3. Use the pre-trained VGG19 model and VGGface model as style discriminators.

[0029] Step 3 determines the optimization objective function of the anonymous face generation adversarial network, as follows:

[0030] The face generation adversarial network optimization objective function is as follows:

[0031]

[0032] This includes adversarial loss functions Style loss function for face images Feature matching loss function And face semantic loss function

[0033] 3-1. Adversarial loss function.

[0034] The idea of ​​combining pix2pix and StyleGAN is used to control the generation of different contents of the image to ensure the controllability of privacy attributes and non-privacy attributes. The specific operation is to extract the original image x from the style encoder r The style feature vector and key point identification ξ are input into the fuser as conditions. The style feature vector controls the AdaIN block in the fuser network to generate the image x f ; Then identify the key points and generate the corresponding image x f For a set of input discriminators to discriminate, the adversarial loss function between the generator and the true and false discriminator is expressed as:

[0035] x f =G(ξ,x r )

[0036]

[0037] 3-2. Style loss function of face image.

[0038] Use the pre-trained VGG19 model and VGGface model as the style discriminator to calculate the generated image x f The specific process is to use the output of the style discriminator feature layer to obtain the feature maps of the generated image and the original image respectively, and calculate the L between the two feature maps. 1 The distance difference makes the style of the original image and the generated image consistent. It can be expressed in mathematical form as:

[0039]

[0040] Where T is the selected feature layer set, VGG i It is the intermediate feature extracted by the i-th layer of the style discriminator.

[0041] 3-3. Feature matching loss function.

[0042] In order to improve the stability of network model training and the quality of generated images, a feature matching loss function is introduced. Specifically, similar to the style loss function, multiple intermediate layer features are extracted from the true and false discriminator for the original image and the generated image, and multi-scale feature matching is performed. Let the feature matching loss function be:

[0043]

[0044] Where T is the selected feature layer set, D i It is the intermediate feature extracted by the i-th layer of the true and false discriminator.

[0045] 3-4. Face semantic loss function.

[0046] In order to better preserve the background and reduce the difficulty of network optimization, an additional face semantic generator is used to generate a mask M containing only face semantics. The loss function can be expressed mathematically as:

[0047]

[0048] Among them B ξ Represents the face area marked by facial key points.

[0049] Step 4 obtains the improved differential privacy algorithm for anonymous facial key point identification, as follows:

[0050] Differential privacy theory algorithm: For given adjacent data sets U and U', the two differ from each other by at most one record, that is, |DΔD'|≤1. Given a privacy algorithm M, S is the value range of M. If any output result S of algorithm M on data sets U and U' satisfies the following inequality, then M satisfies ε-differential privacy.

[0051] Pr[M(U)∈S]≤e ε Pr[M(U')∈S]

[0052] This parameter ε is often called the privacy budget, which is a bound on the probability ratio of seeing the same output on adjacent data sets. The smaller the value of ε, the stronger the privacy protection.

[0053] Use the Laplace mechanism to add random perturbations to key point identifiers to achieve differential privacy. Let R ξ (X) is the key point identification set, f(X) is the ξ The mapping function of , X and X' are any two images of the dataset U, Δf is the global privacy sensitivity, which is used to calibrate the noise amount of the function f(X), and is the maximum L1 norm of the key point identification of any X and X', defined as:

[0054] f(X)=R ξ (X)

[0055]

[0056] The M(X) function is used to perturb the identity vector by adding a Laplace noise mechanism:

[0057]

[0058] According to the definition of differential privacy, M(X) satisfies ε-differential privacy:

[0059]

[0060] Step 5:

[0061] 5-1. Training process:

[0062] The face dataset and key point identification preprocessed in step 1 are input into the constructed anonymous face generative adversarial network to obtain the generated image. The true and false discriminator is used to discriminate the authenticity of the image and calculate the corresponding loss. The style discriminator is used to discriminate the style consistency between the generated image and the original image and calculate the corresponding loss. Finally, the gradient is back-propagated to update the network parameters.

[0063] 5-2. Anonymous face image generation:

[0064] According to step 4, the key point is identified R through the differential privacy function M(X) ξ Perform disturbance and then mark the key points after disturbance and the original image as the input of the generator, and output the final anonymous face image with differential privacy protection.

[0065] The beneficial effects of the present invention are:

[0066] This method modifies the facial key point structure of the face in the image to achieve face anonymity. Compared with traditional mosaic, occlusion and other methods, this method achieves better data availability and visual effects. Compared with deep learning methods, it has higher quality in generating images, can maintain the original non-identity attributes, and does not require the use of any attribute labels. The experimental results clearly confirm the efficiency and practicality of the proposed method. In short, the proposed method is more efficient and beautiful for the privacy protection of human images. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of the overall network architecture of an embodiment of the present invention;

[0068] Figure 2 is a flow chart of an anonymization process according to an embodiment of the present invention;

[0069] Figure 3The anonymous space is identified by 16 key points of the face in the embodiment of the present invention;

[0070] Figure 4 is a curve diagram of experimental indicators under different privacy budgets e according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0072] Facial key point identification privacy protection technology based on generative adversarial network, for the specific overall architecture flow chart, please refer to Figure 1 As shown, the anonymization flow chart can be found in Figure 2 As shown, the key points of the face are defined in Figure 3 As shown:

[0073] The technical solution adopted by the present invention to solve the technical problem comprises the following steps:

[0074] Step 1: Data preprocessing;

[0075] 1-1. Data preparation.

[0076] Select the face datasets CelebA-HQ and VGGFACE2, use the face key point detector (such as facealignment, dlib) to detect the face dataset images, and obtain the face key points.

[0077] 1-2. Construct an anonymous space for key point identification.

[0078] The key points are analyzed to obtain 16 identity-related components that may contain identity information. These components constitute a 16-dimensional key point identification vector, including eyebrow length, eyebrow curvature, eyebrow distance, eyebrow-eye distance, eye distance, eye width, eye size, nose bridge length, nose tip height, nose width, mouth-nose distance, mouth-nose distance, upper and lower lip thickness, face width, and chin curvature.

[0079] Step 2: Construct an anonymous face generation adversarial network structure. The specific steps are as follows:

[0080] The anonymous face generation adversarial network includes a generator, a true and false discriminator and a style discriminator.

[0081] 2-1. Construct generator G.

[0082] The generator is composed of a style encoder, a fusion device and a mask generator. The style encoder is composed of 6 residual blocks, in which the last layer of results is summed and a 512-dimensional vector is output as the style feature vector. The fusion device is composed of 5 down-sampling residual blocks and 8 up-sampling residual blocks, in which each up-sampling block adds an AdaIN block and uses Tanh as the activation function in the last layer. The fusion device decodes the key point identifier and embeds the style feature vector into the AdaIN block with affine parameters. The mask generator is composed of 3 down-sampling residual blocks and 5 up-sampling residual blocks, and finally generates a single-channel facial semantic mask. The mask can better integrate the generated face image into the background.

[0083] 2-2. Construct true and false discriminator D.

[0084] Compared with the style encoder, the discriminator network has an additional down-sampling residual block and a fully connected network at the end, and finally outputs the true and false score of the input image. The true and false discriminator is used to determine whether the input image is a real face image and whether it matches the key point identification of the input;

[0085] 2-3. Use the pre-trained VGG19 model and VGGface model as style discriminators.

[0086] Step 3 determines the face generation adversarial network optimization objective function, as follows:

[0087] The face generation adversarial network optimization objective function is as follows:

[0088]

[0089] This includes adversarial loss functions Style loss function for face images Feature matching loss function And face semantic loss function

[0090] 3-1. Adversarial loss function.

[0091] The idea of ​​combining pix2pix and StyleGAN is used to control the generation of different contents of the image to ensure the controllability of privacy attributes and non-privacy attributes. The specific operation is to extract the original image x from the style encoder r The style feature vector and key point identification ξ are input into the fuser as conditions. The style feature vector controls the AdaIN block in the fuser network to generate the image x f ; Then identify the key points and generate the corresponding image x f For a set of input discriminators to discriminate, the adversarial loss function between the generator and the true and false discriminator is expressed as:

[0092] x f =G(ξ,x r )

[0093]

[0094] 3-2. Style loss function of face image.

[0095] Use the pre-trained VGG19 model and VGGface model as the style discriminator to calculate the generated image x f The specific process is to use the output of the style discriminator feature layer to obtain the feature maps of the generated image and the original image respectively, and calculate the L between the two feature maps. 1 The distance difference makes the style of the original image and the generated image consistent. It can be expressed in mathematical form as:

[0096]

[0097] Where T is the selected feature layer set, VGG i It is the intermediate feature extracted by the i-th layer of the style discriminator.

[0098] 3-3. Feature matching loss function.

[0099] In order to improve the stability of network model training and the quality of generated images, a feature matching loss function is introduced. Specifically, similar to the style loss function, multiple intermediate layer features are extracted from the true and false discriminator for the original image and the generated image, and multi-scale feature matching is performed. Let the feature matching loss function be:

[0100]

[0101] Where T is the selected feature layer set, D i It is the intermediate feature extracted by the i-th layer of the true and false discriminator.

[0102] 3-4. Face semantic loss function.

[0103] In order to better preserve the background and reduce the difficulty of network optimization, an additional face semantic generator is used to generate a mask M containing only face semantics. The loss function can be expressed mathematically as:

[0104]

[0105] Among them B ξ Represents the face area marked by facial key points.

[0106] Step 4: Obtain the improved differential privacy algorithm for anonymous facial key point identification, as follows:

[0107] Differential privacy algorithm:

[0108] For given adjacent data sets U and U', the two differ by at most one record, that is, |DΔD'|≤1. Given a privacy algorithm M, S is the value range of M, if any output result S of algorithm M on data sets U and U' satisfies the following inequality, then M satisfies ε-differential privacy.

[0109] Pr[M(U)∈S]≤e ε Pr[M(U')∈S]

[0110] This parameter ε is often called the privacy budget, which is a bound on the probability ratio of seeing the same output on adjacent data sets. The smaller the value of ε, the stronger the privacy protection.

[0111] Use the Laplace mechanism to add random perturbations to key point identifiers to achieve differential privacy. Let R ξ (X) is the key point identification set, f(X) is the ξ The mapping function of , X and X' are any two images of the dataset U, Δf is the global privacy sensitivity, which is used to calibrate the noise amount of the function f(X), and is the maximum L1 norm of the key point identification of any X and X', defined as:

[0112] f(X)=R ξ (X)

[0113]

[0114] The M(X) function is used to perturb the identity vector by adding a Laplace noise mechanism:

[0115]

[0116] According to the definition of differential privacy, M(X) satisfies ε-differential privacy:

[0117]

[0118] Step 5: Use the dataset preprocessed in step 1 to train the anonymous face generation adversarial network and output the final result, as follows:

[0119] 5-1. Training process:

[0120] Input the preprocessed data set and key point identifiers in step 1 into the constructed anonymous face generative adversarial network to obtain the generated image. Use the true and false discriminator to discriminate the authenticity of the image and calculate the corresponding loss. Use the style discriminator to discriminate the style consistency between the generated image and the original image and calculate the corresponding loss. Finally, backpropagate the gradient to update the network parameters.

[0121] 5-2. Anonymous face image generation:

[0122] According to step 4, the key point is identified R through the differential privacy function M(X) ξ Perform disturbance and then mark the key points after disturbance and the original image as the input of the generator, and output the final anonymous face image with differential privacy protection.

[0123] 5-3. In order to comprehensively evaluate the effectiveness of the proposed method, it is compared with current excellent methods (such as AnonymousNet, CIAGAN, DP_image, etc.), and the identity re-identification rate (ReID), identity exchange rate (IDS), image quality (FID, SSIM, PSNR) and face attribute classification accuracy are calculated. The anonymity effect of the method is comprehensively analyzed, and anonymous images under different privacy budgets e are tested to analyze the controllability of the re-identification rate of the method under different privacy budgets.

[0124] Experimental results:

[0125] 1. The re-recognition rates of this method and the blurring, mosaic, Gaussian noise, CIAGAN, and AnonymousNet methods were tested respectively, and the image quality was calculated using FID, SSIM, and PSNR. The specific data results are detailed in Table 1.

[0126] Table 1 Comparative experimental results of this method and other methods

[0127]

[0128]

[0129] 2. The controllability of the re-identification rate of this method under different privacy budgets was tested, and the quality of the image was calculated using FID, SSIM, and PSNR. The specific data results are detailed in Table 2.

[0130] Table 2 Comparative experimental results of this method under different privacy budgets e

[0131] method TOP1_ReID TOP10_ReID FID SSIM PSNR e＝0.1 0.0836 0.1873 5.3873 0.8189 23.3166 e＝0.5 0.1255 0.2581 4.4795 0.8283 23.8580 e＝1 0.1935 0.3421 3.9719 0.8372 24.4189 e＝2 0.3155 0.4934 3.5274 0.8505 25.2687 e＝4 0.5244 0.6933 3.3232 0.8656 26.3479 e＝1024 0.9190 0.9626 3.2656 0.8944 28.7426

[0132] Figure 4 is a curve diagram of experimental indicators under different privacy budgets e according to an embodiment of the present invention.

[0133] 3. The IDS of this method and the blurring, mosaic, Gaussian noise, CIAGAN, and AnonymousNet methods with different privacy budgets were tested respectively. The specific data results are detailed in Table 3.

[0134] Table 3 Experimental results of IDS with different privacy budgets e of this method

[0135] method IDS(score>0.25) score>0.3 score>0.35 score>0.4 e＝0.1 0.9980 0.7463 0.2445 0.0572 e＝1 0.9988 0.8417 0.3796 0.0935 e＝1024 0.9975 0.9728 0.8123 0.3605 mosaic=15 0.9813 0.8797 0.7375 0.5609 noise=6 0.8899 0.4299 0.1810 0.0980 Blur = 20 0.9730 0.6047 0.1745 0.0597 CIAGAN 0.9974 0.9343 0.4921 0.09094 AnonymousNet 0.9831 0.6275 0.15673 0.02535

[0136] 4. Attribute classification accuracy under different methods As shown in Table 4, the attribute classification accuracy includes five attributes, namely face detection rate, gender, whether smiling, whether wearing heavy makeup, and whether young.

[0137] Table 4 Experimental results of attribute classification accuracy under different methods

[0138] method detect male smiling heavy_makeup young e＝0.1 1.00 0.9656 0.8251 0.8786 0.8780 mosaic=15 0.12 0.4115 0.5404 0.5512 0.7360 noise=6 0.98 0.4304 0.5319 0.5500 0.6318 Blur = 20 0.99 0.7837 0.6031 0.6213 0.8041 CIAGAN 0.99 0.8622 0.792 0.748 0.806 AnonymousNet 0.99 0.6263 0.5476 0.589 0.6581 .

Claims

1. A face image privacy protection method driven by key point differential privacy, characterized in that: Here are the steps: Step 1: Data preprocessing; Step 2: Construct an anonymous face generation adversarial network structure; Step 3: Determine the optimization objective function of the anonymous face generation adversarial network; Step 4: Obtain an improved differential privacy algorithm for anonymous facial key point identification; Step 5: Use the dataset preprocessed in step 1 to train the anonymous face generative adversarial network and output the final result; Step 2: Construct an anonymous face generation adversarial network structure. The specific steps are as follows: The anonymous face generation adversarial network includes a generator, a true and false discriminator and a style discriminator; 2-1. Construct generator G; The generator is composed of a style encoder, a fusion device and a mask generator; the style encoder is composed of 6 residual blocks, wherein the last layer of results is summed and a 512-dimensional vector is output as the style feature vector; the fusion device is composed of 5 down-sampling residual blocks and 8 up-sampling residual blocks, wherein each up-sampling block is added with an AdaIN block and Tanh is used as an activation function in the last layer; the fusion device decodes the key point identifier and embeds the style feature vector into the AdaIN block with affine parameters; the mask generator is composed of 3 down-sampling residual blocks and 5 up-sampling residual blocks, and finally generates a single-channel facial semantic mask; the mask can better fuse the generated face image into the background; 2-2. Construct true and false discriminator D; Compared with the style encoder, the discriminator network has an additional down-sampling residual block and a fully connected network at the end, and finally outputs the true and false score of the input image; the true and false discriminator is used to determine whether the input image is a real face image and whether it matches the key point identification of the input; 2-3. Use the pre-trained VGG19 model and VGGface model as style discriminators; Step 3 determines the optimization objective function of the anonymous face generation adversarial network, as follows: The face generation adversarial network optimization objective function is as follows: This includes adversarial loss functions Style loss function for face images Feature matching loss function And face semantic loss function 3-1. Adversarial loss function; The idea of ​​combining pix2pix and StyleGAN is used to control the generation of different contents of the image to ensure the controllability of privacy attributes and non-privacy attributes; the specific operation is to extract the original image x from the style encoder r The style feature vector and key point identification ξ are input into the fuser as conditions. The style feature vector controls the AdaIN block in the fuser network to generate the image x f ; Then the key points are identified and the corresponding generated image x f For a set of input discriminators to discriminate, the adversarial loss function between the generator and the true and false discriminator is expressed as: x f =G(ξ,x r ) 3-2. Face image style loss function; Use the pre-trained VGG19 model and VGGface model as the style discriminator to calculate the generated image x f The specific process is to use the output of the style discriminator feature layer to obtain the feature maps of the generated image and the original image respectively, and calculate the L1 distance difference between the two feature maps to make the style of the original image consistent with that of the generated image; mathematically, it can be expressed as: Where T is the selected feature layer set, VGG i It is the intermediate feature extracted by the i-th layer of the style discriminator; 3-3. Feature matching loss function; In order to improve the stability of network model training and the quality of generated images, a feature matching loss function is introduced. Specifically, similar to the style loss function, multiple intermediate layer features are extracted from the true and false discriminator for the original image and the generated image, and multi-scale feature matching is performed. Let the feature matching loss function be: Where T is the selected feature layer set, D i is the intermediate feature extracted by the i-th layer of the true and false discriminator; 3-4. Face semantic loss function; In order to better preserve the background and reduce the difficulty of network optimization, an additional face semantic generator is used to generate a mask M containing only face semantics; the loss function can be expressed mathematically as: Among them B ξ Represents the face area marked by face key points; Step 4 obtains the improved differential privacy algorithm for anonymous facial key point identification, as follows: Differential privacy theory algorithm: For given adjacent data sets U and U', the two differ by at most one record, that is, |DΔD'|≤1; given a privacy algorithm M, S is the value range of M, if the algorithm M's arbitrary output result S on the data sets U and U' satisfies the following inequality, then M satisfies ε-differential privacy; Pr[M(U)∈S]≤eεPr[M(U')∈S] This parameter ε is usually called the privacy budget, which is the bound on the probability ratio of seeing the same output on adjacent data sets; the smaller the value of ε, the stronger the privacy protection; Use the Laplace mechanism to add random perturbations to key point identifiers to achieve differential privacy; let R ξ (X) is the key point identification set, f(X) is the ξ The mapping function of X and X' is any two images of the dataset U, Δf is the global privacy sensitivity, which is used to calibrate the noise amount of the function f(X), and is the maximum L1 norm of the key point identification of any X and X', defined as: f(X)=R ξ (X) The M(X) function is used to perturb the identity vector by adding a Laplace noise mechanism: According to the definition of differential privacy, M(X) satisfies ε-differential privacy:

2. According to claim 1, a key point differential privacy driven face image privacy protection method is characterized in that: Step 1 The specific steps are as follows: 1-1. Data preparation; Select a face data set, use a face key point detector to detect the face data set image, and obtain the face key points; 1-2. Construct an anonymous space for key point identification; The key points are analyzed to obtain 16 identity-related components that may contain identity information. These components constitute a 16-dimensional key point identification vector, including eyebrow length, eyebrow curvature, eyebrow distance, eyebrow-eye distance, eye distance, eye width, eye size, nose bridge length, nose tip height, nose width, mouth-nose distance, mouth-nose distance, upper and lower lip thickness, face width, and chin curvature.

3. The method for protecting face image privacy driven by key point differential privacy according to claim 2, characterized in that: Step 5: 5-1. Training process: Input the face dataset and key point identifiers preprocessed in step 1 into the constructed anonymous face generative adversarial network to obtain the generated image, use the true and false discriminator to discriminate the authenticity of the image and calculate the corresponding loss, use the style discriminator to discriminate the style consistency between the generated image and the original image and calculate the corresponding loss, and finally back-propagate the gradient to update the network parameters; 5-2. Anonymous face image generation: According to step 4, the key point is identified R through the differential privacy function M(X) ξ Perform disturbance and then mark the key points after disturbance and the original image as the input of the generator, and output the final anonymous face image with differential privacy protection.

Images