Face image privacy protection method based on attention and generative adversarial network

Abstract

The application discloses a face image privacy protection method based on attention and a generative adversarial network, comprising the following steps: step one, constructing a self-attention perception generative network; step two, constructing a spatial attention guide discriminant network; step three, determining an attention-generative adversarial network optimization objective function; and step four, training the attention-generative adversarial network and outputting a final result. The application can improve the utility of the de-identified face image by retaining more abundant details of the de-identified face image, generate a high-fidelity de-identified face image, and ensure that the privacy attribute cannot be detected and the user's personal information privacy cannot be leaked.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data security and privacy protection, and particularly relates to a face image privacy protection method based on attention and generative adversarial network. BACKGROUND

[0002] The rapid development of Internet information technology greatly facilitates people to use smart phones and other portable smart devices to upload photos to personalized Internet platforms and share their social experiences. Although these smart devices provide convenience for users, the widespread dissemination and easy accessibility of face images in devices pose unprecedented challenges to the security and privacy of user identity information. To solve the above problems, face de-identification technology emerges as the times require, which uses image generation or processing operations such as pixelization, masking, blurring or anonymization to provide unique identification of users without revealing personal privacy, and the processed face images can be used for training and application of face recognition models. However, the above methods have problems such as poor clarity, large difference from real face images, and loss of much user feature information.

[0003] In recent years, with the continuous development of deep learning technology, Goodfellow et al. proposed generative adversarial network (GAN) and successfully applied it to face image synthesis, attribute generation, de-identification and other fields. The face image de-identification method based on GAN directly models the feature map conversion between different attribute distributions of face images, and optimizes the mapping function to convert the verified face image into the target attribute group with high visual fidelity, generating high-definition images to replace the original images for publication, which has become one of the most potential research methods for unsupervised learning on complex distributions.

[0004] However, through research and analysis of the prior art, it is found that the existing technology still has the following problems to be solved: 1) Since the method based on GAN usually only focuses on the conversion between image contents, other key semantic condition information (facial attributes) of the training image is ignored, so it is impossible to guide the generation of target face images with high visual fidelity and correct attributes. 2) In practice, due to the high cost of collecting multiple face images of the same object, the face sample imbalance problem of the model trained by the method based on GAN still exists, which leads to the collapse of the model; 3) Without the guidance of high-level condition information, these GAN-based de-identification methods are easy to generate many images with incorrect facial attributes. SUMMARY

[0005] Therefore, the present application provides a face image privacy protection method based on attention and generative adversarial network, which can improve the utility of de-identified face images by retaining more detailed information, generate high-fidelity de-identified face images, and ensure that the privacy attribute cannot be detected, so as to guarantee that the user's personal information privacy will not be leaked.

[0006] The technical solutions of the present application are as follows:

[0007] A face image privacy protection method based on attention and generative adversarial network, comprising the following steps:

[0008] Step 1, construct a self-attention perception generation network;

[0009] Step 2, construct a spatial attention guided discriminant network;

[0010] Step 3, determine the attention-generative adversarial network optimization objective function;

[0011] Step 4, train the attention-generative adversarial network, and output the final result.

[0012] Further, step 1 is specifically: in order to completely retain the detailed information of the generated face image, on the basis of the extracted multi-scale face features, the correlation between each feature layer of the multi-scale face features is calculated, the weighting coefficient of each feature layer fusion is calculated, and the label attribute information of face gender and age is embedded on the basis of multi-scale fusion features combined with the self-attention mechanism model, to generate a face image with global and detailed features.

[0013] Further, step 2 is specifically: through the average pooling and maximum pooling operation of the feature layer, the multi-scale spatial features are extracted, and the spatial feature map is generated by using the feature aggregation operation, which is used as the basis for guiding the subsequent face discrimination and image generation; On this basis, a face attribute classification and matching auxiliary model is introduced into the discriminator, and a visually realistic face image with the required attributes is generated for privacy protection.

[0014] Further, the attention-generative adversarial network optimization objective function is specifically:

[0015]

[0016] wherein, is a multi-scale reconstruction pixel loss function, represents the expectation of the function , is the original face image feature with attribute att, respectively represents the distribution of the generated face image and the expected attribute, S is the distribution of the expected attribute.spa V represents the discriminator function. id Z represents the attribute required to obfuscate a face image. id This represents the identity embedding function.

[0017] Furthermore, the specific process of step four is as follows: through the attention-guided generative network, the face attribute matching function, face image generation function and attribute classification function are combined during the training process. Through training iteration and optimization, a face image with high realism and preservation of global and local visual and contour characteristics of the face is generated.

[0018] Beneficial effects:

[0019] 1) This invention improves the utility of de-identified facial images by preserving richer details, thereby generating high-fidelity de-identified facial images for use in the training and application of facial recognition models, and ensuring that their privacy attributes are not detected, thus guaranteeing that users' personal information privacy is not leaked.

[0020] 2) This invention introduces a dual attention mechanism in the adversarial learning process between the generator and the discriminator, and proposes a novel attention-generative adversarial network framework. Through more robust adversarial learning, it effectively improves the performance of the generative model. The generated de-identified images meet the requirements of protecting user privacy while having high usability and quality, and can be effectively applied to the training and use of face recognition models.

[0021] 3) This invention designs a cycle consistency loss function and a multi-scale reconstruction pixel loss function to train a face image privacy protection model in a collaborative manner. This enables the network to better learn the continuous variation patterns between pixels in the image, ensuring the consistency of global and local quality between the generated image and the real image, improving the texture details of the generated image, and enhancing the realism of the synthetic face image with privacy protection. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention.

[0023] Figure 2 This is a flowchart illustrating a specific implementation of the self-attention network of the present invention.

[0024] Figure 3 This is a flowchart illustrating a specific implementation of the spatial attention network of the present invention. Detailed Implementation

[0025] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] This invention provides a face image privacy protection method based on attention and generative adversarial networks, such as... Figure 1 As shown, it includes the following steps:

[0027] Step 1: Constructing the self-attention-aware generative network, the specific steps are as follows:

[0028] 1) Attribute extraction and embedding: Select a face image dataset (such as the CelebA dataset, MORPH dataset, or RaFD dataset), use a pre-trained GoogleLeNet as an identity encoder, and input each face image into the encoder for identity attribute feature set extraction X = {X id , X att}, where X id , X att represent identity attributes and background attributes, respectively. The implementation of identity attribute embedding is defined as Γ id : Z(X)→ Z id (X id ), and is integrated with the final Fully Connected layer (FC) for facial attribute feature extraction. To preserve the detailed information of images at different levels, a Multi-scale Feature Embedding Strategy (MFES) is designed based on the U-Net-like structure to generate multi-scale feature maps for feature integration. where represents the nth layer feature map from the U-Net-like decoder, k ∈ {1, …, n} represents the layer number of the feature map. Identity embedding Z id (X id ) and attribute embedding Z mfes (X att ) are two three-dimensional tensors with dimensions C k × H k × W k , where C k , H k × W k represent the number of channels and spatial dimensions, respectively.

[0029] 2) Attribute fusion based on self-attention: As shown in Figure 2 , adaptive attentional denormalization (AAD) is used to process identity embedding and attribute embedding respectively:

[0030]

[0031]

[0032]

[0033] where, denote attribute and identity activation, respectively, denote modulation parameters of attribute embedding and identity embedding convolution, respectively, are the mean and standard deviation of the intra-batch channel activation.

[0034] As Figure 2 shown, the outputs of attribute embedding and identity embedding are fed back to two different 1 × 1 convolution layers, aiming to convert them into two one-dimensional vectors of size C Then, the matrix is reshaped to the size of to generate the matrix and the transposed matrix and get the attention feature map where τ is the Softmax function, and the original feature set is converted into a new feature map Z a . Finally, the output of the self-attention mechanism in the generator G is denoted as G self , i is the number of channel layers of the feature map, and ρ is the scale parameter for training the generator.

[0035] Step 2: Construct a spatial attention guided discriminant network, the specific steps are as follows:

[0036] 1) Spatial attention network: perform spatial attention network according to the spatial relationship of local regions in the feature map to focus on regions with more information, as Figure 3 shown. For the feature map obtained by the self-attention network in the generator, dimension compression is performed on the refined multi-scale (or multi-channel) feature Z self through the average pooling operation AvePool(·) and the maximum pooling operation MaxPool(·). Then, the two groups of feature sequences are connected and convolved through the standard convolution layer to obtain the spatial attention feature map where, denote the average pooling feature and the maximum pooling feature, respectively, f denotes the convolution operation, and the size of the convolution kernel is set to 3 × 3. The output of the spatial attention discriminator is used to evaluate the influence of the i-th position on the j-th position on the spatial attention feature map, and ω is the scale parameter for training the discriminator.

[0037] 2) Classification and Matching Auxiliary Network: To generate visually realistic face images with desired attributes for privacy protection, an auxiliary face attribute classifier and matcher are introduced into the discriminator. During training, the auxiliary classifier is embedded in the discriminator to classify multiple face attributes, and then provides attribute-controlled constraints to the generator for synthesizing privacy-preserving face images while maintaining their utility. Simultaneously, a face matcher is introduced to reduce the structural differences between the original and synthesized face images, making them have a more natural and similar statistical distribution to the original face images. In the implementation of the proposed model, state-of-the-art pre-trained GoogLeNet and VGG-19 models are used to ensure the effectiveness of the classification and matching modules, respectively.

[0038] Step 3: Determine the objective function for the attention-generative adversarial network optimization, as shown in Equation 3. The specific steps are as follows:

[0039] For a given face image X, its truth attribute label vector is: Let C represent the attributes required to obfuscate a face image, where C is the number of all attributes. The objective of this invention is to generate a new face image z. self =G self (X,v id The generation process is supervised by a classifier. The objective functions for training the generator and discriminator are respectively... This includes the cycle consistency loss function. Multi-scale reconstruction pixel loss function Reconstruct the loss function Attribute classification constraint loss function Matching loss function η, γ, ζ, and ε represent the hyperparameters that adjust the weights of the corresponding objective functions.

[0040] The generative adversarial loss function is defined as follows:

[0041]

[0042]

[0043] in, D represents the distribution of the true image and the original attributes, respectively. spa G self These represent the discriminator and generator functions, respectively. For multi-scale pixel reconstruction loss function, Expressing the search function Expectations For the original face image features with attribute att, Let v represent the distribution of the generated face image and the desired attribute, respectively.id Z represents the attributes required to obfuscate the face image, id represents the identity embedding function.

[0044] The matching loss function is defined as follows:

[0045]

[0046] where φ l is the i-th layer feature map of the original face image extracted by the pre-trained VGG-19 model, is the i-th layer feature map of the synthesized image.

[0047] The reconstruction loss function is defined as follows:

[0048]

[0049] where Ψ represents the feature map scale, W Ψ and H Ψ are the width and height of the feature map at the corresponding scale Ψ.

[0050] According to experience, the feature map scale Ψ is set to 3, and the corresponding scales are 32×32, 64×64 and 128×128 to achieve the best reconstruction performance.

[0051] The attribute classification constraint loss function is defined as follows:

[0052]

[0053]

[0054]

[0055]

[0056] where represents the feature of the original face image with attribute b id , represents the feature of the generated face image with the expected attribute v id , is the prediction function of the i-th expected attribute, respectively represent the sum operation of all face attribute binary cross-entropy losses in the generator and the discriminator.

[0057] Step 4: Train the attention-generative adversarial network, the specific steps are as follows:

[0058] The face data set is input to the attention-generation adversarial network constructed in steps 1 and 2 for training, and in the training process, a plurality of pairs of face images are selected from the set of pairs of face images for each batch, and the corresponding training samples are taken as the current batch of training samples. Since the setting of the loss is very important for the training of the generative adversarial network, in order to improve the performance of the attention-generation adversarial network obtained by training, the hyperparameters η = 0.05, γ = 0.00005, ζ = 0.005 and ε = 0.001 are set when the network is trained. Through the above setting of the loss, the face image generated by the attention-generation adversarial network can be both different from the original face image and retain the features in the original face image, and the generated face image obtained from the face images of the same user is as similar as possible. The spatial attention guided discriminant network is used to discriminate the true and false of the image, calculate the generative adversarial loss and attribute classification constraint loss, and discriminate the consistency of the generated image and the original image and calculate the reconstruction loss and matching loss, finally the gradient is reversed to update the network parameters, and the final face image with privacy protection is output.

[0059] In summary, the above is only a preferred embodiment of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A face image privacy protection method based on attention-generative adversarial network, characterized in that, Comprise the following steps: Step one, construct attention-generative adversarial network, the attention-generative adversarial network includes self-attention perception generation network and spatial attention guide discriminant network, wherein the self-attention perception generation network, specifically: in order to be able to complete the reservation details information of the generated face image, on the basis of the extracted multi-scale face feature, the correlation between each feature layer of multi-scale face feature is calculated, the weighting coefficient of each feature layer fusion is calculated, and the label attribute information of face gender, age is embedded on the basis of multi-scale fusion feature combining self-attention mechanism model, global and detail feature face image is generated;The spatial attention guide discriminant network is specifically: by the average pooling, maximum pooling operation on the feature layer, multi-scale spatial feature is extracted, and the spatial feature map is generated by using feature aggregation operation, so as to be used as the basis for discrimination to guide the subsequent face discrimination and image generation;On this basis, the face attribute classifier and auxiliary face matcher are introduced into the discriminant network, and the visual realistic face image with the required attribute is generated for privacy protection; Step two, determine the optimization objective function of attention-generative adversarial network; Step three, train attention-generative adversarial network, output the final result. 2.The face image privacy protection method based on attention generative adversarial network of claim 1, wherein, The objective function for training the generator is: The objective function for the discriminator is: wherein, is a cycle consistency loss function, is a multi-scale reconstructed pixel loss function, is a reconstruction loss function, is an attribute classification constraint loss function for training the generator, is an attribute classification constraint loss function for the discriminator, is a matching loss function, and η, γ, ζ and ε represent hyperparameters for adjusting the weights of the respective objective functions. wherein, denotes the expectation of the function is the original face image feature with attribute att, denote the distribution of generated face images and expected attributes, respectively, D spa denotes the discriminator function, v id denotes the attribute required to confuse the face image, Z id denotes the identity embedding function.​ 3.The face image privacy protection method based on attention generative adversarial network of claim 1, wherein, The specific process of step three is: through attention-generative adversarial network, in the training process, combining face attribute matching function, face image generation function and attribute classification function, through training iteration optimization, face image with high fidelity and retaining face global and local visual and contour characteristics is generated.

Images