Privacy-preserving method for face generation based on hierarchical k-anonymous identity replacement

Abstract

The present invention discloses a face generation privacy protection method based on hierarchical k anonymous identity replacement, firstly, preprocessing of face image data set is performed, then hierarchical k anonymous generative adversarial network structure is constructed, hierarchical k anonymous target function is constructed; then face replacement generative adversarial network structure is constructed, target function of face replacement is constructed; finally, public data set is used for training and testing, and trained hierarchical k anonymous generative adversarial network and face replacement generative adversarial network are obtained. The target face replaced by the present invention is also generated by the network, so it will not infringe the privacy of others, which is more effective and visually friendly than the previous mosaic occlusion method. The experimental results clearly confirm the efficiency and practicality of the proposed method, and the privacy protection of character images is more efficient and beautiful.

Description

Technical Field

[0001] The present invention relates to the field of privacy protection of facial images. With the advent of the era of big data artificial intelligence, the privacy security of personal images has received more and more attention. Therefore, the present invention proposes a facial anonymity method, which replaces facial images by using a generated proxy dataset to achieve the purpose of image anonymity protection. Background Art

[0002] In the information age, with the rapid improvement of Internet technology, it has become possible to spread knowledge and ideas in real time, making the most authentic communication between people a reality. Today, the emergence of high-performance equipment and the continuous iteration of technology have put the development of artificial intelligence on the right track. Among them, image processing technology based on deep learning can be seen frequently in people's daily lives. For example, face recognition, image classification, object detection, intelligent monitoring, automatic driving, etc.

[0003] Technology is like a double-edged sword. While it brings convenience to our daily lives, it also needs to protect our safety. At present, face recognition technology is used for functions such as mobile phone password unlocking, access control switches, clocking in for work, transaction payments, and station access. Its inherent essence is to achieve the above functions by utilizing the rich information data contained in facial images. Nowadays, with the popularization of face recognition systems, user privacy and security issues are quite controversial. The face recognition system can be applied to people's daily lives thanks to the facial image information provided by each of us. It needs to collect a large amount of user image data to improve its recognition rate, and these image data are usually stored in the network cloud. If the facial image data is leaked, the user's privacy will suffer unprecedented losses.

[0004] In order to solve the problem of image privacy protection, some countries and regions have promulgated corresponding laws and regulations to protect the privacy of the public, and even recalled some face data sets. Although these methods have a certain degree of effect on protecting the privacy of face images, they cannot fundamentally solve the public's privacy problem. On the contrary, the implementation of these measures will increase the difficulty of research for scientific researchers. In fact, tasks such as face detection and pedestrian tracking do not require the identity information in the face image, so for a reliable anonymization method, it can not only hide the original identity information, but also use the anonymized image for other computer vision tasks.

[0005] In recent years, with the rapid development of deep learning, generative models represented by Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) have provided a technical basis and conditions for the intelligentization of privacy protection. By inputting the face image to be anonymized into a suitable generator model and adding the corresponding information to the appropriate layer, the corresponding anonymous face image is obtained. This is a research issue worthy of in-depth exploration.

[0006] In summary, using generative adversarial networks to anonymize facial images is a direction worthy of research. This patent intends to address several key issues in this task and solve the difficulties and key points of current methods.

[0007] A key point of image face anonymization is to ensure the effectiveness of face anonymity. However, for most of the current methods, the anonymized face is often very similar to the original image, thus failing to achieve the anonymity effect, or the anonymized image quality is very poor. Although the purpose of anonymity can be achieved, this seriously affects the usability of the anonymous image. Specifically, there are two key points:

[0008] (1) Validity of anonymous images. How to ensure that the anonymized images have different identity information from the original images, in other words, whether the anonymized images will appear in the original dataset, is an urgent problem to be solved.

[0009] (2) Currently, existing methods mainly achieve anonymity by replacing faces or using other people’s identities to merge with one’s own data. However, after replacing faces or identities, the target face or target identity information will also be leaked. Therefore, how to anonymize one’s own face image without leaking other people’s identity information is currently a difficult problem. Summary of the invention

[0010] In view of the shortcomings of the prior art, the present invention proposes a face generation privacy protection method based on hierarchical k anonymous identity replacement. The present invention mainly includes three points: 1. The present invention proposes a two-stage generation model for face identity anonymity; 2. It proposes to improve the traditional k-anonymity into a hierarchical k-anonymity method; 3. It proposes a face replacement model, encodes the face identity information, feeds the encoded information into the generator model, obtains the anonymity map of the corresponding identity, and improves the generation quality of the anonymous image.

[0011] The face generation privacy protection method based on hierarchical k anonymous identity replacement includes the following steps:

[0012] Step 1: Preprocessing of face image dataset;

[0013] Step 2: Construct a layer k anonymous generative adversarial network structure;

[0014] Step 3: Construct the objective function with level k anonymity;

[0015] Step 4: Construct a face replacement generative adversarial network structure;

[0016] Step 5: Construct the objective function of face replacement;

[0017] Step 6: Use public datasets for training and testing to obtain the trained layer-k anonymous generative adversarial network and face replacement generative adversarial network;

[0018] Step 7: Complete image anonymization through the trained layer k anonymous generative adversarial network and face replacement generative adversarial network.

[0019] Step 1 The specific steps are as follows:

[0020] 1-1 Use a face detector (dlib or MTCNN) to detect the face area of ​​each image, and crop the face area according to the face coordinates to obtain a cropped image containing only the face to avoid the influence of background information on the anonymity effect.

[0021] 1-2 uses k-means unsupervised clustering method to perform hierarchical clustering on the identity information of the cropped face images to obtain the cluster group id.

[0022] 1-3 uses the existing pre-trained face segmentation network to process the uncropped image and generate a mask for the face segmentation of each image. At the same time, the segmented face mask and the uncropped image are spliced ​​to obtain the uncropped background image.

[0023] Step 2 The specific steps are as follows:

[0024] The hierarchical k anonymous generative adversarial network includes a generator, a style information extractor, a true-false discriminator, and a group id discriminator. First, ordinary Gaussian noise and the group id obtained by hierarchical clustering of images are input into the style information extractor to obtain the style information of the corresponding group id; then the style information and the cropped image are input into the generator to obtain a generated image; secondly, the generated image and the corresponding group id are input into the true-false discriminator to judge the true or false of the image; at the same time, the generated image is also input into the group id discriminator to judge whether the generated image has the correct group id. Through the training of the above process, a generated image with realistic visual effects is finally obtained.

[0025] 2-1 Construct the generator. The generator is composed of multiple symmetrical Resblock residual blocks. Each residual block has a convolution layer, InstanceNorm2d normalization and LeakyReLU activation function. In addition, each residual block accepts a 128-dimensional style information, which is injected through AdaIN, and its expression is:

[0026]

[0027] Among them, θ i represents the image, μ(θ i ) represents the mean of the image, σ(θ i ) represents the standard deviation of the image, μ(s i ) represents the mean value of the style information input to the generator, σ(s i ) represents the standard deviation of style information.

[0028] 2-2 Construct a style information extractor. The style extractor consists of multiple fully connected layers and ReLU activation functions. It finally obtains the style information of the corresponding group based on the input ordinary Gaussian noise and the corresponding group id.

[0029] 2-3 Construct an image true or false discriminator. Similar to the generator structure, the discriminator is also composed of multiple Resblock residual blocks stacked together. The difference is that this residual block does not require input style information, and the discriminator uses the cluster group id to which the input image belongs as an index to distinguish true or false images.

[0030] 2-4 Construct a group ID discriminator. The group ID discriminator uses the network structure of VGG19. The VGG19 network uses the same convolution kernel size (3x3) and maximum pooling size (2x2). VGG19 contains 19 hidden layers, including 16 convolutional layers and 3 fully connected layers. The group ID discriminator is used to determine whether the generated image output from the generator has the correct group ID.

[0031] Step 3 The specific steps are as follows:

[0032] The hierarchical k-anonymous objective function includes an objective function of a generative adversarial network (GAN) and a group id discrimination objective function.

[0033] Objective function of GAN:

[0034] The idea of ​​conditional GAN ​​is used to control the generation of anonymous images, and different anonymous images are obtained by inputting different conditions. The specific operation is to input the cropped image and the style information of the target cluster into the generator to obtain the generated image of the target cluster; at the same time, the cropped image, the generated anonymous image, the corresponding cluster information, the target cluster ID and the gender information are also input into the image true and false discriminator to distinguish the true and false. In mathematical form, it can be expressed as:

[0035]

[0036] Among them, x represents the cropped image, s represents the output of the style extractor, y is jointly encoded by the clustering information, the group id to be mapped, and the gender information, G represents the generator, and D represents the image true and false discriminator.

[0037] Group id discrimination objective function:

[0038] Using the jointly trained VGG19 network model, the identity information loss between anonymous face images belonging to the same group ID is calculated. The VGG19 network is used to extract the identity information of the image, and then the L loss between the features of the images belonging to the same group ID is calculated. 1 Distance difference, for the same group id, the corresponding generated graph has the same identity information. It can be expressed mathematically as:

[0039]

[0040] Among them, x 1 and x 2 represents different cropped images, s is the output of the style extractor, and I represents the group id discriminator.

[0041] Step 4 The specific steps are as follows:

[0042] The face replacement generative adversarial network structure includes a generator, an identity extractor, a mask generator, and a true-false discriminator. First, the generated image obtained by the layer k anonymous generative adversarial network is input into the identity extractor to obtain the identity features of the image; then the obtained identity features and the uncropped background image are input into the generator; secondly, the uncropped original image is input into the mask generator to obtain a soft mask; finally, the output of the generator, the output of the mask generator, and the uncropped original image are spliced ​​to obtain the final anonymous image, and the anonymous image is input into the true-false discriminator to judge the authenticity. Through the training of the above process, an anonymous image with realistic visual effects is finally obtained.

[0043] 4-1 Construct the generator. The generator is formed by symmetrically combining multiple Resblock residual blocks. Each residual block has a convolutional layer, InstanceNorm2d normalization, and LeakyReLU activation function. In addition, each residual block accepts a 512-dimensional identity information, which is injected into the generator network through AdaIN.

[0044] 4-2 Construct an identity encoding extractor. The extractor consists of multiple Resblock residual blocks, which consist of a series of convolutional layers, InstanceNorm2d normalization layers, LeakyReLU activation functions, and average pooling layers. It extracts the identity information of the image based on the input face image.

[0045] 4-3 Build a true-false discriminator. The discriminator is also composed of multiple Resblock residual blocks stacked together. The true-false discriminator only needs to input a face image and finally output the probability of the image being true or false, thereby distinguishing true and false images.

[0046] 4-4 Construct a mask generator. The mask generator adopts the U-net structure, including 5 downsampling blocks and 5 upsampling blocks, and the middle bottleneck layer includes 3 Resblock residual blocks. The downsampling block includes a convolution layer, a BatchNorm normalization layer, and a ReLU activation function; the upsampling block includes an Upsample upsampling layer, a convolution layer, a BatchNorm normalization layer, and a ReLU activation function; the Resblock residual block includes two convolution layers and a ReLU activation function. The mask generator is used to generate a face area mask with a soft boundary.

[0047] Step 5 The specific steps are as follows:

[0048] The objective function of face replacement includes a GAN objective function, an identity preservation objective function, a reconstruction objective function and a mask generation objective function.

[0049] 5-1 Objective function of GAN. The idea of ​​CGAN is used to control the generation of images, thereby ensuring that the id of the image that provides identity features can be migrated to the original image. The specific operation is to input the uncropped background image into the generator, and feed the encoded identity information into the last five layers of Resblock residual blocks in the generator as a condition; similarly, the original image and the generated anonymous image are input into the discriminator for discrimination. In mathematical form, it can be expressed as:

[0050]

[0051] Where x is the original uncropped image, id is the encoded identity information, and m represents the face mask obtained by segmentation using the pre-trained segmentation network.

[0052] 5-2 Identity Preservation Objective Function. Use the pre-trained VGG16 network to calculate the feature loss, use the network to calculate the ID identity similarity between the generated anonymous image and the image with the identity feature, input the generated anonymous image and the image with the identity feature into the VGG16 network, and calculate the L between the feature maps. 1 At the same time, the generated anonymous image and the image providing identity features are input into the identity code extractor, and the cosine similarity of the extracted identity information is calculated, so that the generated anonymous image and the image providing identity features have the same identity information. In mathematical form, it can be expressed as:

[0053]

[0054] where V(·) represents multiple feature maps extracted from the VGG16 network, x t represents the face image of the target id, cos(·,·) represents the cosine similarity between two vectors, and Z id (·) denotes an identity code extractor.

[0055] 5-3 Reconstruction target function. When the target identity ID information is itself, the generated anonymous image should be the same as the original uncropped image, so the pixel level L is used. 1 Distance is used to construct the loss. In mathematical form, it can be expressed as:

[0056]

[0057] Where x is the original uncropped image, x rec Indicates a reconstructed image obtained using the same id information as itself.

[0058] 5-4 Mask generation objective function. Since the mask boundary obtained by using the pre-trained face segmentation network is not soft and is affected by the network pre-training performance, some face areas are not detected, so the L value of the generated mask and the segmented mask is used. 1 The distance is controlled so that the generated mask has similar boundaries to the segmented mask. In mathematical form, it can be expressed as:

[0059]

[0060] Among them, M(·) represents the mask generator, x represents the original image with background, and m represents the face mask obtained using the pre-trained segmentation network.

[0061] Step 6 The specific steps are as follows:

[0062] 6-1 Prepare the data set. Use the public face data set and process it according to the preprocessing process described in step 1 to obtain a cropped image containing only the face, the clustered group ID, the face segmentation mask, and the uncropped background image.

[0063] 6-2 Input the cropped image and cluster group ID into the hierarchical k anonymous generative adversarial network for training, and use the test data for testing. After the training, the hierarchical k anonymous generative adversarial network obtains the generated image corresponding to the training data, which is used to provide identity features for the face replacement generative adversarial network model.

[0064] 6-3 Input the face segmentation mask, the uncropped original image, the uncropped background image, and the generated image obtained by the layer-k anonymous generative adversarial network into the face replacement generative adversarial network for training, and test it with test data.

[0065] Step 7 The specific steps are as follows:

[0066] 7-1 The image to be anonymized is preprocessed in step 1 to obtain its cropped image, hierarchical clustering group ID and face segmentation mask.

[0067] 7-2 Input the cropped image and hierarchical clustering group ID into the trained layer-k anonymous generative adversarial network to obtain its generated graph.

[0068] 7-3 Input the generated image, face segmentation mask, background image and uncropped image obtained by the layer k anonymous generative adversarial network into the trained face replacement generative adversarial network, and finally obtain the anonymous map of the uncropped image.

[0069] The benefits of the present invention are:

[0070] This method replaces the face area in the image to achieve the effect of face anonymity. At the same time, the target face is also generated through the network, so it will not infringe on the privacy of others. Compared with the previous mosaic occlusion method, it is more effective and visually friendly. The experimental results clearly confirm the efficiency and practicality of the proposed method. In short, the proposed method is more efficient and beautiful for the privacy protection of human images. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is a flowchart of the steps of an embodiment of the method of the present invention;

[0072] Figure 2 This is a diagram of the architecture of a k-level anonymous generative adversarial network according to an embodiment of the method of the present invention;

[0073] Figure 3 This is a network architecture diagram of a face replacement model according to an embodiment of the method of the present invention. DETAILED DESCRIPTION

[0074] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0075] like Figure 1 As shown in FIG. 1 , the face generation privacy protection method based on hierarchical k anonymous identity replacement has the following specific steps:

[0076] Step 1: Dataset preprocessing:

[0077] 1-1 Use a face detector (dlib or MTCNN) to detect the face area of ​​each image, and crop the face area according to the face coordinates to obtain a cropped image containing only the face to avoid the influence of background information on the anonymity effect.

[0078] 1-2 uses k-means unsupervised clustering method to hierarchically cluster the identity information of the cropped face images to obtain the cluster group id.

[0079] 1-3 uses the existing pre-trained face segmentation network to process the uncropped image and generate a mask for the face segmentation of each image. At the same time, the segmented face mask and the uncropped image are spliced ​​to obtain the uncropped background image.

[0080] Step 2: Construct a layer k anonymous generative adversarial network structure:

[0081] like Figure 2 As shown, the hierarchical k anonymous generative adversarial network includes a generator, a style information extractor, a true-false discriminator and a group id discriminator. First, ordinary Gaussian noise and the group id obtained by hierarchical clustering of images are input into the style information extractor to obtain the style information of the corresponding group id; then the style information and the cropped image are input into the generator to obtain a generated graph; secondly, the generated graph and the corresponding group id are input into the true-false discriminator to judge the true or false of the image; at the same time, the generated graph is also input into the group id discriminator to judge whether the generated graph has the correct group id. Through the training of the above process, a generated graph with a more realistic visual effect is finally obtained. The generated graph generated in this process provides identity features for the identity replacement network in step 4.

[0082] 2-1 Construct the generator. The generator is composed of multiple symmetrical Resblock residual blocks. Five Resblocks are used to construct the network structure, and the 128-dimensional style information will be injected into the following five Resblock residual blocks in the form of AdaIN. The style information is injected in the form of AdaIN, and its expression is:

[0083]

[0084] Among them, θ i represents the image, μ(θ i ) represents the mean of the image, σ(θ i ) represents the standard deviation of the image, μ(s i ) represents the mean value of the style information input to the generator, σ(s i ) represents the standard deviation of style information.

[0085] 2-2 Construct a style information extractor. The style extractor consists of six fully connected layers and six ReLu activation functions. It obtains the style information of the corresponding group based on the input ordinary Gaussian noise and the corresponding group id.

[0086] 2-3 Construct an image true or false discriminator. The discriminator consists of five Resblock residual blocks, four convolutional layers, and four LeakyReLU activation functions. This residual block does not need to receive style information, and the discriminator uses the cluster group id to which the input image belongs as an index to distinguish true or false images.

[0087] 2-4 Construct a group ID discriminator. The group ID discriminator uses the network structure of VGG19. The VGG19 network uses the same convolution kernel size (3x3) and maximum pooling size (2x2). VGG19 contains 19 hidden layers, including 16 convolutional layers and 3 fully connected layers. The group ID discriminator is used to determine whether the generated image output from the generator has the correct group ID.

[0088] Step 3: Construct the objective function with level k anonymity:

[0089] The hierarchical k-anonymous objective function includes an objective function of a generative adversarial network (GAN) and a group id discrimination objective function.

[0090] Objective function of GAN:

[0091] The idea of ​​conditional GAN ​​is used to control the generation of anonymous images. Since the present invention adopts the idea of ​​k-anonymity, different anonymous graphs are obtained by inputting different conditions. The specific operation is to input the cropped image and the style information of the target cluster into the generator to obtain the generated graph of the target cluster; at the same time, the cropped image, the generated anonymous graph, the corresponding cluster information, the target cluster ID and the gender information are also input into the image true and false discriminator to distinguish the true and false. In mathematical form, it can be expressed as:

[0092]

[0093] Among them, x represents the cropped image, s represents the output of the style extractor, y is jointly encoded by the clustering information, the group id to be mapped, and the gender information, G represents the generator, and D represents the image discriminator for authenticity.

[0094] Group id discriminant objective function:

[0095] Using the jointly trained VGG19 network model, calculate the loss of the identity information between the face anonymized images belonging to the same group id. Use the VGG19 network to extract the identity information of the images, and then calculate the L 1 distance difference. For the same group id, the corresponding generated images have the same identity information. Mathematically, it can be expressed as:

[0096]

[0097] where, x 1 and x 2 represent different cropped images, s is the output of the style extractor, and I represents the group id discriminator.

[0098] Step 4: Construct the face replacement generative adversarial network structure:

[0099] As Figure 3 shown, the described face replacement generative adversarial network structure includes a generator, an identity extractor, a mask generator, and a discriminator for authenticity. First, input the generated image obtained by the hierarchical k-anonymization generative adversarial network into the identity extractor to obtain the identity features of the image; then input the obtained identity features and the uncropped background image into the generator; secondly, input the uncropped original image into the mask generator to obtain a soft mask; finally, splice the output of the generator, the output of the mask generator, and the uncropped original image to obtain the final anonymized image, and input this anonymized image into the discriminator for authenticity to judge its authenticity. Through the training of the above process, finally obtain an anonymized image that is visually more realistic.

[0100] 4-1 Construct the generator. The generator is symmetrically composed of five Resblock residual blocks at the front and back. Each residual block has three convolutional layers, two InstanceNorm2d normalization layers, and two LeakyReLU activation functions. In addition, each residual block receives an identity information of 512 dimensions, and this identity information is injected into the generator network in the way of AdaIN.

[0101] 4-2 Construct an identity encoding extractor. The extractor consists of five Resblock residual blocks, which consist of three convolutional layers, two InstanceNorm2d normalization layers, two LeakyReLU activation functions, and an average pooling layer. It extracts the identity information of the image based on the input face image.

[0102] 4-3 Build a true-false discriminator. The discriminator is also composed of five Resblock residual blocks, four convolution blocks and four LeakyReLU activation functions. The true-false discriminator only needs to input a face image and finally output the probability of the image being true or false, thereby distinguishing true and false images.

[0103] 4-4 Construct a mask generator. The mask generator adopts the U-net structure, including 5 downsampling blocks and 5 upsampling blocks, and the middle bottleneck layer includes 3 Resblock residual blocks. The downsampling block includes a convolution layer, a BatchNorm normalization layer, and a ReLU activation function; the upsampling block includes an Upsample upsampling layer, a convolution layer, a BatchNorm normalization layer, and a ReLU activation function; the Resblock residual block includes two convolution layers and a ReLU activation function. The mask generator is used to generate a face area mask with a soft boundary.

[0104] Step 5: Construct the objective function of face replacement:

[0105] The objective function of face replacement includes a GAN objective function, an identity preservation objective function, a reconstruction objective function and a mask generation objective function.

[0106] 5-1 Objective function of GAN. The idea of ​​CGAN is used to control the generation of images, thereby ensuring that the ID that provides identity features can be migrated to the original image. The specific operation is to input the uncropped background image into the generator, and at the same time feed the encoded identity information into the last five layers of Resblock residual blocks in the generator as a condition; similarly, the original image and the generated anonymous image are input into the discriminator for discrimination. In mathematical form, it can be expressed as:

[0107]

[0108] Where x is the original uncropped image, id is the encoded identity information, and m represents the face mask obtained by segmentation using the pre-trained segmentation network.

[0109] 5-2 Identity Preservation Objective Function. Use the pre-trained VGG16 network to calculate the feature loss, use the network to calculate the ID identity similarity between the generated anonymous image and the image with identity features, input the generated anonymous image and the image with identity features into the VGG16 network, and calculate the L between the feature maps of the middle layers. 1 At the same time, the generated anonymous image and the image providing identity features are input into the identity code extractor, and the cosine similarity of the extracted identity information is calculated, so that the generated anonymous image and the image providing identity features have the same identity information. In mathematical form, it can be expressed as:

[0110]

[0111] where V(·) represents multiple feature maps extracted from the VGG16 network, x t represents the face image of the target id, cos(·,·) represents the cosine similarity between two vectors, and Z id (·) denotes an identity code extractor.

[0112] 5-3 Reconstruction target function. When the target identity ID information is itself, the generated anonymous image should be the same as the original uncropped image, so the pixel level L is used. 1 Distance is used to construct the loss. In mathematical form, it can be expressed as:

[0113]

[0114] Where x is the original uncropped image, x rec Indicates a reconstructed image obtained using the same id information as itself.

[0115] 5-4 Mask generation objective function. Since the mask boundary obtained by using the pre-trained face segmentation network is not soft and is affected by the network pre-training performance, some face areas are not detected, so the L value of the generated mask and the segmented mask is used. 1 The distance is controlled so that the generated mask has similar boundaries to the segmented mask. In mathematical form, it can be expressed as:

[0116]

[0117] Among them, M(·) represents the mask generator, x represents the original image with background, and m represents the face mask obtained using the pre-trained segmentation network.

[0118] Step 6: Use public datasets for training and testing to obtain the trained layer-k anonymous generative adversarial network and face replacement generative adversarial network:

[0119] 6-1 Prepare a data set, for example, use public data sets such as VGGFACE2 and CelebA and process them according to the preprocessing process described in step 1, and obtain a cropped image containing only the face, a clustered group ID, a face segmentation mask, and an uncropped background image.

[0120] 6-2 Input the cropped image and cluster group ID into the hierarchical k anonymous generative adversarial network for training, and use the test data for testing. After the training, the hierarchical k anonymous generative adversarial network obtains the generated image corresponding to the training data, which is used to provide identity features for the face replacement generative adversarial network model.

[0121] 6-3 Input the face segmentation mask, the uncropped original image, the uncropped background image, and the generated image obtained by the layer-k anonymous generative adversarial network into the face replacement generative adversarial network for training, and test it with test data.

[0122] Step 7: Complete image anonymity through the trained layer k anonymous generative adversarial network and face replacement generative adversarial network:

[0123] 7-1 The image to be anonymized is preprocessed in step 1 to obtain its cropped image, hierarchical clustering group ID and face segmentation mask.

[0124] 7-2 Input the cropped image and hierarchical clustering group ID into the trained layer-k anonymous generative adversarial network to obtain its generated graph.

[0125] 7-3 Input the generated image, face segmentation mask, background image and uncropped image obtained by the layer k anonymous generative adversarial network into the trained face replacement generative adversarial network, and finally obtain the anonymous map of the uncropped image.

[0126] Experimental results:

[0127] 1. The anonymity rates of this method and the blurring, mosaic, NEO, and CIAGAN methods were tested respectively. The specific data results are shown in Table 1.

[0128] Table 1 Experimental results of anonymity rate comparison between this method and other methods

[0129] method Occlusion Vague Pixelation Random Noise NEO CIAGAN This method Anonymity rate 1.000 0.6989 0.9090 0.2399 0.8433 0.9694 0.9992

[0130] 2 The identity exchange rates of this method and the fuzzy, mosaic, NEO, and CIAGAN methods were tested respectively. The specific data results are shown in Table 2.

[0131] Table 2 Comparison experimental results of identity exchange rate between this method and other methods

[0132] method Occlusion Vague Pixelation Random Noise NEO CIAGAN This method Identity exchange rate 0.0000 0.3577 0.1383 0.9970 0.9990 0.0639 0.0524

[0133] 3. The facial recognition efficiency of this method was tested against that of the blurring, mosaic, NEO, and CIAGAN methods. See Table 3 for details of the specific data results.

[0134] Table 3 Comparison experimental results of the facial recognition efficiency of this method and other methods

[0135]

[0136] 4. The image quality of this method and the blurring, mosaic, NEO, and CIAGAN methods were tested respectively. The specific data results are detailed in Table 4.

[0137] Table 4 Experimental results of image quality comparison between this method and other methods

[0138] method Occlusion Vague Pixelation Random Noise NEO CIAGAN This method Image Quality 2400.69 434.82 550.97 746.12 53.36 36.03 13.56 .

Claims

1. Privacy-preserving face generation method based on hierarchical k-anonymous identity replacement, It is characterized in that The following steps are involved: Step 1: Preprocessing of face image dataset; Step 2: Construct a layer k anonymous generative adversarial network structure; Step 3: Construct the objective function with level k anonymity; Step 4: Construct a face replacement generative adversarial network structure; Step 5: Construct the objective function of face replacement; Step 6: Use public datasets for training and testing to obtain the trained layer-k anonymous generative adversarial network and face replacement generative adversarial network; Step 7: Complete image anonymity through the trained layer k anonymous generative adversarial network and face replacement generative adversarial network; Step 1 The specific steps are as follows: 1-1 Use a face detector to detect the face area of ​​each image, and crop the face area according to the face coordinates to obtain a cropped image containing only the face to avoid the influence of background information on the anonymity effect; 1-2 Use k-means unsupervised clustering method to perform hierarchical clustering on the identity information of the cropped face image to obtain the cluster group id; 1-3 Use the existing pre-trained face segmentation network to process the uncropped image and generate a mask of the face segmentation of each image, and concatenate the segmented face mask and the uncropped image to obtain the uncropped background image; Step 2 The specific steps are as follows: The hierarchical k anonymous generative adversarial network comprises a generator, a style information extractor, a true-false discriminator and a group id discriminator; firstly, ordinary Gaussian noise and the group id obtained by hierarchical clustering of images are input into the style information extractor to obtain the style information of the corresponding group id; then the style information and the cropped image are input into the generator to obtain a generated image; secondly, the generated image and the corresponding group id are input into the true-false discriminator to judge the true or false of the image; at the same time, the generated image is also input into the group id discriminator to judge whether the generated image has the correct group id; through the training of the above process, a generated image with realistic visual effects is finally obtained; 2-1 Construct the generator; the generator is composed of multiple symmetrical Resblock residual blocks; each residual block has a convolution layer, InstanceNorm2d normalization and LeakyReLU activation function. In addition, each residual block accepts a 128-dimensional style information, which is injected through AdaIN. Its expression is: Among them, θ i represents the image, μ(θ i ) represents the mean of the image, σ(θ i ) represents the standard deviation of the image, μ(s i ) represents the mean value of the style information input to the generator, σ(s i ) represents the standard deviation of style information; 2-2 Construct a style information extractor; the style information extractor consists of multiple fully connected layers and ReLU activation functions, which finally obtains the style information of the corresponding group based on the input ordinary Gaussian noise and the corresponding group id; 2-3 Construct an image true or false discriminator; Similar to the generator structure, the discriminator is also composed of multiple Resblock residual blocks stacked together. The difference is that this residual block does not require input style information, and the discriminator uses the group id of the cluster to which the input image belongs as an index to distinguish true or false images; 2-4 Construct a group ID discriminator; the group ID discriminator adopts the network structure of VGG19. The VGG19 network uses the same size of convolutional kernel and max pooling size. VGG19 contains 19 hidden layers, including 16 convolutional layers and 3 fully connected layers. The group ID discriminator is used to determine whether the generated graph output from the generator has the correct group ID. The specific steps of step 4 are as follows: The described face replacement generative adversarial network structure includes a generator, an identity extractor, a mask generator, and a real / fake discriminator. First, the generated image obtained from the hierarchical k-anonymous generative adversarial network is input into the identity extractor to obtain the identity features of the image. Then, the obtained identity features and the uncropped background image are input into the generator. Secondly, the uncropped original image is input into the mask generator to obtain a soft mask. Finally, the output of the generator, the output of the mask generator, and the uncropped original image are concatenated to obtain the final anonymized image, and this anonymized image is input into the real / fake discriminator to judge its authenticity. Through the training of the above process, a visually realistic anonymized image is finally obtained. 4-1 Construct a generator; the generator is symmetrically composed of multiple Resblock residual blocks. Each residual block has a convolutional layer, InstanceNorm2d normalization, and a LeakyReLU activation function. In addition, each residual block receives an identity information with a dimension of 512, and this identity information is injected into the generator network in the way of AdaIN. 4-2 Construct an identity encoding extractor; the extractor is composed of multiple Resblock residual blocks. The residual block consists of a series of convolutional layers, InstanceNorm2d normalization layers, LeakyReLU activation functions, and average pooling layers, and it extracts the identity information of the input face image. 4-3 Construct a real / fake discriminator; the discriminator is also stacked by multiple Resblock residual blocks. The real / fake discriminator only needs to input a face image, and finally outputs the probability of the authenticity of the image, so as to discriminate between real and fake images. 4-4 Construct a mask generator; the mask generator adopts the structure of U-net, including 5 downsampling blocks and 5 upsampling blocks. The middle bottleneck layer includes 3 Resblock residual blocks. The downsampling block includes a convolutional layer, BatchNorm normalization layer, and ReLU activation function. The upsampling block includes an Upsample upsampling layer, a convolutional layer, BatchNorm normalization layer, and ReLU activation function. The Resblock residual block includes two convolutional layers and a ReLU activation function. The mask generator is used to generate a face region mask with a soft boundary. The specific steps of step 7 are as follows: 7-1 Preprocess the image to be anonymized according to step 1 to obtain its cropped image, the group ID of hierarchical clustering, and the face segmentation mask. 7-2 Input its cropped image and the group ID of hierarchical clustering into the trained hierarchical k-anonymous generative adversarial network to obtain its generated graph. 7-3 Input the generated image, face segmentation mask, background image and uncropped image obtained by the layer k anonymous generative adversarial network into the trained face replacement generative adversarial network, and finally obtain the anonymous map of the uncropped image.

2. The face generation privacy protection method based on hierarchical k anonymous identity replacement according to claim 1, It is characterized in that Step 3 The specific steps are as follows: The objective function of the hierarchical k anonymity includes the objective function of the generative adversarial network GAN and the group id discrimination objective function; Objective function of GAN: The idea of ​​conditional GAN ​​is used to control the generation of anonymous images. Different anonymous images are obtained by inputting different conditions. The specific operation is to input the cropped image and the style information of the target cluster into the generator to obtain the generated image of the target cluster. At the same time, the cropped image, the generated anonymous image, the corresponding cluster information, the group ID and gender information of the target cluster are input into the image true and false discriminator to distinguish the true and false. In mathematical form, it is expressed as: Among them, x represents the cropped image, s represents the output of the style extractor, y is jointly encoded by the clustering information, the group id to be mapped, and the gender information, G represents the generator, and D represents the image true and false discriminator; Group id discrimination objective function: Use the jointly trained VGG19 network model to calculate the loss of identity information between anonymous face images belonging to the same group ID; use the VGG19 network to extract the identity information of the image, and then calculate the L loss between the features of the images belonging to the same group ID. 1 Distance difference, for the same group id, the corresponding generated graph has the same identity information; mathematically expressed as: Among them, x 1 and x 2 represents different cropped images, s is the output of the style extractor, and I represents the group id discriminator.

3. The face generation privacy protection method based on hierarchical k anonymous identity replacement according to claim 2, It is characterized in that Step 5 The specific steps are as follows: The objective function of face replacement includes the objective function of GAN, the identity preservation objective function, the reconstruction objective function and the mask generation objective function; 5-1 Objective function of GAN; The idea of ​​CGAN is used to control the generation of images, thereby ensuring that the id of the image that provides identity features can be migrated to the original image; The specific operation is to input the uncropped background image into the generator, and feed the encoded identity information into the last five layers of Resblock residual blocks in the generator as a condition; Similarly, the original image and the generated anonymous image are input into the discriminator for discrimination; In mathematical form, it is expressed as: Where x' is the original uncropped image, id is the encoded identity information, and m represents the face mask obtained by segmentation using the pre-trained segmentation network; 5-2 Identity Preservation Objective Function; Use the pre-trained VGG16 network to calculate the feature loss, use the network to calculate the ID identity similarity between the generated anonymous image and the image with identity features, input the generated anonymous image and the image with identity features into the VGG16 network, and calculate the L between the feature maps 1 distance; at the same time, the generated anonymous image and the image providing identity features are input into the identity code extractor, and the cosine similarity of the extracted identity information is calculated, so that the generated anonymous image and the image providing identity features have the same identity information; in mathematical form, it is expressed as: where V(·) represents multiple feature maps extracted from the VGG16 network, x t represents the face image of the target id, cos(·,·) represents the cosine similarity between two vectors, and Z id (·) represents identity code extractor; 5-3 Reconstruction target function; When the target ID identity information used is itself, the generated anonymous image should be the same as the original uncropped image, so the pixel level L is used 1 Distance is used to construct the loss; in mathematical form it is expressed as: Among them, x' is the original uncropped image, x rec Indicates a reconstructed image obtained using the same id information as itself; 5-4 Mask generation objective function; Since the mask boundary obtained by using the pre-trained face segmentation network is not soft and is affected by the network pre-training performance, some face areas are not detected, so the L of the generated mask and the segmented mask is used. 1 The distance is controlled so that the generated mask has a similar boundary to the segmented mask; in mathematical form: Where M(·) represents the mask generator, x″ represents the original image with background, and m represents the face mask obtained using the pre-trained segmentation network.

4. The face generation privacy protection method based on hierarchical k anonymous identity replacement according to claim 3, It is characterized in that Step 6 The specific steps are as follows: 6-1 Prepare a data set, use a public face data set and process it according to the preprocessing process described in step 1, and obtain a cropped image containing only the face, a clustered group ID, a face segmentation mask, and an uncropped background image; 6-2 Input the cropped image and the clustered group ID into the layer k anonymous generative adversarial network for training, and use the test data for testing. After the training, the layer k anonymous generative adversarial network is used to obtain the generated image corresponding to the training data, which is used to provide identity features for the face replacement generative adversarial network model; 6-3 Input the face segmentation mask, the uncropped original image, the uncropped background image, and the generated image obtained by the layer-k anonymous generative adversarial network into the face replacement generative adversarial network for training, and test it with test data.

Images