A face anonymization unified framework method based on knowledge distillation

Abstract

The application provides a face anonymization unified framework method based on knowledge distillation, comprising the following steps: step 1, obtaining a face image dataset; step 2, constructing a face face latent space conversion network, taking the face image data as input to obtain a face latent code and a face pixel space through the face face latent space conversion network; step 3, constructing a latent space feature conversion module, and taking the face latent code as input to obtain a domain conversion code in the latent code space through the latent space feature conversion module; step 4, face anonymization face generation; step 5, constructing an anonymous face generation target function; step 6, training and testing using a public dataset, and outputting a final result. The framework realizes multi-domain conversion of traditional, deep learning-based and other anonymization methods, and flexibly protects identity information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face image processing, and particularly relates to a face anonymization unified framework method based on knowledge distillation. BACKGROUND

[0002] Due to the continuous development of scientific and technological information, the explosive growth of face images in social media, and the wide application of face recognition technology in various fields, the security of personal identity privacy information is facing unprecedented challenges. In order to solve the privacy protection problem of images, some countries and regions have enacted corresponding laws and regulations to protect the privacy of the public, and even some face data sets are recalled. Although these methods have a certain effect on protecting the privacy of face images, these methods cannot fundamentally solve the privacy problem of the public, and the implementation of these measures will increase the research difficulty of researchers. Face de-identification technology is considered as an effective visual privacy protection tool, and its work is to hide or replace the identity information contained in the image.

[0003] Computer vision technology has been widely used in visual recognition and other tasks, bringing great convenience to people's daily life, but also bringing great risks. Many private photos and videos are uploaded to the cloud or sent to third parties for analysis and recognition tasks, and facial information is included. However, facial information is sensitive information containing personal information. If not carefully protected, highly sensitive facial information is easy to be accessed and illegally used by third parties or malicious attackers. Therefore, when performing conventional computer vision tasks, the privacy of sensitive information needs to be protected, and a technology that can guarantee the security of facial information is needed. In traditional face anonymization algorithms, face privacy protection methods are all achieved by irreversible processing of the original face. Traditional face anonymization methods mainly include: blur processing, pixelization processing, image filtering, etc. These methods have the advantages of simple implementation and small amount of calculation, but often cannot well preserve the visual information of the image while protecting the privacy.

[0004] At present, there are many works on the market to realize the privacy and anonymity protection of human faces. Among them, there are many different methods, including traditional blurring, pixelization processing, face changing based face protection, and deep learning based face image anonymity method. However, the use and effect of different methods are very different. For example, the traditional blurring and pixelization processing method usually only needs simple image processing algorithms such as blurring filter or pixelization operation. While the deep learning based method may need a large amount of labeled data and training time to build an accurate face feature extraction model. In addition, some methods may need additional auxiliary information such as pose estimation, face key point detection or face recognition model to provide more detailed privacy protection. In addition, the size of the model is also related to the calculation efficiency and real-time performance. Larger models may require longer inference time and more computing resources, which are not suitable for real-time applications.

[0005] In view of the above problems, there are differences in auxiliary conditions and model size between different anonymity methods, which makes it complex to choose and use anonymity methods in practical applications. The significance of the research of unified anonymity framework lies in solving the complexity between different anonymity methods and promoting the comparison, integration and innovation between methods.

[0006] In summary, by establishing a unified framework, anonymity methods can be better selected and used, the effect of privacy protection can be improved, and the development of the entire field can be promoted, which has important value and research significance.

[0007] The traditional and deep learning based anonymity methods have limitations. For traditional methods, the anonymity effect is significant but the usability is poor; for deep learning methods, the usability is improved, but the consumption of computing resources and time is increased. Moreover, with the continuous progress of technology, new anonymity methods are gradually emerging, and the selection of anonymity methods is difficult. Specifically, there are mainly two difficulties:

[0008] I. How to integrate traditional and deep learning based anonymity methods, and whether the balance between anonymity and data reuse is balanced.

[0009] II. How to face the continuous emergence of new anonymity methods, whether they can be integrated into the existing unified anonymity framework. SUMMARY

[0010] The present application proposes a face anonymity unified framework method based on knowledge distillation to solve the problems of the prior art, which integrates the multi-domain conversion of traditional, deep learning based and other anonymity methods, and flexibly protects the identity information.

[0011] In order to solve the above technical problems, the technical scheme of the present application is:

[0012] A face anonymization unified framework method based on knowledge distillation, comprising the following steps:

[0013] Step 1, obtaining a face image dataset;

[0014] Step 2, constructing a face face latent space conversion network, taking the face image data as input through the face face latent space conversion network to obtain the face latent code and the face pixel space;

[0015] The face face latent space conversion network comprises an encoder, a decoder and a discriminator,

[0016] Step 3, constructing a latent space feature conversion module, and taking the face latent code as input through the latent space feature conversion module to obtain the domain conversion code in the latent code space;

[0017] The latent space feature conversion module comprises an anonymous method selection module and an anonymous method analysis module, the anonymous method selection module is composed of a plurality of shared fully connected layers FC and ReLu layers, and finally connects a multi-branch fully connected layer; The anonymous method analysis module is composed of a plurality of ResNet blocks stacked;

[0018] Step 4, face anonymization face generation;

[0019] Step 5, constructing an anonymous face generation target function;

[0020] Step 6, training and testing using public datasets, and outputting the final result.

[0021] As preferred, in the step 1, the obtained face image dataset is a clear face image dataset with balanced male and female proportions.

[0022] As preferred, in the step 2, the encoder E maps the input face image x i into the latent space Z to obtain the face latent code z i , the decoder G recovers the corresponding face pixel space from the face latent code z i . The discriminator D assists the VAE to improve the image authenticity, and the expression is as follows:

[0023] z i =E(x i )

[0024]

[0025] As preferred, the input of the anonymous method selection module S is 16-dimensional Gaussian distribution acquisition data Latentcode, and the output of each branch corresponds to the domain encoding s i of an anonymous method.

[0026] As preferred, the input of the anonymous method analysis module A includes two parts: one part of the input is the face latent code z i , and the other part is the selected anonymous method domain code s i , and the output is a domain conversion code z' i of the same dimension as the face latent code, expressed as follows:

[0027] s i =S(n)

[0028] z′ i =A(z i |s i ).

[0029] As preferred, in step 3, the face latent code z i is input into the anonymous method analysis module A, and the intermediate output is θ i , s i is obtained by the AdaIN method acting on the anonymous method analysis module A, and its expression is as follows:

[0030]

[0031] As preferred, in step 6, z' i is converted to the face pixel domain space by the G module of the VAE-GAN, to obtain the anonymous domain s i and the corresponding anonymous image x' i , and its expression is as follows:

[0032] x′ i =G(z′ i ).

[0033] As preferred, the objective function includes:

[0034] The objective function of the face facial latent space conversion network is expressed as follows:

[0035]

[0036] wherein z represents the face facial latent code, x represents the original image data, G represents the generator, and D represents the discriminator;

[0037] The face image style loss objective function is expressed as follows:

[0038]

[0039] wherein x represents the original image, and x t represents the target anonymous image;

[0040] The feature matching loss function is expressed as follows:

[0041]

[0042] Wherein x is the original image, s is the anonymous domain coding, E is the encoder;

[0043] The latent space feature loss function is expressed as follows:

[0044]

[0045] Wherein x is the original image, x t is the target anonymous image, s is the anonymous domain coding, E is the encoder, A is the anonymous method analysis module, and G is the decoding generator;

[0046] The mean absolute error loss function is expressed as follows:

[0047]

[0048] Wherein x is the original image, x t is the target anonymous image.

[0049] Preferably, in step 6, the training method is: using the Adam optimizer, the learning rate is set to 0.0001, the beta 1 coefficient is set to 0.5, the beta 2 coefficient is set to 0.999, and the Batch size size is 8.

[0050] The present application has the following characteristics and beneficial effects:

[0051] The present application designs a new face anonymization unified framework based on knowledge distillation. The fusion of traditional and deep learning-based anonymization methods is realized, different anonymization methods can be switched according to different scenes, and the usability and effectiveness are neutralized. In addition, the framework also supports expansion to cope with new face anonymization methods. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0053] Figure 1 is the face face latent space conversion network flowchart of the present application.

[0054] Figure 2 is the overall architecture flowchart of the present application.

[0055] Figure 3 is the anonymous image generation effect comparison chart of the present application. DETAILED DESCRIPTION

[0056] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0057] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0058] In the description of the present application, it should be noted that unless otherwise specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0059] The present application provides a face anonymization unified framework method based on knowledge distillation, as shown in Figure 1 and Figure 2 The method comprises the following steps:

[0060] Step 1, obtaining a face image data set.

[0061] In this embodiment, the selected data set is CelebA-HQ data set, wherein the face image must be clear, and the male and female ratio is balanced, and there are 22000 male and female images. The image is normalized and the size is unified to 256*256.

[0062] Step 2, constructing a face face latent space conversion network, encoding the data set, doing compression dimension reduction processing, reducing parameter calculation, and discriminator auxiliary decoding to face pixel space to ensure image quality.

[0063] Wherein, the face latent space conversion network is composed of an encoder E, a decoder G and a discriminator D, and the face image data is taken as input to obtain the face latent code and the face pixel space through the face latent space conversion network.

[0064] Specifically, the encoder maps the input face x i to the latent space Z to obtain the face latent code z i . The decoder G restores the corresponding face pixel space according to the latent code.

[0065] z i =E(x i )

[0066]

[0067] The network encodes the data set, performs compression and dimension reduction processing, reduces parameter calculation, and the discriminator assists decoding to the face pixel space to ensure image quality.

[0068] Step 3, constructing a latent space feature conversion module, and taking the face latent code as input to obtain the domain conversion code in the latent space through the latent space feature conversion module.

[0069] Specifically, in the embodiment, the latent space feature conversion module includes two parts: an anonymous method selection module S and an anonymous method analysis module A.

[0070] S is composed of multiple shared fully connected layers FC and ReLu layers, and finally connects multiple branch fully connected layers corresponding to multiple anonymous methods. The input is 16-dimensional Gaussian distribution acquisition data Latent code(n), and the output of each branch corresponds to the domain encoding s i of an anonymous method.

[0071] In the embodiment, the module maps the 16-dimensional data collected in the Gaussian distribution to a 512-dimensional vector of each anonymous method. The intermediate hidden layer is a 1024-dimensional fully connected neural network.

[0072] The anonymous method analysis module A is composed of multiple ResNet blocks stacked and receives two parts of input. One part of the input is the face latent code z i , and the other part is the selected anonymous method domain encoding s i , and the output is a domain conversion code z' i with the same dimension as z i , and the expression is as follows:

[0073] s i =S(n)

[0074] z′ i= A(z i |s i )。

[0075] In this embodiment, the module is composed of multiple ResNet blocks stacked, the input is the face latent code, and the 512-dimensional anonymous method encoding output by the anonymous method selection module is received to act on the ResNet block in the AdaIN manner to realize the conversion of different anonymous domain latent spaces.

[0076] The face image latent code z i As the input of module A, the intermediate output is θ i , s i acted on A in the AdaIN manner, and its expression is:

[0077]

[0078] Step 4, face anonymous face generation.

[0079] Through the G module of the face face latent space conversion network, z' i is converted to the face pixel domain space to obtain the anonymous domain s i corresponding anonymous image x' i , and its expression is:

[0080] x′ i = G(z′ i ).

[0081] It can be understood that the face latent code has a dimension of 8*8, and the corresponding 8*8 latent space encoding of the anonymous domain is generated through the anonymous method selection and analysis module. The decoder of the face face latent space conversion network generates a 256*256, 3-channel face pixel space image.

[0082] Step 5, construct an anonymous face generation target function;

[0083] The total loss includes the adversarial loss, the style loss, the feature matching loss function, the latent space feature loss function, and the mean absolute error loss function, which ensures the stability of the conversion and generated image effect of different anonymous domains.

[0084] In this embodiment, the Adam optimizer is used in the training stage. The learning rate is set to 0.0001, the β1 coefficient is set to 0.5, and the β2 coefficient is set to 0.999. The size of the batch size is 8.

[0085] Specifically, the target function of the face face latent space conversion network. The idea of CGAN is used to control the generation of images to ensure the high usability of the images. The specific operation is to input the face face x i , and the encoder E of the VAE-GAN will x iEncoding into latent space z i The decoder G restores z to the face pixel space The discriminator D discriminates x i And Guarantee image quality. In mathematical form can be expressed as:

[0086]

[0087] Where z represents the face facial latent code representation, x represents the original image data, G represents the generator, and D represents the discriminator.

[0088] Face image style loss objective function. Use the pre-trained VGG16 network model to calculate the style loss between the generated face image and the original image. By obtaining the feature map of the image through the VGG16 model specific layer, use the Gram matrix to calculate the L1 distance difference between each layer of the generated image and the original image, the purpose is to keep the style of the original image and the generated image consistent. The loss function and the loss function of the generator are jointly back propagated. In mathematical form can be expressed as:

[0089]

[0090] Where x represents the original image, x t represents the target anonymous image.

[0091] Feature matching loss function. In order to ensure the stability of network training and produce natural and clear images, feature matching loss is adopted. Specifically, this loss is based on ArcFace discriminator, extract features from ArcFace, calculate the feature loss between the generated face image and the original image, then the feature matching loss is:

[0092]

[0093] Where x is the original image, s is the anonymous domain encoding, and E is the encoder.

[0094] Latent space feature loss function. In order to ensure that the generated image conforms to the corresponding anonymous domain, the latent code of the original image after conversion and the latent code of the generated image should be consistent with the latent code of the target image. Realize the unity of anonymous domain, adopt latent space feature matching loss. Then the latent space feature matching loss is:

[0095]

[0096] Where x is the original image, x t is the target anonymous image, s is the anonymous domain encoding, e is the encoder, A is the anonymous method analysis module, and G is the decoding generator.

[0097] Mean Absolute Error Loss Function. In order to ensure the similarity between the generated results of the network and the target image, the mean absolute error loss function (MAE) is introduced. The loss is:

[0098]

[0099] where x is the original image, x t is the target anonymous image.

[0100] Step 6, training and testing using public data sets, output the final result.

[0101] 6-1. Select a suitable data set, in this embodiment, the public data set CelebA-HQ is selected: a high-definition face recognition data, containing 30,000 pictures with a resolution of 1024*1024, 6216 identities, each picture is marked with 16 semantic representations, and then the data set is preprocessed according to step 1.

[0102] 6-2. First, train the face facial latent space conversion network: use the CelebA-HQ data set for training to realize the functions of face facial latent space encoding and decoding.

[0103] 6-3. Train the latent space feature conversion module, each original image has multiple reference images of anonymous methods, in the latent space, use the method of knowledge distillation to adaptively train the anonymous method selection module and the anonymous method analysis module. Use the loss function in step 5 to iteratively optimize the model.

[0104] 6-4. In order to verify the efficiency of the proposed method, compare it with the current traditional and excellent anonymous methods based on deep learning, in this embodiment, use CIAGAN, RIDDLE, Mosaict, etc. to compare the anonymity rate and image quality. Two methods are used to determine the anonymity rate, the first method is to use the best face recognizer to classify the image and determine whether the original identity is recognized. The second method is to use FaceNet and ArcFace models for detection, respectively input the anonymous image and the original image into the FaceNet, ArcFace model network to calculate the difference value between them, when greater than the set threshold, it is anonymous success, otherwise it fails. In terms of image quality, FID, PSNR, SSIM, etc. are used to detect the generation quality of the anonymous image.

[0105] As Figure 3 shown, in this embodiment, the generated image using the anonymous unified framework is shown, which effectively balances anonymity and usability.

[0106] The experimental results of the parameter quantity of each module of this embodiment are shown in Table 1.

[0107] Table 1 Parameter quantity of each module of the method

[0108] Parameter amount Rounding (M) Encoder 21386048 21.3M Decoder 22081347 22.0M Mapping network 1617472 1.6M Latent space plug-in 9443328 9.4M Sum 59249859 54.2M

[0109] Comparative Example:

[0110] 1. The image generation quality of the present method was tested and compared with the methods of blurring, mosaic, CIAGAN, Cao and RIDDLE respectively. The specific data results are shown in Table 2.

[0111] Table 2 Comparison of structural similarity and other picture quality between the present method and other methods

[0112] FID LPIPS SSIM MSE PSNR Cao 14.44 0.07958 0.8245 246.17 24.53 Cao (this experiment) 33.15 0.165 0.644 417.83 22.29 Riddle 73.06 0.28924 0.4726 1303.92 17.22 Riddle (this experiment) 112.79 0.309 0.495 1125.54 17.84 CIAGAN 63.34 0.439 0.4034 2162.58 15.632 CIAGAN (this experiment) 115.52 0.424 0.4272 1746.30 16.01 Mosaic 162.55 0.44807 0.6311 1328.93 17.28 Mosaic (this experiment) 383.30 0.4494 0.490 1176.05 17.63 Blur 138.4 0.32681 0.7805 249.79 24.68 Blur (this experiment) 129.06 0.381 0.585 541.201 21.06

[0113] The ArcFace test anonymity rate comparison experiment results of the present embodiment and other methods are shown in Table 3.

[0114] Table 3 Comparison of ArcFace test anonymity rate between the present method and other methods

[0115] ArcFace (0.5) ArcFace (0.6) ArcFace (0.7) Cao 0％ 0％ 0％ Cao (this experiment) 5.0％ 0％ 0％ Riddle 0％ 0％ 0％ Riddle (this experiment) 0％ 0％ 0％ CIAGAN 1.28％ 1.28％ 1.28％ CIAGAN (this experiment) 0％ 0％ 0％ Mosaic 0.3％ 0.15％ 0.1％ Mosaic (this experiment) 0％ 0％ 0％ Blur 9.2％ 2.9％ 2.4％ Blur (this experiment) 0.15％ 0％ 0％

[0116] The FaceNet test anonymity rate comparison experiment results of the present embodiment and other methods are shown in Table 4.

[0117] Table 4 Comparison of FaceNet test anonymity rate between the present method and other methods

[0118] FaceNet 1.2 1.3 1.4 1.5 1.6 Cao 99.3％ 95.4％ 80.7％ 47.5％ 14.6％ Cao (this experiment) 2.7％ 0.5％ 0.07％ 0.02％ 0％ Riddle 87.8％ 67.3％ 33.1％ 6.3％ 0.32％ Riddle (this experiment) 82.4％ 54.4％ 19.7％ 2.7％ 0.03％ CIAGAN 56.92％ 29.94％ 9.27％ 1.15％ 0.05％ CIAGAN (this experiment) 44.58％ 18.63％ 4.92％ 0.68％ 0.03％ Mosaic 17.2％ 4.2％ 0.55％ 0.02％ 0％ Mosaic (this experiment) 55.7％ 27.8％ 8.79％ 1.32％ 0.03％ Blur 10.69％ 2.89％ 0.4％ 0.07％ 0％ Blur (this experiment) 17.8％ 4.42％ 0.58％ 0.02％ 0％

[0119] The embodiments of the present application are described in detail above with reference to the drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments including components can be made without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.

Claims

1. A face anonymization unified framework method based on knowledge distillation, characterized in that, The method comprises the following steps: Step 1, obtaining a face image dataset; Step 2, constructing a face facial latent space conversion network, taking the face image data as input to obtain a face latent code and a face pixel space through the face facial latent space conversion network; The face facial latent space conversion network comprises an encoder, a decoder and a discriminator, Step 3, constructing a latent space feature conversion module, and taking the face latent code as input to obtain a domain conversion code in the latent code space through the latent space feature conversion module; The latent space feature conversion module comprises an anonymous method selection module and an anonymous method analysis module, the anonymous method selection module is composed of a plurality of shared full connection layers FC and ReLu layers, and finally connects a multi-branch full connection layer; the anonymous method analysis module is composed of a plurality of ResNet blocks stacked; The input of the anonymous method selection module S is 16-dimensional Gaussian distribution acquisition data Latent code, and the output of each branch corresponds to the domain code s of an anonymous method i ; The input of the anonymous method analysis module A includes two parts: one part is the face latent code z i , and the other part is the selected anonymous method domain code s i . The output is a domain conversion code z′ i of the same dimension as the face latent code, and the expression is as follows: s i = S(n) z' i = A(z i |s i ); Step 4, generating an anonymous face; Step 5, constructing an anonymous face generation target function; Step 6, training and testing using a public dataset, and outputting a final result.

2. The knowledge distillation-based face anonymization uniform framework method according to claim 1, characterized in that, In step 1, the obtained face image dataset is an image dataset with clear face and balanced male and female ratio.

3. The knowledge distillation-based face anonymization uniform framework method of claim 1, wherein, In step 2, encoder E processes the input face image x. i Mapping into the latent space Z, we obtain the face latent code z. i Decoder G is based on the face latent code z i Restore to the corresponding face pixel space Discriminator D assists VAE in improving image realism, as shown in the following expression: z i = E(x i ) 4. The knowledge distillation-based face anonymization uniform framework method according to claim 3, characterized in that, In step 3, the face latent code z i As the input of the anonymous method analysis module A, the intermediate output is θ i s i acted on the anonymous method analysis module A by the AdaIN method, and its expression is:

5. The knowledge distillation-based face anonymization uniform framework method according to claim 4, characterized in that, In step 6, the G module of the face latent space transformation network is used to transform z′ i Transform to the face pixel domain space to obtain the anonymous domain s i The corresponding anonymous graph x′ i Its expression is: x′ i = G(z′ i ).

6. The knowledge distillation-based face anonymization uniform framework method according to claim 1, characterized in that, The target function comprises: The target function of the face facial latent space conversion network is expressed as follows: Wherein, z represents the face facial latent code representation, x represents the original image data, G represents the generator, and D represents the discriminator; The face image style loss target function is expressed as follows: where x represents the original image, x t represents the target anonymous image; The feature matching loss function is expressed as follows: Wherein x is the original image, s is the anonymous domain code, and E is the encoder; The latent space feature loss function is expressed as follows: where x is the original image, x t is the target anonymous image, s is the anonymous domain encoding, E is the encoder, A is the anonymous method analysis module, and G is the decoding generator. The mean absolute error loss function is expressed as follows: where x is the original image, x t is the target anonymous image.

7. The knowledge distillation-based face anonymization uniform framework method according to claim 1, characterized in that, In step 6, the training method is: using Adam optimizer, learning rate is set to 0.0001, β1 coefficient is set to 0.5, β2 coefficient is set to 0.999, and the size of Batch size is 8.

Images