Human face anonymous unified framework method based on knowledge distillation

Abstract

The invention provides a face anonymous unified framework method based on knowledge distillation. The method comprises the following steps: step 1, acquiring a face image data set; step 2, constructing a human face hidden space conversion network, taking the human face image data as input, and obtaining a human face hidden code and a human face pixel space through the human face hidden space conversion network; 3, constructing a latent space feature conversion module, and obtaining a domain conversion code in a latent code space through the latent space feature conversion module by taking the face latent code as input; 4, generating an anonymous face of the human face; 5, constructing an anonymous face generation objective function; and step 6, training and testing by using the public data set, and outputting a final result. The framework fusion realizes traditional multi-domain conversion based on deep learning and other anonymous methods, and flexibly protects identity information.

Description

Technical Field

[0001] The present invention relates to the technical field of face image processing, and in particular to a face anonymity unified framework method based on knowledge distillation. Background Art

[0002] Due to the continuous development of scientific and technological information, the explosive growth of facial images in social media, and the widespread application of facial recognition technology in various fields, the security of personal identity privacy information is facing unprecedented challenges. In order to solve the problem of image privacy protection, some countries and regions have promulgated corresponding laws and regulations to protect the privacy of the public, and even recovered some facial data sets. Although these methods have a certain degree of effect on protecting the privacy of facial images, these methods cannot fundamentally solve the public's privacy problems, and the implementation of these measures will increase the difficulty of research for scientific researchers. Face de-identification technology is considered to be an effective visual privacy protection tool. What it does is to hide or replace the identity information contained in the image.

[0003] Computer vision technology has been widely used in tasks such as visual recognition, which has brought great convenience to people's daily lives, but also brought huge risks. Many private photos and videos are uploaded to the cloud or sent to third parties for analysis and recognition tasks, and facial information is included in them. However, facial information is sensitive information that contains personal information. If not carefully protected, highly sensitive facial information can be easily accessed and illegally used by third parties or malicious attackers. Therefore, when performing routine computer vision tasks, it is necessary to protect the privacy of sensitive information, and a technology that can ensure the security of facial information is needed. In traditional face anonymization algorithms, face privacy protection methods are all achieved by irreversibly processing the original face. Traditional face anonymization methods mainly include: blur processing, pixelation processing, image filtering, etc. These methods have the advantages of simple implementation and low computational complexity, but they often cannot well preserve the visual information of the image while protecting privacy.

[0004] At present, there are a lot of works on the market to achieve privacy and anonymity protection of faces. Among them, there are many different methods, including traditional blurring and pixelation processing; face protection based on face swapping and face image anonymity methods based on deep learning. However, the use and effects of different methods vary greatly. For example, traditional blurring and pixelation processing methods usually only require simple image processing algorithms, such as blur filters or pixelation operations. Deep learning-based methods may require a large amount of labeled data and training time to build an accurate face feature extraction model. In addition, some methods may require additional auxiliary information, such as posture estimation, face key point detection, or face recognition models to provide more sophisticated privacy protection. In addition, the size of the model is also related to computational efficiency and real-time performance. Larger models may require longer inference time and more computing resources, and are not suitable for real-time applications.

[0005] In view of the above problems, there are differences in auxiliary conditions and model size between different anonymity methods, which makes it complicated to select and use anonymous methods in practical applications. The significance of the research on a unified anonymous framework is to solve the complexity between different anonymous methods and promote comparison, integration and innovation between methods.

[0006] In summary, by establishing a unified framework, we can better select and use anonymous methods, improve the effectiveness of privacy protection, and promote the development of the entire field, which is of great value and research significance.

[0007] Traditional and deep learning-based anonymization methods have limitations. For traditional methods, the anonymization effect is significant but the usability is poor; for deep learning methods, the usability has been improved, but the consumption of computing resources and time has increased. Moreover, with the continuous advancement of technology, new anonymization methods have gradually emerged, and the selection of anonymization methods is also difficult. Specifically, there are two main difficulties:

[0008] 1. How to integrate anonymity methods based on traditional and deep learning methods, and whether anonymity and data reusability are balanced.

[0009] Second, how to deal with the new anonymous methods that are constantly emerging and whether they can be integrated into the existing unified anonymous framework. Summary of the invention

[0010] In view of the shortcomings of the existing technology, the present invention proposes a unified framework method for face anonymity based on knowledge distillation. The framework integrates and realizes multi-domain conversion of traditional anonymity methods such as those based on deep learning, and flexibly protects identity information.

[0011] In order to solve the above technical problems, the technical solution of the present invention is:

[0012] A face anonymization unified framework method based on knowledge distillation includes the following steps:

[0013] Step 1: Obtain a face image dataset;

[0014] Step 2: construct a face latent space conversion network, take the face image data as input and pass it through the face latent space conversion network to obtain the face latent code and face pixel space;

[0015] The face latent space conversion network includes an encoder, a decoder and a discriminator.

[0016] Step 3: construct a latent space feature conversion module, and use the face latent code as input to obtain a domain conversion code in the latent code space through the latent space feature conversion module;

[0017] The latent space feature conversion module includes an anonymous method selection module and an anonymous method analysis module. The anonymous method selection module is composed of multiple shared fully connected layers FC and ReLu layers, and finally connected to a multi-branch fully connected layer; the anonymous method analysis module is composed of multiple ResNet blocks stacked;

[0018] Step 4: Generate anonymous face;

[0019] Step 5: Construct an anonymous face generation objective function;

[0020] Step 6: Use the public data set for training and testing, and output the final results.

[0021] Preferably, in step 1, the facial image data set obtained is image data with clear faces and a balanced ratio of men to women.

[0022] Preferably, in step 2, the encoder E processes the input face image x i Mapped into the latent space Z, we get the face latent code z i , the decoder G is based on the face latent code z i Restore to the corresponding face pixel space The discriminator D assists VAE to improve the authenticity of the image. The expression is as follows:

[0023] z i =E(x i )

[0024]

[0025] Preferably, the anonymous method selection module S input is a 16-dimensional Gaussian distribution collection data Latentcode, and the output of each branch corresponds to a domain code s of an anonymous method. i .

[0026] Preferably, the input of the anonymous method analysis module A includes two parts: one part is the face latent code z i , the other part is the selected anonymous method domain code s i , output the domain conversion code z' with the same dimension as the face latent code i , the expression is as follows:

[0027] s i =S(n)

[0028] z′ i =A(z i |s i ).

[0029] Preferably, in step 3, the face latent code z i As the input of anonymous method analysis module A, the intermediate output is θ i ,s i The expression is:

[0030]

[0031] Preferably, in step 6, z' is converted into i Convert to face pixel domain space and get anonymous domain s i The corresponding anonymous graph x' i , whose expression is:

[0032] x′ i =G(z′ i ).

[0033] Preferably, the objective function comprises:

[0034] The objective function of the face latent space conversion network is expressed as follows:

[0035]

[0036] Among them, z represents the latent code representation of the face, x represents the original image data, G represents the generator, and D represents the discriminator;

[0037] The objective function of face image style loss is expressed as follows:

[0038]

[0039] Among them, x represents the original image, x t represents the target anonymous image;

[0040] The feature matching loss function is expressed as follows:

[0041]

[0042] Where x is the original image, s is the anonymous domain encoding, and E is the encoder;

[0043] The latent space feature loss function is expressed as follows:

[0044]

[0045] Where x is the original image, x t is the target anonymous image, s is the anonymous domain encoding, E is the encoder, A is the anonymous method analysis module, and G is the decoding generator;

[0046] The mean absolute error loss function is expressed as follows:

[0047]

[0048] Where x is the original image, x t Anonymize the target image.

[0049] Preferably, in step 6, the training method is: using the Adam optimizer, setting the learning rate to 0.0001, the β1 coefficient to 0.5, the β2 coefficient to 0.999, and the batch size to 8.

[0050] The present invention has the following characteristics and beneficial effects:

[0051] This paper designs a new face anonymization unified framework based on knowledge distillation. It realizes the integration of traditional and deep learning-based anonymization methods, and can switch different anonymization methods according to different scenarios, which balances usability and effectiveness. In addition, the framework also supports expansion to cope with newly generated face anonymization methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 It is a flow chart of the facial latent space conversion network of the present invention.

[0054] Figure 2 It is a flow chart of the overall architecture of the present invention.

[0055] Figure 3 This is a comparison chart of the anonymous graph generation effect of the present invention. DETAILED DESCRIPTION

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0057] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0058] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0059] The present invention provides a unified framework method for face anonymity based on knowledge distillation, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0060] Step 1: Obtain a face image dataset.

[0061] In this embodiment, the selected dataset is the CelebA-HQ dataset, in which the face images must be clear, with a balanced ratio of male to female, and 22,000 male and female images. The images are normalized to a uniform size of 256*256.

[0062] Step 2: Build a facial latent space conversion network, encode the data set, perform compression and dimensionality reduction processing, reduce parameter calculations, and use the discriminator to assist in decoding into the face pixel space to ensure image quality.

[0063] Among them, the face latent space conversion network is composed of an encoder E, a decoder G and a discriminator D. The face image data is taken as input and the face latent code and face pixel space are obtained through the face latent space conversion network.

[0064] Specifically, the encoder takes the input face x i Mapped into the latent space Z, we get the face latent code z i , the decoder G restores the corresponding face pixel space according to the latent code The discriminator D assists VAE to improve the authenticity of the image. Its mathematical form is expressed as:

[0065] z i =E(x i )

[0066]

[0067] The network encodes the data set, performs compression and dimensionality reduction processing, reduces parameter calculations, and the discriminator assists in decoding into the face pixel space to ensure image quality.

[0068] Step 3: construct a latent space feature conversion module, and use the face latent code as input to obtain the domain conversion code in the latent code space through the latent space feature conversion module.

[0069] Specifically, in this embodiment, the latent space feature conversion module includes two parts: an anonymous method selection module S and an anonymous method analysis module A.

[0070] S consists of multiple shared fully connected layers FC and ReLu layers, and finally connects to a multi-branch fully connected layer, corresponding to multiple anonymous methods. Its input is a 16-dimensional Gaussian distribution collection data Latent code (n), and the output of each branch corresponds to a domain code s of an anonymous method i .

[0071] In this embodiment, the module maps the 16-dimensional data collected in the Gaussian distribution into a 512-dimensional vector for each anonymous method. The middle hidden layer is a 1024-dimensional fully connected neural network.

[0072] Anonymous method analysis module A module consists of multiple ResNet blocks stacked together and receives two parts of input. One part of the input is the face latent code representation z i , the other part is the selected anonymous method domain code s i , the output is the same as z i Domain conversion code z' of the same dimension i , the expression is as follows:

[0073] s i =S(n)

[0074] z′ i=A(z i |s i ).

[0075] In this embodiment, the module is composed of a plurality of stacked ResNet blocks, the input is the face latent code, and receives the 512-dimensional anonymous method code output by the anonymous method selection module, and acts on the ResNet block in the AdaIN mode to realize the conversion of different anonymous domain latent spaces.

[0076] Face image latent code z i As the input of module A, the intermediate output is θ i ,s i Acting on A through AdaIN, the expression is:

[0077]

[0078] Step 4: Generate anonymous face.

[0079] Through the G module of the face latent space conversion network, z' i Convert to face pixel domain space and get anonymous domain s i The corresponding anonymous graph x' i , whose expression is:

[0080] x′ i =G(z′ i ).

[0081] It can be understood that the face latent code has a dimension of 8*8. After the selection and analysis module of the anonymous method, an 8*8 latent space code corresponding to the anonymous domain is generated. After the decoder of the face latent space conversion network, a 256*256, 3-channel face pixel space image is generated.

[0082] Step 5: Construct an anonymous face generation objective function;

[0083] The total loss includes adversarial loss, style loss, feature matching loss function, latent space feature loss function and mean absolute error loss function, which ensures the stability of conversion and generated image effects in different anonymous domains.

[0084] In this embodiment, the Adam optimizer is used in the training phase. The learning rate is set to 0.0001, the β1 coefficient is set to 0.5, and the β2 coefficient is set to 0.999. The batch size is 8.

[0085] Specifically, the objective function of the face latent space conversion network is to use the idea of ​​CGAN to control the generation of images to ensure the high quality of the image. The specific operation is to input the face x i , the encoder E of VAE-GAN converts x iEncoded into the latent space z i , the decoder G restores z to the face pixel space Discriminator D discriminates x i and Ensure image quality. It can be expressed in mathematical form as:

[0086]

[0087] Among them, z represents the latent code representation of the face, x represents the original image data, G represents the generator, and D represents the discriminator.

[0088] The objective function of face image style loss. Use the pre-trained VGG16 network model to calculate the style loss between the generated face image and the original image. By obtaining the feature map of the image through a specific layer of the VGG16 model, the Gram matrix is ​​used to calculate the L1 distance difference between the generated map of each layer and the original image, in order to keep the style of the original image consistent with that of the generated image. This loss function can be combined with the loss function of the generator for back propagation. It can be expressed mathematically as:

[0089]

[0090] Among them, x represents the original image, x t Represents the target anonymous image.

[0091] Feature matching loss function. In order to ensure the stability of network training and produce natural and clear images, feature matching loss is used. Specifically, this loss is based on the ArcFace discriminator, extracts features from ArcFace, and calculates the feature loss between the generated face image and the original image. The feature matching loss is:

[0092]

[0093] Where x is the original image, s is the anonymous domain encoding, and E is the encoder.

[0094] Latent space feature loss function. In order to ensure that the generated graph conforms to the corresponding anonymous domain, the latent code of the original graph after conversion and the latent code of the generated graph should be consistent with the latent code of the target graph. To achieve the unification of the anonymous domain, the latent space feature matching loss is used. , then the latent space feature matching loss is:

[0095]

[0096] Where x is the original image, x t is the target anonymous image, s is the anonymous domain encoding, e is the encoder, A is the anonymous method analysis module, and G is the decoding generator.

[0097] Mean absolute error loss function. In order to ensure the similarity between the generated result of the network and the target graph, the mean absolute error loss function (MAE) is introduced. Its loss is:

[0098]

[0099] Where x is the original image, x t Anonymize the target image.

[0100] Step 6: Use the public data set for training and testing, and output the final results.

[0101] 6-1. Select a suitable dataset. In this embodiment, the public dataset CelebA-HQ is selected: a high-definition face recognition data containing 30,000 images with a resolution of 1024*1024, 6216 identities, and each image is marked with 16 semantic representations. Then, the dataset is preprocessed as described in step 1.

[0102] 6-2. First, train the facial latent space conversion network: use the CelebA-HQ dataset to train and realize the functions of facial latent space encoding and decoding.

[0103] 6-3. Train the latent space feature conversion module. Each original image has multiple reference images of anonymous methods. In the latent space, the anonymous method selection module and the anonymous method analysis module are adaptively trained using the knowledge distillation method. The loss function in step 5 is used to iteratively optimize the model.

[0104] 6-4. In order to verify the efficiency of the proposed method, it is compared with the current traditional and excellent methods based on deep learning anonymity. In this embodiment, CIAGAN, RIDDLE, Mosaict and other methods are used to compare their anonymity rate and image quality. Two methods are used for the anonymity rate. The first method is to use the best face recognizer currently for image classification to determine whether the anonymous image is recognized by the original identity. The second method is to use FaceNet and ArcFace models for detection. The anonymous image and the original image are input into the FaceNet and ArcFace model networks respectively to calculate the difference between them. When it is greater than the set threshold, the anonymity is successful, otherwise it fails. In terms of picture quality, FID, PSNR, SSIM, etc. are used to detect the generation quality of anonymous images.

[0105] like Figure 3 As shown, in this embodiment, a generated graph using an anonymous unified framework is shown, which effectively balances anonymity and usability.

[0106] The experimental results of the parameters of each module in this embodiment are shown in Table 1.

[0107] Table 1 Parameters of each module in this method

[0108] Parameter quantity Round(M) Encoder 21386048 21.3M Decoder 22081347 22.0M Mapping the network 1617472 1.6M Latent Space Plugin 9443328 9.4M sum 59249859 54.2M

[0109] Comparative Example:

[0110] 1. The image generation quality of this method was compared with that of blurring, mosaic, CIAGAN, Cao, and RIDDLE. See Table 2 for details of the specific data results.

[0111] Table 2 Comparative experiment on the image quality of this method and other methods in terms of structural similarity

[0112] FID LPIPS SSIM MSE PSNR Cao 14.44 0.07958 0.8245 246.17 24.53 Cao (this experiment) 33.15 0.165 0.644 417.83 22.29 Riddle 73.06 0.28924 0.4726 1303.92 17.22 Riddle (this experiment) 112.79 0.309 0.495 1125.54 17.84 CIAGAN 63.34 0.439 0.4034 2162.58 15.632 CIAGAN (this experiment) 115.52 0.424 0.4272 1746.30 16.01 Mosaic 162.55 0.44807 0.6311 1328.93 17.28 Mosaic (this experiment) 383.30 0.4494 0.490 1176.05 17.63 Vague 138.4 0.32681 0.7805 249.79 24.68 Blur (this experiment) 129.06 0.381 0.585 541.201 21.06

[0113] The experimental results of the ArcFace test anonymity rate comparison between this embodiment and other methods are shown in Table 3.

[0114] Table 3 Comparison experimental results of ArcFace test anonymity rate of this method and other methods

[0115] ArcFace(0.5) ArcFace(0.6) ArcFace(0.7) Cao 0％ 0％ 0％ Cao (this experiment) 5.0％ 0％ 0％ Riddle 0％ 0％ 0％ Riddle (this experiment) 0％ 0％ 0％ CIAGAN 1.28％ 1.28％ 1.28％ CIAGAN (this experiment) 0％ 0％ 0％ Mosaic 0.3％ 0.15％ 0.1％ Mosaic (this experiment) 0％ 0％ 0％ Vague 9.2％ 2.9％ 2.4％ Blur (this experiment) 0.15％ 0％ 0％

[0116] The experimental results of the FaceNet test anonymity rate comparison between this embodiment and other methods are detailed in Table 4.

[0117] Table 4 Comparison experimental results of FaceNet test anonymity rate of this method and other methods

[0118] FaceNet 1.2 1.3 1.4 1.5 1.6 Cao 99.3％ 95.4％ 80.7％ 47.5％ 14.6％ Cao (this experiment) 2.7％ 0.5％ 0.07％ 0.02％ 0％ Riddle 87.8％ 67.3％ 33.1％ 6.3％ 0.32％ Riddle (this experiment) 82.4％ 54.4％ 19.7％ 2.7％ 0.03％ CIAGAN 56.92％ 29.94％ 9.27％ 1.15％ 0.05％ CIAGAN (this experiment) 44.58％ 18.63％ 4.92％ 0.68％ 0.03％ Mosaic 17.2％ 4.2％ 0.55％ 0.02％ 0％ Mosaic (this experiment) 55.7％ 27.8％ 8.79％ 1.32％ 0.03％ Vague 10.69％ 2.89％ 0.4％ 0.07％ 0％ Blur (this experiment) 17.8％ 4.42％ 0.58％ 0.02％ 0％

[0119] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A face anonymization unified framework method based on knowledge distillation, characterized in that: The steps include: Step 1: Obtain a face image dataset; Step 2: construct a face latent space conversion network, take the face image data as input and pass it through the face latent space conversion network to obtain the face latent code and face pixel space; The face latent space conversion network includes an encoder, a decoder and a discriminator. Step 3: construct a latent space feature conversion module, and use the face latent code as input to obtain a domain conversion code in the latent code space through the latent space feature conversion module; The latent space feature conversion module includes an anonymous method selection module and an anonymous method analysis module. The anonymous method selection module is composed of multiple shared fully connected layers FC and ReLu layers, and finally connected to a multi-branch fully connected layer; the anonymous method analysis module is composed of multiple ResNet blocks stacked; Step 4: Generate anonymous face; Step 5: Construct an anonymous face generation objective function; Step 6: Use the public data set for training and testing, and output the final results.

2. According to the method of face anonymity unified framework based on knowledge distillation in claim 1, it is characterized in that: In step 1, the acquired face image data set is image data with clear faces and a balanced ratio of men to women.

3. According to the method of face anonymity unified framework based on knowledge distillation in claim 1, it is characterized in that: In step 2, the encoder E processes the input face image x i Mapped into the latent space Z, we get the face latent code z i , the decoder G is based on the face latent code z i Restore to the corresponding face pixel space The discriminator D assists VAE to improve the authenticity of the image. The expression is as follows: z i =E(x i ) 4. According to claim 3, a face anonymity unified framework method based on knowledge distillation is characterized in that: The input of the anonymous method selection module S is a 16-dimensional Gaussian distribution collection data latent code, and the output of each branch corresponds to the domain code s of an anonymous method i .

5. According to the method of face anonymity unified framework based on knowledge distillation in claim 4, it is characterized in that: The input of the anonymous method analysis module A includes two parts: one part is the face latent code z i , the other part is the selected anonymous method domain code s i , output the domain conversion code z' with the same dimension as the face latent code i , the expression is as follows: s i =S(n) from' i =A(from i |with i )。 6. The method for face anonymization unified framework based on knowledge distillation according to claim 5, characterized in that: In step 3, the face latent code z i As the input of anonymous method analysis module A, the intermediate output is θ i ,s i The expression is:

7. The method for face anonymization unified framework based on knowledge distillation according to claim 6, characterized in that: In step 6, z' is transformed through the G module of the face latent space conversion network i Convert to face pixel domain space and get anonymous domain s i The corresponding anonymous graph x' i , whose expression is: x’ i =G(z’ i )。 8. The method for face anonymity unified framework based on knowledge distillation according to claim 1, characterized in that: The objective function includes: The objective function of the face latent space conversion network is expressed as follows: Among them, z represents the latent code representation of the face, x represents the original image data, G represents the generator, and D represents the discriminator; The objective function of face image style loss is expressed as follows: Among them, x represents the original image, x t represents the target anonymous image; The feature matching loss function is expressed as follows: Where x is the original image, s is the anonymous domain encoding, and E is the encoder; The latent space feature loss function is expressed as follows: Where x is the original image, x t is the target anonymous image, s is the anonymous domain encoding, E is the encoder, A is the anonymous method analysis module, and G is the decoding generator; The mean absolute error loss function is expressed as follows: Where x is the original image, x t Anonymize the target image.

9. The method for face anonymization unified framework based on knowledge distillation according to claim 1, characterized in that: In step 6, the training method is: using the Adam optimizer, the learning rate is set to 0.0001, the β1 coefficient is set to 0.5, the β2 coefficient is set to 0.999, and the batch size is 8.

Images