Structural Adaptive Facial Identity Information Protection Method Based on Identity Deactivation

Abstract

The present invention discloses a method for protecting face identity information with structure adaptation based on identity deactivation, comprising the following steps: Step 1: Data preprocessing; Step 2: Identity information deactivation; Step 3: Structure information de-identification; Step 4: Constructing an anonymous face generative adversarial network structure; Step 5: Using a public dataset for training and testing, and outputting the final result. This method enables users to generate understandable anonymous face images, which can not only deceive unaware observers, but also well preserve the usability of the data.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition and information protection, and specifically refers to a method for protecting face identity information with structure adaptation based on identity deactivation. Background Art

[0002] Modern artificial intelligence technologies make full use of face images, making our lives more convenient. However, they may arouse widespread concern in society about identity security because face images are easy to capture but not easy to change. This has led to increasingly strict restrictions on data owners for data release and sharing, which will hinder technological progress and many intelligent applications. For example, many publicly available datasets, such as Dukes MTMC, stanford Brainwash, and Meta, have been taken offline due to privacy reasons.

[0003] Anonymization is considered an effective solution, which generally has two basic requirements. One is to ensure identity security, including identity anti-recognition and anti-intrusion. The other is to retain the usability of the data, such as image quality, face detectability, pose, expression, and user-defined attributes, which may vary in different scenarios. This technology has multiple advantages, such as: (1) preventing unauthorized parties, users, and applications from collecting and using personal data; (2) preventing the recognition of the relationship between identity and factors such as location, action, event, and image, helping people avoid trouble; (3) maintaining the data usability in various applications such as data release (such as video websites), street view maps, social networks, and remote medical systems, so that even if the data is attacked or misused, there is no need to worry about information leakage.

[0004] In recent years, with the continuous development of deep learning technology, Goodfellow et al. proposed the Generative Adversarial Network (GAN) in 2014. GANs generate realistic images based on a pair of adversarial models, namely the generator and the discriminator, where the generator learns to fool the discriminator and the discriminator learns to judge whether the image is real or fake. A large number of studies have applied GAN to various visual tasks, such as style transfer, image-to-image translation, etc. The development of GAN has also brought an opportunity for the development of face anonymization. By synthesizing realistic faces, existing methods mainly focus on changing facial landmarks, attributes, or the composition of identities. Ren et al. introduced a video face anonymizer that has minimal impact on action detectors. Karla et al. established a GAN-based model to recognize a complete body. Sun et al. proposed a two-stage blurred anonymized face synthesis method. Li et al. developed an anonymization network for face de-identification through a four-stage face synthesis process: face attribute estimation, face obfuscation, image synthesis, and adversarial perturbation. Deepfake and image-to-image translation technologies can also be used to swap face identities, and they can both be regarded as de-identification in a sense. In addition, Maximov et al. proposed a CIAGAN model for face anonymization by mixing original facial landmarks and other identity IDs. However, existing methods simply deceive recognition algorithms or perform identity replacement, and their results are not satisfactory because they cannot achieve a good balance between data security and reusability to realize the recognition capabilities of machines and humans.

[0005] In summary, the field of facial identity privacy protection is a topic worthy of in-depth research. This patent intends to explore from several key points in this field to solve the difficulties and key points existing in current methods.

[0006] One of the key points of facial identity privacy protection is the effectiveness of face anonymization and the balance between it and data reusability. However, in most current methods, the effectiveness of anonymization is emphasized. Unfortunately, these methods also lose non-identity attribute information at the same time. Specifically, there are mainly the following two difficulties:

[0007] 1. How to balance the relationship between anonymity and data reusability. The reuse of data mainly targets information related to non-identity attributes. How to transfer this information in the original image to the generated image is an important factor affecting reusability.

[0008] 2. Most current methods mainly achieve anonymization by swapping faces or fusing the data of others' IDs with their own data. However, this may lead to suspicion of infringing on others' privacy. Therefore, how to comprehensively evaluate the effectiveness of the method is a difficult problem at present. Summary of the Invention

[0009] The object of the present invention is to address the deficiencies of the prior art and propose a method for protecting face identity information with structure adaptability based on identity deactivation, which enables users to generate understandable anonymous face images. This can not only deceive unaware observers but also well preserve the usability of the data.

[0010] To solve the above technical problems, the technical solution of the present invention is as follows:

[0011] A method for protecting face identity information with structure adaptability based on identity deactivation includes the following steps:

[0012] Step 1: Data preprocessing;

[0013] Step 2: Identity information deactivation;

[0014] Step 3: Structure information de-identification;

[0015] Step 4: Construct an anonymous face generation adversarial network structure;

[0016] Step 5: Use a public dataset for training and testing, and output the final result.

[0017] Preferably, for Step 1 of data preprocessing, the specific steps are as follows:

[0018] 1-1. Image correction. Correct images with distorted or rotated faces through existing methods so that the face can be kept parallel to the lower edge.

[0019] 1-2. Face recognition and cropping. Obtain the coordinates of the main part of the face and crop the face as the input for training and testing.

[0020] Preferably, for Step 2 of identity deactivation based on a pre-trained classification network, the specific steps are as follows:

[0021] 2-1. Use a pre-trained classification network (such as FaceNet) as the encoder E1, and extract the identity feature f from the fully connected (FC) layer of the classification task. The output A of the last convolutional layer of E1 is used to calculate the subsequent CAM heatmap H.

[0022] 2-2. Obtain the output of the deactivated convolutional layer through the CAM heatmap H and the gradient α obtained by the backpropagation of the network

[0023] 2-3. Calculate the deactivated facial feature through the output of the deactivated convolutional layer

[0024] Preferably, for Step 3 of removing face structure information, the specific steps are as follows: ​

[0025] 3-1. Calculate the current structure S and all the remaining candidate structures to find the distance between them. Under the same pose condition, using the size and distance of facial parts as features, calculate the feature l2 distance between the structures, denoted as d(S, X i ).

[0026] 3-2. Calculate the feature distance of the current structure S pair for each candidate structure X i . Based on the global max-min normalization result of the feature distance, use it as the utility score u of each structure for the current structure S, and its expression is:

[0027]

[0028] 3-3. For each candidate structure, through the differential privacy of the exponential mechanism, calculate the selection probability P of each candidate structure X i , and randomly select an agent structure according to the probability . The probability calculation formula is:

[0029]

[0030] 3-4. According to , adjust the oral cavity and contour markers of S, replace the thickness of the upper and lower lips with 's thickness, and adjust the contour of S to 's thickness. Then, replace the oral and contour markers of with the adjusted result of S to obtain an anonymous entrusted structure T.

[0031] Preferably, step 4 constructs an anonymous face generative adversarial network structure, and the specific steps are as follows:

[0032] 4-1. Construct a style mapping network. Through stacking five downsampling blocks of the residual structure, extract the downsampled features of the facial style, and obtain its abstract features through sum pooling (SumPooling).

[0033] 4-2. Construct a generator. Use four downsampled residual blocks as the decoupled part of the facial structure features. Then stack four upsampling blocks as the generation module that fuses the decoupled facial structure features, the facial style features through the style mapping network, and the deactivated facial identity features. The main method used in the generation is AdaIN, and its expression is:

[0034]

[0035] Among them, μ(θ i ) and σ(θ i ) respectively represent the mean and standard deviation of the features of the original image, μ(si ) and σ(s i ) represent the mean and standard deviation of the features of the original image respectively. This formula can be understood as first de-styling (subtracting its own mean and then dividing by its own standard deviation), and then styling to the style of the target image (multiplying by the standard deviation of the target image and then adding the mean).

[0036] 4-3. Construct an image authenticity discriminator. The discriminator consists of five downsampling residual blocks, one bottleneck layer block, one sum pooling layer, and one fully connected layer. It can judge whether the input picture is a real face.

[0037] Preferably, step 5 uses a public dataset for training and testing, and outputs the final result, which is specifically as follows:

[0038] 5-1. Prepare the dataset, for example, use the CelebA-HQ, VGGFACE2, and LFW datasets and perform preprocessing according to the description in step 1 to obtain the required input images.

[0039] 5-2. Obtain the deactivated representation of the facial identity features according to the description in step 2, and then use the method described in step 3 to select the structural features.

[0040] 5-3. Input the training data into the network for training and use the test data to test the model.

[0041] 5-4. To verify the efficiency of the proposed method, compare it with current excellent methods (such as NEO, CIAGAN, etc.), calculate the anonymity rate, identity exchange rate, and the anonymity effect of the image quality analysis method, and perform non-identity attribute classification on the generated images, such as gender, age, etc., and analyze the data reusability of the method.

[0042] The present invention has the following characteristics and beneficial effects:

[0043] This method modifies the facial structure of the face in the image to achieve the effect of face anonymity, which is more efficient and visually more friendly than the previous mosaic occlusion method, has higher quality in the generated images for existing deep methods, and does not require the use of attribute labels to maintain the original non-identity attributes. Different results can be flexibly generated according to user requirements. The experimental results clearly confirm the efficiency and practicality of the proposed method. The proposed method is more efficient and beautiful for the privacy protection of portrait images.

[0044] The specific manifestations have the following characteristics: (1) Naturally remove recognizable facial features; (2) Intuitively identify structural information to expand the visual difference between the original and anonymous faces; (3) Since certain attributes are identity-related attributes related to identity representation, identity-unrelated attributes can be well preserved; (4) Flexibly support face synthesis according to user needs; (5) Can output real face images. Brief Description of the Drawings

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0046] Figure 1 It is a flowchart of the network architecture for the embodiments of the present invention.

[0047] Figure 2 It is a comparison chart of the effects between the embodiments of the present invention and the prior art.

[0048] Figure 3 It is the experimental result of the comparison of the anonymity rate between this method and other methods.

[0049] Figure 4 It is the experimental result of the comparison of the picture quality retention ability between this method and other methods. Detailed Embodiments

[0050] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0051] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0052] In the description of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "installation", "connection", and "coupling" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] The present invention provides a face anonymization and privacy protection method based on a generative adversarial network, as Figure 1 shown below: It includes the following steps:

[0054] Step 1: Data preprocessing.

[0055] Step 2: Identity information deactivation;

[0056] Step 3: Structure information de-identification;

[0057] Step 4: Construct an anonymized face generative adversarial network structure;

[0058] Step 5: Use a public dataset for training and testing, and output the final result.

[0059] Specifically, for Step 1 data preprocessing, the specific steps are as follows:

[0060] 1-1. Image correction. Correct images with distorted or rotated faces so that the faces can be kept parallel to the lower edge.

[0061] 1-2. Face recognition and cropping. Obtain the coordinates of the key parts of the face and the coordinates of the face edges (rectangular area, providing the coordinates of four corners) through the face_alignment library, and crop out the face part as the input for training and testing.

[0062] Step 2 is based on a pre-trained classification network for identity deactivation, and the specific steps are as follows:

[0063] 2-1. Use a pre-trained classification network (such as FaceNet) as the encoder E1, extract the identity feature f from the fully connected (FC) layer of the classification task, and the dimension of the feature is (1, 512). The output A of the last convolutional layer of E1 is used to calculate the subsequent CAM heatmap H.

[0064] 2-2. Through the CAM heatmap H and the gradient α obtained by the backpropagation of the network, guided by the direction opposite to the gradient, obtain the identity feature that is least related to the current identity, and finally obtain the output of the deactivated convolutional layer

[0065] 2-3. Output of the convolutional layer after deactivation Re-pass through the pooling layer to obtain the identity features after deactivation

[0066] Step 3 removes the facial structure information, and the specific steps are as follows:

[0067] 3-1. Calculate the current structure S and all other candidate structures The distance between them. Under the same pose condition, taking the size and distance of facial parts as features, calculate the feature l2 distance between structures, which is expressed as d(S, X i ).

[0068] 3-2. Calculate the feature distance of the current structure S pair with each candidate structure X i Based on the global max-min normalization result of the feature distance, take it as the utility score of each structure for the current structure S, and its expression is:

[0069]

[0070] 3-3. For each candidate structure, through the differential privacy of the exponential mechanism, calculate the selection probability P of each candidate structure X i And randomly select an agent structure according to the probability The probability calculation formula is:

[0071]

[0072] 3-4. According to Adjust the oral and contour marks of S, replace the thickness of the upper and lower lips with The thickness of, and adjust the contour of S to The thickness of. Then, replace the oral and contour marks of With the adjustment result of S to obtain an anonymous proxy structure T

[0073] Step 4 constructs an anonymous face generation adversarial network structure, and the specific steps are as follows:

[0074] 4-1. Construct a style mapping network. By stacking five layers of downsampling blocks with residual structures, perform downsampling feature extraction on the facial style, and obtain its abstract features through sum pooling (SumPooling). The input size of this module is 256*256*3, and the output size is 1*512

[0075] 4-2. Build the generator. Use four downsampled residual blocks as the decoupled part of the facial structure features. Then stack four upsampling blocks as the generation module for fusing the decoupled facial structure features with the facial style features passed through the style mapping network and the deactivated facial identity features. Connect the upsampling layer blocks and the downsampling layer blocks with a bottleneck block layer in the middle. The main method used for generation is AdaIN, and its expression is:

[0076]

[0077] The input of this module is a feature with a dimension of 1*1024 (i.e., the fused feature of the facial style feature passed through the style mapping network and the deactivated facial identity feature). First, pass it through a fully connected layer to obtain a feature with a dimension of 1*13056 for use as the AdaIN feature. The output dimension is 256*256*3, which is an RGB image.

[0078] 4-3. Build the image real / fake discriminator. The discriminator consists of five downsampled residual blocks, one bottleneck layer block, one sum pooling, and one fully connected layer. It can determine whether the input image is a real face. The input dimension of the discriminator is 256*256*3, that is, a real image or a generated fake image, and its output is a scalar representing the authenticity of this image.

[0079] Step 5 Use the public dataset for training and testing, and output the final result, as follows:

[0080] 5-1. Prepare the dataset. For example, use the CelebA-HQ, VGGFACE2, and LFW datasets and perform preprocessing according to the description in Step 1 to obtain the required input images. The CelebA-HQ dataset contains 30,000 photos of 6,216 people with different identities, and 5,000 of them are used for testing. The VGGFACE2 dataset has 3.31 million images from 9,131 people, and 5,000 of them are used for testing. The LFW dataset contains 13,233 images from 5,749 people, and 5,000 of them are used for testing.

[0081] 5-2. Obtain the deactivated representation of the facial identity features according to the description in Step 2, and then select the structural features using the method described in Step 3.

[0082] 5-2. Input the training data into the network for training and use the test data to test the model. The Adam optimizer is selected for the training stage; for the division of the face and the background, use BiSeNet; for the extraction of the identity features, use the pre-trained FaceNet network.

[0083] 5-3. To verify the efficiency of the proposed method, it is compared with current excellent methods (such as NEO, CIAGAN, etc.), the anonymity rate, identity exchange rate, and the anonymity effect of the image quality analysis method are calculated, and the generated images are classified according to non-identity attributes such as gender and age, and the data reusability of the analysis method is analyzed. The specific comparison pictures are as shown in Figure 2 shown

[0084] Comparison of the experimental results of this embodiment with the prior art:

[0085] 1. The re-identification rates of this method, blurring, mosaic, DeepPrivacy, CIAGAN, and IdentityDP methods are detected respectively. The specific data results are as shown in Figure 3 shown

[0086] 2. The image quality retention rates of this method, blurring, mosaic, DeepPrivacy, CIAGAN, and IdentityDP methods are detected respectively. The specific data results are as shown in Figure 4 shown

[0087] The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, without departing from the principle and spirit of the present invention, various changes, modifications, substitutions, and variations to these embodiments including components still fall within the protection scope of the present invention.

Claims

1. A method for protecting face identity information with structure adaptation based on identity deactivation, characterized in that, It includes the following steps: S1. Image data preprocessing; S2. Perform identity deactivation based on a pre-trained classification network, and the pre-trained classification network is FaceNet; The method for performing identity deactivation through FaceNet is: S2-1. Use a pre-trained FaceNet as the encoder E1, extract the identity feature f from the fully connected layer of the classification task, and for the output A of the last convolutional layer of the encoder E1, calculate the CAM heatmap H; S2-1. Obtain the output of the deactivated convolutional layer by using the CAM heatmap H and the gradient α obtained through the backpropagation of FaceNet S2-1. Output through the deactivated convolutional layer Calculate the facial features after deactivation S3. Remove the facial structure information; 3-1. Calculate the current face structure S and the distances to all other candidate structures Under the same pose condition, calculate the feature l2 distance between face structures using the sizes and distances of facial parts as features, which is denoted as d(S, X i ); 3-2. Calculate the feature distance between the current face structure S and each candidate structure X i Based on the global maximum-minimum normalization result, use it as the utility score of each structure for the current face structure S. Its expression is: 3-3. For each candidate structure, calculate the selection probability P of each candidate structure X through the differential privacy of the exponential mechanism, and randomly select an agent structure according to the probability i The probability calculation formula is:​ 3-4. According to Adjust the oral and contour marks of S, replace the thickness of the upper and lower lips with thickness, adjust the contour of S to thickness, then Replace the oral and contour marks of with the adjusted result of S to obtain an anonymous entrusted structure T; S4. Construct an anonymous face generation adversarial network, and the anonymous face generation adversarial network includes a style mapping network, a generator, and an image authenticity discriminator; S5. Use a public dataset for training and testing, and output the final result.

2. The method for protecting face identity information with structure self - adaptation based on identity de - activation according to claim 1, wherein, The method for the image data preprocessing is: S1-1. Picture correction, by correcting pictures with distorted or rotated faces to keep the face parallel to the lower edge; S1-2. Facial recognition and cropping, by obtaining the coordinates of the main part of the face and cropping out the face as the input for training and testing.

3. The method for protecting face identity information with structure self - adaptation based on identity de - activation according to claim 1, characterized in that, The style mapping network extracts downsampled features of the facial style by stacking five downsampling blocks with residual structures, and obtains its abstract features through sum pooling.

4. A method for protecting face identity information with structure adaptation based on identity deactivation according to claim 1, characterized in that, The generator uses four downsampled residual blocks as the decoupled part of the facial structure features, and then stacks four upsampling blocks as the generation module for the fused features of the decoupled facial structure features, the facial style features passed through the style mapping network, and the deactivated facial identity features. The main method used for generation is AdaIN, and its expression is: Among them, μ(θ i ) and σ(θ i ) respectively represent the mean and standard deviation of the features of the original image, and μ(s i ) and σ(s i ) respectively represent the mean and standard deviation of the features of the original image.

5. The method for protecting face identity information with structure adaptation based on identity deactivation according to claim 1, wherein The image authenticity discriminator is composed of five downsampled residual blocks, one bottleneck layer block, one sum pooling layer, and one fully connected layer, and is used to judge whether the input picture is a real face.

6. The method for protecting face identity information with structure adaptation based on identity deactivation according to claim 2, wherein The method for the step S5 is: S5-1. Prepare the dataset and obtain the required input images through preprocessing according to step S1; S5-2. Obtain the deactivated representation of the facial identity feature according to the description in step S2, and then select the structural features using the method described in step S3; S5-3. Input the training data into the network for training and use the test data to test the model; S5-4. Verify the efficiency of the proposed method, calculate the anonymity rate, identity exchange rate, and the anonymity effect of the image quality analysis method, and perform non-identity attribute classification on the generated images.

Images