Face identity privacy protection method based on local modification

Abstract

The application provides a face identity privacy protection method based on local modification, comprising the following steps: firstly, data sets are acquired and preprocessed; face organs in the preprocessed image are segmented; in step 3, a local anonymizer is used, and each selected face organ is replaced with an anonymous organ based on an anonymous strategy, and the coordinates of the anonymous face organ are replaced; in step 4, a generation model is constructed and trained, wherein the generation model comprises a local region encoder, a face encoder and a generator; in step 5, the trained generation model is applied to image reconstruction: the anonymous organs of eyes, noses, mouths and skins are taken as the input of the local region encoder, the anonymous organs of the face organ coordinates are taken as the input of the face encoder, and finally, the reconstructed image is generated through the generator. The method can not only achieve the purpose of deceiving an unaware observer, but also protect the availability of data.

Description

Technical Field

[0001] The present invention relates to the field of face recognition and information protection technology, and in particular to a face identity privacy protection method based on local modification. Background Art

[0002] We are currently living in an era of intelligence and informatization. The rapid development of computer technology has driven the widespread emergence of smart devices and applications, providing people with personalized and diversified services. Everyday social networking platforms are flooded with easily recognizable facial images. This data mostly describes individual behavioral characteristics in specific scenarios. As a highly personalized information carrier, the human face is an important identifier that distinguishes different individuals and is therefore considered sensitive information.

[0003] As the accuracy of facial recognition technology improves, this information can identify individuals and their various attributes in real time. This poses a significant threat to personal privacy and security, necessitating an urgent need to protect unauthorized facial image data. To address the privacy concerns associated with the massive volume of images, some countries have taken measures, such as enacting laws and regulations, to prevent the unauthorized use of images containing private information. While these laws and regulations somewhat restrict data users from obtaining authorization, thereby protecting privacy, they also increase the difficulty of data acquisition and significantly limit the development of related scientific research.

[0004] In recent years, many interesting traditional and generative anonymization methods have been proposed. Traditional methods, such as occlusion, mosaic, and blurring, are often used to protect personal privacy because they are simple to implement and low-cost. However, these methods provide protection by damaging a large amount of image content and visual effects, resulting in a significant decrease in data availability. With the continuous development of computer technology hardware and software, generative anonymization methods have gradually become the dominant anonymization methods. Tao Li et al. proposed AnonymousNet, which mainly synthesizes anonymous images by modifying non-identifying attribute information in the original image, and then adds noise to the synthesized image to make it undetectable by the recognition system, thereby improving privacy protection. et al. proposed the DeepPrivacy method in 2019, which achieved an anonymous scenario that can be applied to multiple people in a picture, and the method can also be extended to videos. The method is to pre-mask the face with random noise and use the target detection Mask R-CNN method to locate the position of the face and organs, and then use CGAN to generate anonymous images. CIAGAN researchers use a popular carrier of facial landmarks and identities to remove people's identifying features, while still retaining the necessary features to allow face and body detectors to work. This type of method requires borrowing the identity of others as an operation guide, which may affect the privacy and security of the identity provider. Secondly, the generated image will only be good when the fake ID provider is the same gender, similar age, and similar makeup as the original person. It is very inconvenient to use and does not maintain some special attributes, such as glasses, heavy makeup, and beards.

[0005] In summary, the field of facial identity privacy protection is a topic worthy of in-depth research. This patent intends to explore several coordinate points in this field to solve the difficulties and key points of current methods.

[0006] A key aspect of facial identity privacy protection lies in the balance between the effectiveness of facial anonymity and data reusability. However, most current approaches prioritize the effectiveness of anonymity globally, but unfortunately, these methods also lose other potentially retainable attribute information. Specifically, there are two main challenges:

[0007] 1. Few methods consider the feasibility of anonymity from a local perspective. Existing methods all consider applying anonymity to faces from a global perspective, while few consider modifying local regions to achieve anonymity, preserving information that can be left unchanged.

[0008] 2. How to balance the cost of anonymity and the degree of anonymity protection. Pursuing improved anonymity under limited resource conditions to meet the high anonymity requirements in different scenarios is also an important direction for promoting the development of privacy protection. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of the existing technology and propose a facial identity privacy protection method based on local modification. Users can freely select the local location that needs to be changed and generate personalized anonymous images, which can not only deceive uninformed observers but also protect the availability of data.

[0010] In order to solve the above technical problems, the technical solution of the present invention is:

[0011] A method for protecting face identity privacy based on local modification includes the following steps:

[0012] Step 1: Dataset preprocessing;

[0013] Step 2: facial organ segmentation;

[0014] Step 3: Local anonymous replacement;

[0015] Step 4: Generate model construction;

[0016] Step 5: Model training and testing.

[0017] Preferably, the specific steps of step 1 are as follows:

[0018] 1-1. Image Correction: Use existing methods to correct images with distorted or rotated faces so that the face remains parallel to the image border.

[0019] 1-2. Facial recognition and cropping. Obtain the coordinates of the main part of the face and crop the face for subsequent use.

[0020] Preferably, the specific steps of step 2 are as follows:

[0021] 2-1. Obtaining image semantics: First, semantic segmentation information of the face image is obtained based on facial segmentation semantic segmentation.

[0022] 2-2. Perform AND operation on the pre-processed image x pixels according to the semantic segmentation information to obtain the facial area image With background image x b .

[0023] 2-3. Use the Dlib library to obtain facial organ coordinate points, and combine different facial organ coordinate points into a facial organ coordinate set in It refers to the coordinate point of the eye. refers to the coordinate point of the nose, Refers to the coordinate point of the mouth, Refers to the coordinate points of the facial contour.

[0024] 2-4. Based on the obtained semantic segmentation information and the facial organ coordinate points, the facial organ images and coordinate points are combined to obtain organ images and coordinate point pairs, including eyes. nose Mouth skin and the coordinate points of the facial contour

[0025] Preferably, the specific steps of step 3 are as follows:

[0026] Using a local anonymizer, each selected facial part is replaced with an anonymous part based on some reliable anonymity strategy (e.g., k-nearest neighbor strategy), and the facial part coordinate points before anonymity are also replaced.

[0027] The anonymous strategy takes images of eyes, nose, mouth, and skin as input, and uses a pre-trained convolutional neural network M to extract feature vectors. By calculating the similarity between vectors, a random one is selected from the component group with a value greater than a set threshold to replace the corresponding organ of the original face, and the morphological contour of the corresponding organ is replaced at the same time. The similarity between feature vectors is calculated using cosine similarity, and the formula is:

[0028] S(x1,x2)=cos(M(x1),M(x1))

[0029] Additional anonymity rules: Since facial information is highly dependent on the coordinate points of facial organs, in order to make the generated facial structure more natural, some anonymity rules are added for the anonymous replacement of facial organ coordinate points. For example, when replacing the mouth, the positions of the coordinate points on both sides of the replaced mouth should be close to those on both sides of the original mouth coordinate points.

[0030] Preferably, the specific steps of step 4 are as follows:

[0031] 4-1. Construct a local region encoder. The local region encoder takes the local image and its coordinate points as input and uses a pre-trained convolutional neural network to extract feature vectors as input for the subsequent generation process.

[0032] 4-2. Construct a facial encoder. The facial encoder takes the skin and facial contours of the original face image as input and uses a pre-trained convolutional neural network to extract feature vectors as input for the subsequent generation process.

[0033] 4-3. Build the generator. The generator takes the feature vectors extracted by the facial encoder as input and uses a pre-trained residual neural network to incorporate the feature vectors extracted by the component encoders into each layer of the network structure to generate the image. After the facial region is generated, the background content of the original facial image is added to the final layer of the network structure for a fitting reconstruction, generating an anonymous face with a consistent background content.

[0034] As a preference, in step 5, model training and testing, use a public dataset for training and testing, and output the final results, as follows:

[0035] Prepare a dataset, for example, use the CelebA-HQ or FFHQ dataset and preprocess it as described in step 1 to obtain the required input images.

[0036] 5-2. Reconstruction training uses facial region information belonging to the same person as all training inputs, aiming to enable the model to acquire excellent generation capabilities and generate images with good visual effects based on arbitrary input information.

[0037] 5-3. Introducing the discriminant model, using the discriminator D1 for judging the authenticity of the image and the discriminator D2 for judging the rationality of the face generation result.

[0038] 5-4. Use the loss function to improve the generation effect of the generative model. The definition formula of the loss function is as follows:

[0039] L=L GD +λ1L1+λ2L2+λ3L3

[0040] Among them, λ1~λ3 are the weights of each loss. L1 is the reconstruction loss, L2 is the style loss, L3 is the perceptual loss, and L GD It’s about fighting losses.

[0041] 5-5. In order to verify the efficiency of the proposed method, it is compared with current excellent methods (such as Riddle, CIAGAN, DeepPrivacy, etc.) and indicators such as anonymity rate, re-identification rate, and attribute preservation rate are calculated.

[0042] The present invention has the following characteristics and beneficial effects:

[0043] Using the above technical solution, this method modifies localized facial regions in images to achieve facial anonymity. Compared to previous mosaic occlusion methods, this method is more efficient and visually friendly, generating higher-quality images than existing deep learning methods. It eliminates the need for attribute labels to preserve the original non-identifying attributes. Different resulting images can be flexibly generated based on user preferences. Experimental results clearly demonstrate the efficiency and practicality of the proposed method. The proposed method provides more efficient and aesthetically pleasing privacy protection for human images.

[0044] The specific features are as follows: (1) Changing the original identifiable identity features; (2) Achieving fine-grained anonymity effects by changing some areas; (3) Flexible support for face synthesis based on user personalized needs; (4) Generating visually realistic face images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is a flow chart of the network architecture of an embodiment of the present invention.

[0047] Figure 2 2 is a diagram comparing the effects of the embodiment of the present invention and the prior art. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0049] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0050] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0051] The present invention provides a method for protecting face identity privacy based on local modification, such as Figure 1 As shown, the following steps are included:

[0052] Step 1: Obtain the dataset and preprocess it.

[0053] Specifically, the steps are as follows:

[0054] 1-1. Image Correction: Correct images with distorted or rotated faces so that the face remains parallel to the image frame.

[0055] 1-2. Facial recognition and cropping.

[0056] Step 2: Segment the facial organs in the preprocessed image to obtain eyes, nose, mouth, skin and background.

[0057] Specifically, the facial organ segmentation method is:

[0058] Step 2-1, obtaining semantic segmentation information of the face image based on facial semantic segmentation;

[0059] Step 2-2: Perform AND operation on the pre-processed image x pixels according to the semantic segmentation information to obtain the facial area image With background image x b ;

[0060] Step 2-3: Use the Dlib library to obtain facial coordinate points and combine them into a facial organ coordinate set based on different facial organ coordinate points in It refers to the coordinate point of the eye. refers to the coordinate point of the nose, Refers to the coordinate point of the mouth, Refers to the coordinate points of the facial contour;

[0061] Step 2-4: Based on the obtained semantic segmentation information and facial organ coordinate points, in this embodiment, based on the influence of different regions on the character identity information, some regions are combined to obtain the eye nose Mouth skin and the coordinate points of the facial contour

[0062] Step 3: Use the local anonymizer to replace each selected facial part with an anonymous part based on the anonymity strategy.

[0063] Specifically, a local anonymizer is used to replace each selected facial organ with an anonymous organ based on some reliable anonymity strategies, and the coordinate points of the facial organs before anonymity are replaced, where the facial organs include eyes, nose, and mouth.

[0064] Furthermore, in this embodiment, the anonymity strategy uses a pre-trained convolutional neural network M to extract feature vectors for each organ type. By calculating the similarity between the vectors, a random one is selected from the component group with a value greater than a set threshold to replace the corresponding facial organ of the original face, and the morphological contour of the corresponding facial organ is replaced at the same time. The similarity between feature vectors is calculated using cosine similarity, and the formula is:

[0065] S(x1,x2)=cos(M(x1),M(x1))

[0066] It should be noted that in this embodiment, the anonymity strategy may also adopt the k-nearest neighbor strategy.

[0067] A further configuration of this embodiment includes additional anonymity rules. Since facial information is highly dependent on the coordinate points of the facial organs of the face, in order to make the generated facial structure more natural, some anonymity rules are added for the anonymous replacement of the coordinate points of the facial organs of the face. In this embodiment, the replacement of the mouth should ensure that the positions of the coordinate points on both sides of the replaced mouth are close to the positions of the coordinate points on both sides of the original mouth.

[0068] Step 4: Build and train a generative model, which includes a local region encoder, a facial encoder, and a generator.

[0069] Specifically, a local area encoder is constructed. First, a pre-trained VGG19 network and a MUST network structure are used to encode the anonymous eyes through the local area encoder. nose Mouth skin Perform feature extraction and concatenate the four feature vectors through a concatenation operation. This module takes the anonymous replacement region as input, with an input size of 128*128*3 and an output size of 320*1*2.

[0070] Construct a facial encoder. By stacking four layers of residual downsampling blocks, we extract features from the downsampled facial organ coordinates and then perform four layers of convolution to generate abstract features. This module takes the facial organ coordinates and skin as input, with an input size of 128*128*5 and an output size of 320*16*16.

[0071] Construct a generator. A four-layer AdaIN module and a residual upsampling module are stacked to form the feature conversion module for the face region. Four convolution operations and a concatenation operation are then used to generate the facial structural features and the original face background information. Finally, the Tanh activation function is used to obtain the final output. The main method used for generation is AdaIN, which is expressed as:

[0072]

[0073] Among them, μ(θ i ) and σ(θ i ) represent the mean and standard deviation of the original image features, μ(s i ) and σ(s i ) represent the mean and standard deviation of the features of the original image respectively.

[0074] This formula can be understood as first de-stylizing (subtracting the mean and dividing by the standard deviation), then stylizing to the target image's style (multiplying the target image's standard deviation and adding the mean). This module takes the output of the component encoder and the output of the face encoder as input and ultimately outputs an RGB image with an output dimension of 128*128*3.

[0075] A further configuration of this embodiment is to generate a training method for the model. In this embodiment, a public data set is used for training and testing, and the final result is output.

[0076] First, prepare the dataset. In this example, we use the CelebAMask-HQ and FFHQ datasets and perform preprocessing as described in Step 1 to obtain the required input images and facial components. The CelebAMask-HQ dataset contains 30,000 photos of 6,216 people of different identities, 5,000 of which are used for testing. The FFHQ dataset contains 70,000 facial images, 5,000 of which are used for testing.

[0077] Reconstruction training: input the facial parts obtained in step 1 of the same face image into the component encoder; input the remaining parts and the facial organ coordinate point information into the face shape encoder, and use L1 loss, style loss, etc. to control the model training, hoping that the model can have a better generation effect.

[0078] Furthermore, a discriminant model is introduced, using discriminator D1 for determining image authenticity and discriminator D2 for determining the rationality of facial generation results. Both discriminators have the same structure, consisting of one convolutional layer, two downsampling layers, a six-layer residual structure, and a sigmoid activation function. The facial organ coordinate point discriminator determines whether the generated image matches the corresponding facial organ coordinate points. The input dimensions are 128*128*3 and 128*128*5, that is, the generated image and the corresponding facial organ coordinate points. The output is a scalar indicating the degree of match between the image and the corresponding facial organ coordinate points. The image authenticity discriminator determines whether the generated image is a real face. The input dimensions are 128*128*3, that is, the real image or the generated fake image. Its output is a scalar indicating the authenticity of the image.

[0079] In this embodiment, a loss function is used to improve the generation effect of the generative model. The definition formula of the loss function is as follows:

[0080] L=L GD +λ1L1+λ2L2+λ3L3

[0081] Among them, λ1~λ3 are the weights of each loss. L1 is the reconstruction loss, L2 is the style loss, L3 is the perceptual loss, and LGD It’s about fighting losses.

[0082] In order to verify the efficiency of the proposed method, we compared it with the current excellent methods (such as Riddle, CIAGAN, DeepPrivacy, etc.) and calculated indicators such as anonymity rate, re-identification rate, and attribute preservation rate. For the comparison of re-identification rate and face detection rate, we used models such as ArcFace, AdaFace, and FaceNet. The specific generated images are compared with other methods as follows Figure 2 shown.

[0083] Step 5: Use the trained generative model to reconstruct the image: use the anonymous organs of the eyes, nose, mouth, and skin as the input of the local area encoder, and use the coordinate points of the replaced facial organs as the input of the facial encoder. Finally, generate the reconstructed image through the generator.

[0084] Comparison of experimental results using this embodiment with those of the prior art:

[0085] 1. We tested the anonymity effect and attribute preservation rate of our method on the FFHQ and CelebAMask-HQ datasets. The detailed data results are shown in Table 1.

[0086]

[0087]

[0088] 2. We tested and compared the anonymity and attribute preservation rates of our method, DeepPrivacy, DeepPrivacy2, CIAGAN, and Riddle. The detailed data results are shown in Table 2.

[0089]

[0090] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for protecting face identity privacy based on local modification, characterized in that: The steps include: Step 1: Obtain the data set and preprocess it; Step 2: Segment the facial organs in the preprocessed image to obtain eyes, nose, mouth, skin and background; The facial organ segmentation method is: Step 2-1, obtaining semantic segmentation information of the face image based on facial semantic segmentation; Step 2-2: Perform AND operation on the pre-processed image x pixels according to the semantic segmentation information to obtain the facial area image With background image x b ; Step 2-3: Use the Dlib library to obtain the facial organ coordinate points, and combine the facial organ coordinate points into a facial organ coordinate set in It refers to the coordinate point of the eye. refers to the coordinate point of the nose, Refers to the coordinate point of the mouth, Refers to the coordinate points of the facial contour; Step 2-4: Based on the semantic segmentation information and facial organ coordinate points, the facial organ images and coordinate points are combined to obtain organ images and coordinate point pairs, including eyes. nose Mouth skin and the coordinate points of the facial contour Step 3: Use a local anonymizer to replace each selected facial part with an anonymous part based on the anonymity strategy, and replace the coordinate points of the facial parts before anonymization. The facial parts include eyes, nose, and mouth. The anonymity strategy is as follows: taking pictures of the eyes, nose, mouth, and skin as input, extracting feature vectors using a pre-trained convolutional neural network M, calculating the similarity between the feature vectors, and randomly selecting one of the organ image and coordinate point pairs that is greater than a set threshold to replace the corresponding organ of the original face; Step 4: Build and train a generative model, where the generative model includes a local region encoder, a facial encoder, and a generator. The local area encoder includes a pre-trained VGG19 network and a MUST network structure, and the local area encoder is used to train the anonymized eyes. nose Mouth skin Perform feature extraction and concatenate the four feature vectors through concatenation operation; The facial encoder is a downsampling block with a stacked four-layer residual structure. The facial encoder extracts downsampled features of organ coordinate points and obtains their abstract features through a four-layer convolution operation. The input of the facial encoder is skin and the coordinate points of the facial contour The generator takes the output of the local region encoder and the facial encoder as input and uses a four-layer AdaIN module and a residual structure upsampling module as a feature conversion module for the face region; Then, four convolution operations and Concat operations are used as a splicing generation module to generate facial structural features and original face background information, and finally the Tanh activation function is used to obtain the final output; Step 5: Use the trained generative model to reconstruct the image: use the anonymous organs of the eyes, nose, mouth, and skin as the input of the local area encoder, and use the coordinate points of the replaced facial organs as the input of the facial encoder. Finally, generate the reconstructed image through the generator.

2. The method for protecting face identity privacy based on local modification according to claim 1, characterized in that: The preprocessing method includes correcting a picture with a distorted or rotated face, so that the face is parallel to the picture frame after correction.

3. The method for protecting face identity privacy based on local modification according to claim 1, characterized in that: The similarity between the feature vectors is calculated using cosine similarity.

4. The method for protecting face identity privacy based on local modification according to claim 1, characterized in that: The generative model introduces a discriminant model, which includes a discriminator D1 for judging the authenticity of the image and a discriminator D2 for judging the rationality of the face generation result; the discriminator D1 and the discriminator D2 are composed of a convolutional layer, two downsampling layers, a six-layer residual structure and a Sigmoid activation function.

Images