Face identity privacy protection method based on local modification

Abstract

The invention provides a face identity privacy protection method based on local modification. The face identity privacy protection method comprises the following steps: firstly, acquiring a data set and preprocessing the data set; segmenting facial organs in the preprocessed image; 3, replacing each selected facial organ with an anonymous organ based on an anonymous strategy by using a local anonymous device, and replacing the coordinate point of the facial organ before anonymization; step 4, constructing and training a generative model, wherein the generative model comprises a local area encoder, a face encoder and a generator; and 5, performing image reconstruction by using the trained generative model: taking anonymous organs of eyes, a nose, a mouth and skin as input of a local region encoder, taking anonymous organs of facial organ coordinate points as input of a facial encoder, and finally, generating a reconstructed image through a generator. The method not only can achieve the purpose of cheating uninformed observers, but also can protect the availability of data.

Description

Technical Field

[0001] The present invention relates to the field of face recognition and information protection technology, and in particular to a face identity privacy protection method based on local modification. Background Art

[0002] We are now in an era of intelligence and informatization. The rapid development of computer technology has promoted the widespread emergence of smart devices and applications, providing people with personalized and diversified services. On the social networking platforms used in daily life, there are a large number of easily recognizable facial images. Most of these data describe individual behavioral characteristics in specific scenarios. As a highly personalized information carrier, the face is an important symbol to distinguish different individuals and is sensitive information.

[0003] As the accuracy of face recognition technology improves, this information can identify the identity of individuals in the scene and their various attribute characteristics in real time, which poses a major threat to personal privacy and security. Therefore, there is an urgent need to protect unauthorized face image data. In order to solve the problem of privacy leakage caused by a large number of images, some countries have taken measures, such as formulating relevant laws and regulations to prevent images containing private information from being used without authorization. Although these laws and regulations restrict data users from obtaining authorization to a certain extent, thereby protecting privacy, they also increase the difficulty of data acquisition and greatly restrict the development of related scientific research.

[0004] In recent years, many interesting traditional and generative anonymization methods have been proposed. Traditional methods, such as occlusion, mosaic, and blur, are often used to protect personal privacy because they are simple to implement and low cost. However, these methods provide protection by damaging a large amount of image content and visual effects, resulting in a significant decrease in data availability. With the continuous development of computer technology hardware and software, generative anonymization methods have gradually become the dominant anonymity methods. Tao Li et al. proposed AnonymousNet, which mainly synthesizes anonymous images by modifying non-identity attribute information in the original image, and then adds noise to the synthesized image to make it undetectable by the recognition system, thereby improving privacy protection. The DeepPrivacy method proposed by et al. in 2019 realizes anonymous scenarios that can be applied to multiple people in one picture, and can also be extended to videos. The method is to use random noise to block the face in advance and use the target detection Mask R-CNN method to locate the position of the face and organs, and then use CGAN to generate anonymous images. CIAGAN researchers use a hot carrier of facial landmarks and identities to remove people's identifying features, while still retaining the necessary features to allow face and body detectors to work. This type of method requires borrowing the identity of others as an operation guide, which may affect the privacy and security of the identity provider. Secondly, the generated image will only be very good when the fake ID provider is of the same gender, similar age, and similar makeup as the original spouse. It is very inconvenient to use, and some special attributes such as glasses, heavy makeup, and beards are not maintained.

[0005] In summary, the field of facial identity privacy protection is a topic worthy of in-depth research. This patent intends to start the discussion from several coordinate points in this field to solve the difficulties and key points of the current methods.

[0006] One of the key points of facial identity privacy protection is the effectiveness of face anonymity and the balance between it and data reusability. However, most current methods focus on the effectiveness of anonymity in the global context. Unfortunately, these methods also lose other attribute information that can be retained. Specifically, there are two main difficulties:

[0007] 1. Few methods consider the feasibility of anonymity from a local perspective. Existing methods all consider the anonymity effect on the face from a global perspective, but few consider making changes to the local area to achieve the anonymity effect and retain some information that can be left unchanged.

[0008] 2. How to balance the relationship between anonymity cost and anonymity protection level. Pursuing the improvement of anonymity under limited resource conditions to meet the high anonymity requirements in different scenarios is also an important direction for promoting the development of the privacy protection field. Summary of the invention

[0009] The purpose of the present invention is to address the deficiencies of the prior art and propose a facial identity privacy protection method based on local modification. Users can freely select the local position that needs to be changed and generate personalized anonymous images, which can not only deceive uninformed observers but also protect the availability of data.

[0010] In order to solve the above technical problems, the technical solution of the present invention is:

[0011] A method for protecting face identity privacy based on local modification includes the following steps:

[0012] Step 1: Dataset preprocessing;

[0013] Step 2: facial organ segmentation;

[0014] Step 3: Local anonymous replacement;

[0015] Step 4: Generate model construction;

[0016] Step 5: Model training and testing.

[0017] Preferably, the specific steps of step 1 are as follows:

[0018] 1-1. Image correction: Images with distorted or rotated faces are corrected using existing methods so that the faces can remain parallel to the image borders.

[0019] 1-2. Face recognition and cropping. Obtain the coordinates of the main part of the face and crop the face for subsequent use.

[0020] Preferably, the specific steps of step 2 are as follows:

[0021] 2-1. Obtaining image semantics: First, obtain the semantic segmentation information of the face image based on facial segmentation semantic segmentation.

[0022] 2-2. Perform AND operation on the preprocessed image x pixels according to the semantic segmentation information to obtain the facial area image With background image x b .

[0023] 2-3. Use the Dlib library to obtain the facial organ coordinate points, and combine the facial organ coordinate sets according to different facial organ coordinate points in It refers to the coordinate point of the eye. refers to the coordinate point of the nose, refers to the coordinate point of the mouth, Refers to the coordinate points of the facial contour.

[0024] 2-4. Based on the semantic segmentation information and the facial organ coordinate points, the facial organ images and coordinate points are combined to obtain organ images and coordinate point pairs, including eyes. nose Mouth skin And the coordinate points of the facial contour

[0025] Preferably, the specific steps of step 3 are as follows:

[0026] Using a local anonymizer, each selected facial organ is replaced with an anonymous organ based on some reliable anonymity strategy (e.g., k-nearest neighbor strategy), and the facial organ coordinate points before anonymity are replaced.

[0027] The anonymous strategy takes pictures of eyes, nose, mouth and skin as input, extracts feature vectors using the pre-trained convolutional neural network M, and randomly selects one from the component group that is greater than the set threshold to replace the corresponding organ of the original face by calculating the similarity between the vectors, while replacing the morphological contour of the corresponding organ. The similarity between feature vectors is calculated using cosine similarity, and the formula is

[0028] S(x1,x2)=cos(M(x1),M(x1))

[0029] Additional anonymity rules: Since facial information is highly dependent on the coordinate points of facial organs of the human face, in order to make the generated facial structure more natural, some anonymity rules are added for the anonymous replacement of facial organ coordinate points. For example, the replacement of the mouth should ensure that the positions of the coordinate points on both sides of the replaced mouth are close to the positions of the coordinate points on both sides of the original mouth.

[0030] Preferably, the specific steps of step 4 are as follows:

[0031] 4-1. Construct a local region encoder. The local region encoder takes the local image and its coordinate points as input, and uses a pre-trained convolutional neural network to extract feature vectors as input for the subsequent generation process.

[0032] 4-2. Construct a facial encoder. The facial encoder takes the skin and facial contour of the original face image as input, and uses a pre-trained convolutional neural network to extract feature vectors as input for the subsequent generation process.

[0033] 4-3. Construct the generator. The generator takes the feature vector extracted by the facial encoder as input, uses the pre-trained residual neural network, adds the feature vector extracted by the component encoder to each layer of the network structure, and generates the image. After the facial area is generated, the background content of the original face image is added to the last layer of the network structure for fitting reconstruction to generate an anonymous face with consistent background content.

[0034] As a preference, in step 5, model training and testing, public data sets are used for training and testing, and the final results are output as follows:

[0035] 5-1. Prepare a dataset, for example, use the CelebA-HQ or FFHQ dataset and perform preprocessing as described in step 1 to obtain the required input images.

[0036] 5-2. Reconstruction training uses facial area information belonging to the same person as all training inputs, aiming to enable the model to acquire excellent generation capabilities and generate images with good visual effects based on any input information.

[0037] 5-3. The discriminant model is introduced, and the discriminator D1 for judging the authenticity of the image and the discriminator D2 for judging the rationality of the face generation result are used respectively.

[0038] 5-4. Use the loss function to improve the generation effect of the generative model. The definition formula of the loss function is as follows:

[0039] L=L GD +λ1L1+λ2L2+λ3L3

[0040] Among them, λ1~λ3 are the weights of each loss. L1 is the reconstruction loss, L2 is the style loss, L3 is the perceptual loss, and L GD It’s about fighting against loss.

[0041] 5-5. In order to verify the efficiency of the proposed method, it is compared with current excellent methods (such as Riddle, CIAGAN, DeepPrivacy, etc.) and the anonymity rate, re-identification rate, attribute retention rate and other indicators are calculated.

[0042] The present invention has the following characteristics and beneficial effects:

[0043] By adopting the above technical solution, this method modifies the local area of ​​the face in the image to achieve the effect of face anonymity. Compared with the previous mosaic occlusion method, it is more efficient and visually friendly. It has higher quality in generating images than the existing deep methods, and does not need to use attribute labels to maintain the original non-identity attributes. Different result images can be flexibly generated according to the personalized requirements of users. The experimental results clearly confirm the efficiency and practicality of the proposed method. The proposed method is more efficient and beautiful in protecting the privacy of human images.

[0044] The specific features are as follows: (1) Changing the original identifiable identity features; (2) Achieving fine-grained anonymity effects by changing some areas; (3) Flexible support for face synthesis based on user personalized needs; (4) Generating visually realistic face images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 The network architecture flow chart of an embodiment of the present invention.

[0047] Figure 2 The figure is a comparison diagram of the effects of the embodiment of the present invention and the prior art. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0049] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0050] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0051] The present invention provides a method for protecting face identity privacy based on local modification, such as Figure 1 As shown, the following steps are included:

[0052] Step 1: Obtain and preprocess the dataset.

[0053] Specifically, the steps are as follows:

[0054] 1-1. Image correction: Correct images with distorted or rotated faces so that the faces remain parallel to the image borders.

[0055] 1-2. Facial recognition and cropping.

[0056] Step 2: Segment the facial organs in the preprocessed image to obtain eyes, nose, mouth, skin and background.

[0057] Specifically, the facial organ segmentation method is:

[0058] Step 2-1, obtaining semantic segmentation information of the face image based on facial semantic segmentation;

[0059] Step 2-2: Perform AND operation on the preprocessed image x pixels according to the semantic segmentation information to obtain the facial area image With background image x b ;

[0060] Step 2-3: Use the Dlib library to obtain facial coordinate points, and combine the facial organ coordinate points into a facial organ coordinate set in It refers to the coordinate point of the eye. refers to the coordinate point of the nose, refers to the coordinate point of the mouth, Refers to the coordinate points of the facial contour;

[0061] Step 2-4: Based on the obtained semantic segmentation information and facial organ coordinate points, in this embodiment, based on the influence of different regions on the person's identity information, some regions are combined to obtain the eye nose Mouth skin And the coordinate points of the facial contour

[0062] Step 3: Use the local anonymizer to replace each selected facial organ with an anonymous organ based on the anonymity strategy.

[0063] Specifically, a local anonymizer is used to replace each selected facial organ with an anonymous organ based on some reliable anonymity strategies, and the facial organ coordinate points before anonymization are replaced, wherein the facial organs include eyes, nose, and mouth.

[0064] Furthermore, in the anonymity strategy of this embodiment, each organ type uses a pre-trained convolutional neural network M to extract feature vectors, and by calculating the similarity between vectors, randomly select one from the component group greater than a set threshold to replace the corresponding facial organ of the original face, and replace the morphological contour of the corresponding facial organ. The similarity between feature vectors is calculated using cosine similarity, and the formula is:

[0065] S(x1,x2)=cos(M(x1),M(x1))

[0066] It should be noted that in this embodiment, the anonymity strategy may also adopt the k-nearest neighbor strategy.

[0067] A further setting of this embodiment is an additional anonymity rule. Since facial information is highly dependent on the coordinate points of facial organs of the human face, in order to make the generated facial structure more natural, some anonymity rules are added for the anonymous replacement of the coordinate points of facial organs of the human face. In this embodiment, the replacement of the mouth should ensure that the positions on both sides of the coordinate points of the replaced mouth are close to the positions on both sides of the coordinate points of the original mouth.

[0068] Step 4: construct and train a generative model, wherein the generative model includes a local area encoder, a facial encoder, and a generator.

[0069] Specifically, a local area encoder is constructed. First, a pre-trained VGG19 network and a MUST network structure are used to train the local area encoder on the anonymous eyes. nose Mouth skin Perform feature extraction and concatenate the four feature vectors through concatenation. This module takes the anonymous replaced area as input, the input size is 128*128*3, and the output size is 320*1*2.

[0070] Construct a facial encoder. By stacking four layers of residual structured downsampling blocks, downsample the facial organ coordinate points to extract features, and obtain their abstract features through four layers of convolution operations. This module takes the facial organ coordinate points and skin as input, with an input size of 128*128*5 and an output size of 320*16*16.

[0071] Construct the generator. Stack four layers of AdaIN modules and the upsampling module of the residual structure as the feature conversion module of the face area. Then use four convolution operations and Concat operations as the splicing generation module to generate facial structure features and the original face background information, and finally use the Tanh activation function to get the final output. The main method used for generation is AdaIN, and its expression is:

[0072]

[0073] Among them, μ(θ i ) and σ(θ i ) represent the mean and standard deviation of the original image features, μ(s i ) and σ(s i ) represent the mean and standard deviation of the features of the original image respectively.

[0074] It can be understood that this formula can be understood as first de-stylizing (subtracting its own mean and then dividing it by its own standard deviation), and then stylizing it to the style of the target image (multiplying the standard deviation of the target image and adding the mean). This module takes the output of the component encoder and the output of the face encoder as input, and finally outputs an RGB image, that is, the output dimension is 128*128*3.

[0075] A further configuration of this embodiment is to generate a training method for the model. In this embodiment, a public data set is used for training and testing, and a final result is output.

[0076] First, prepare the data set. In this embodiment, the CelebAMask-HQ and FFHQ data sets are used and preprocessed as described in step 1 to obtain the required input images and facial parts. The CelebAMask-HQ data set contains 30,000 photos of 6,216 people of different identities, of which 5,000 are used for testing. The FFHQ data set contains 70,000 face images, of which 5,000 are used for testing.

[0077] Reconstruction training: input the facial parts obtained from the same face image in step 1 into the component encoder; input the remaining parts and the coordinate point information of the facial organs into the face shape encoder, and use L1 loss, style loss, etc. to control the model training, hoping that the model can have a better generation effect.

[0078] Furthermore, a discriminant model is introduced, and a discriminator D1 for judging the authenticity of the image and a discriminator D2 for judging the rationality of the face generation result are used respectively. The two discriminators have the same structure, consisting of a convolution layer, two downsampling layers, six residual structures and a Sigmoid activation function. The facial organ coordinate point discriminator can judge whether the generated image is consistent with the corresponding facial organ coordinate points of the face. The input dimensions are 128*128*3 and 128*128*5, that is, the generated image and the corresponding facial organ coordinate points of the face. The output is a scalar, indicating the degree of matching between this image and the corresponding facial organ coordinate points; the image true and false discriminator can judge whether the generated image is a real face. The input dimension is 128*128*3, that is, a real image or a generated fake image. Its output is a scalar, indicating the authenticity of this image.

[0079] In this embodiment, a loss function is used to improve the generation effect of the generation model. The definition formula of the loss function is as follows:

[0080] L=L GD +λ1L1+λ2L2+λ3L3

[0081] Among them, λ1~λ3 are the weights of each loss. L1 is the reconstruction loss, L2 is the style loss, L3 is the perceptual loss, and LGD It’s about fighting against loss.

[0082] In order to verify the efficiency of the proposed method, we compared it with the current excellent methods (such as Riddle, CIAGAN, DeepPrivacy, etc.) and calculated indicators such as anonymity rate, re-identification rate, and attribute retention rate. For the comparison of re-identification rate and face detection rate, we used models such as ArcFace, AdaFace, and FaceNet. The specific generated images are compared with other methods. Figure 2 shown.

[0083] Step 5: Use the trained generative model to reconstruct the image: use the anonymous organs of the eyes, nose, mouth, and skin as the input of the local area encoder, and use the replaced facial organ coordinate points as the input of the facial encoder. Finally, generate the reconstructed image through the generator.

[0084] Comparison of experimental results of this embodiment with those of the prior art:

[0085] 1. The anonymity effect and attribute retention rate of this method on the FFHQ and CelebAMask-HQ datasets were tested. The specific data results are shown in Table 1.

[0086]

[0087]

[0088] 2. The anonymity effects and attribute retention rates of this method, DeepPrivacy, DeepPrivacy2, CIAGAN, and Riddle were tested and compared. The specific data results are shown in Table 2.

[0089]

[0090] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A method for protecting face identity privacy based on local modification, characterized in that: The steps include: Step 1: Obtain and preprocess the data set; Step 2: segment the facial organs in the preprocessed image to obtain eyes, nose, mouth, skin and background; Step 3: Use a local anonymizer to replace each selected facial organ with an anonymous organ based on anonymity strategy, and replace the facial organ coordinate points before anonymization, where the facial organs include eyes, nose, and mouth; Step 4: construct and train a generative model, wherein the generative model includes a local region encoder, a facial encoder, and a generator; Step 5: Use the trained generative model to reconstruct the image: use the anonymous organs of the eyes, nose, mouth, and skin as the input of the local area encoder, and use the replaced facial organ coordinate points as the input of the facial encoder. Finally, generate the reconstructed image through the generator.

2. According to claim 1, a method for protecting face identity privacy based on local modification is characterized in that: The preprocessing method includes correcting a picture with a distorted or rotated face, so that the face is parallel to the picture frame after correction.

3. The method for protecting face identity privacy based on local modification according to claim 1, characterized in that: The facial organ segmentation method is: Step 2-1, obtaining semantic segmentation information of the face image based on facial semantic segmentation; Step 2-2: Perform AND operation on the preprocessed image x pixels according to the semantic segmentation information to obtain the facial area image With background image x b ; Step 2-3: Use the Dlib library to obtain the facial organ coordinate points, and combine the facial organ coordinate points into a facial organ coordinate set in It refers to the coordinate point of the eye. refers to the coordinate point of the nose, refers to the coordinate point of the mouth, Refers to the coordinate points of the facial contour; Step 2-4: Based on the semantic segmentation information and the facial organ coordinate points, the facial organ images and coordinate points are combined to obtain organ images and coordinate point pairs, including eyes. nose Mouth skin And the coordinate points of the facial contour 4. The method for protecting face identity privacy based on local modification according to claim 1, characterized in that: In step 3, the anonymity strategy is: taking pictures of eyes, nose, mouth and skin as input, extracting feature vectors using a pre-trained convolutional neural network M, and calculating the similarity between feature vectors, randomly selecting one of the organ pictures and coordinate point pairs that are greater than a set threshold to replace the corresponding organ of the original face.

5. The method for protecting face identity privacy based on local modification according to claim 4, characterized in that: The similarity between the feature vectors is calculated using cosine similarity.

6. The method for protecting face identity privacy based on local modification according to claim 1, characterized in that: The local area encoder includes a pre-trained VGG19 network and a MUST network structure. nose Mouth skin Perform feature extraction and concatenate the four feature vectors through concatenation operation.

7. The method for protecting face identity privacy based on local modification according to claim 1, characterized in that: The facial encoder is a downsampling block with a stacked four-layer residual structure. The facial encoder performs downsampling feature extraction on organ coordinate points and obtains its abstract features through a four-layer convolution operation.

8. The method for protecting face identity privacy based on local modification according to claim 7, characterized in that: The input of the facial encoder is the skin And the coordinate points of the facial contour 9. The method for protecting face identity privacy based on local modification according to claim 8, characterized in that: The generator uses the output of the local area encoder and the face encoder as input, and stacks four layers of AdaIN modules and an upsampling module of a residual structure as a feature conversion module for the face area; Then four convolution operations and Concat operations are used as a splicing generation module to generate facial structure features and original face background information, and finally the Tanh activation function is used to obtain the final output.

10. The method for protecting face identity privacy based on local modification according to claim 1, characterized in that: The generative model introduces a discriminant model, which includes a discriminator D1 for discriminating the authenticity of an image and a discriminator D2 for discriminating the rationality of a face generation result; the discriminator D1 and the discriminator D2 are composed of a convolutional layer, two downsampling layers, six residual structures, and a Sigmoid activation function.

Images