A facial image de-identification and restoration system and method

Abstract

The present invention provides a personalized and restorable face de-identification system and method, which obtains the identity features and attribute features of the original face image or the de-identified face image; edits the obtained identity features according to the password and privacy protection level set by the user, inputs the identity features extracted from the original face image in the protection stage to calculate the de-identified face identity features, and inputs the identity features extracted from the de-identified face image in the restoration stage to calculate the restored face identity features; synthesizes the corresponding de-identified face image or the restored face identity features based on the calculated de-identified face identity features or the restored face identity features and the attribute features extracted from the input face image. The present invention integrates the codec network structure, multi-scale feature fusion and identity feature editing, and realizes diversified and restorable face image privacy protection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a facial image de-identification and restoration system and method. Background Art

[0002] Facial image privacy protection is a widely concerned task in the field of computer vision, and its goal is to remove sensitive private information before uploading and sharing facial images with unknown third parties. With the development of modern social media, millions of private photos have been uploaded and shared, which has brought convenience to life but also greatly threatened personal privacy. Currently, the more common face de-identification method is to delete or modify the identity information in the original image while retaining the visual similarity and usability of the image as much as possible. Although some traditional face de-identification methods have been widely used to hide sensitive facial image privacy information, most existing methods are only aimed at the protection stage, which helps to enhance security when daily monitoring equipment is used or when users upload images on social media. However, considering some special scenarios such as criminal investigation, the identity information of the original image needs to be used, so how to restore the original image is also a crucial task.

[0003] In computer vision, there are many traditional methods to enhance facial privacy. The most widely used methods include blurring, pixelation, and masking. However, relevant studies have shown that these methods are easily attacked by deep learning-based face recognition models, have limited protection effects on identity information, and will greatly affect the usability of images. These disturbances can also be easily perceived by the human eye. With the development of deep learning technology, the de-identification method based on generative adversarial networks has greatly improved the visual perception of de-identification results. The challenge of face image privacy protection tasks is first reflected in how to control the balance between privacy and image utility, and the degree of privacy protection can be controlled according to the application scenario; secondly, it is to effectively modify identity information while retaining non-sensitive information such as facial expressions or behaviors; existing methods mainly focus on the de-identification process, and most methods lack controllability over the degree of privacy protection of de-identification results and cannot restore the original image.

[0004] After searching, Chinese invention patent application CN202010343798.5 discloses a face de-identification generation method based on a generative adversarial network, wherein: N pairs of face images are obtained, feature vectors are extracted for each face image, the facial features of each face image are occluded with random noise to obtain an occluded face image, the occluded face image is combined with the corresponding face image feature vector, and the two are combined as the input of the generator in the generative adversarial network, and the original face image is used as the real face image of the discriminator to form a training sample; the generator and the discriminator are trained using the training sample. After the training is completed, in the application stage, for each face image to be de-identified, the occluded face image and the face image feature vector are also obtained, and then combined and input into the generator of the trained face de-identification generation model to obtain the de-identified face image. However, the invention does not involve the problem of restoring the de-identified face image. Summary of the invention

[0005] In view of the above technical problems existing in the prior art, the present invention proposes a facial image de-identification and restoration system and method.

[0006] A first aspect of the present invention provides a method for de-identifying and restoring a facial image, comprising:

[0007] Extracting features from an input facial image to obtain relatively independent identity features and attribute features of the input facial image; the input facial image is an original facial image or a de-identified facial image;

[0008] The obtained identity features are edited according to the password and privacy protection level set by the user, including a protection stage and a recovery stage. In the protection stage, the identity features extracted from the original face image are input to calculate the identity features of the de-identified face. In the recovery stage, the identity features extracted from the de-identified face image are input to calculate the identity features of the restored face.

[0009] According to the calculated de-identified face identity features or restored face identity features and the attribute features extracted from the input face image, a corresponding de-identified face image or restored face image is synthesized, wherein:

[0010] In the protection stage, the de-identified face identity features and the attribute features extracted from the input original face image are synthesized to form a de-identified face image;

[0011] In the restoration stage, the restored facial identity features and the attribute features extracted from the input de-identified facial image are synthesized to restore the facial image.

[0012] Optionally, the feature extraction of the input face image to obtain relatively independent identity features and attribute features of the input face image comprises: extracting the identity features and attribute features of the input face image using an identity feature encoder and an attribute feature encoder; specifically comprising:

[0013] The identity feature encoder is implemented by a face recognition model. For an input face image, the face recognition model extracts a feature vector before the last fully connected layer as an identity feature;

[0014] The attribute feature encoder adopts the U-Net network structure, and extracts the feature vector from the kth layer of the U-Net network decoder part for the input face image. Represents the total number of decoder layers, forming a multi-level feature map to represent attribute features;

[0015] The identity feature extracted by the identity feature encoder and the attribute feature extracted by the attribute feature encoder are de-entangled and independent of each other.

[0016] Optionally, in the protection stage, the input facial image is the original facial image, the identity feature encoder extracts the original identity features, and uses the original identity features and the password and privacy protection degree parameters set by the user as input conditions, and sends them to the identity feature editing module to calculate the de-identified facial identity features; in the recovery stage, the input facial image is the de-identified facial image, the identity feature encoder extracts the de-identified facial identity features, and uses the de-identified facial identity features and the password and privacy protection degree parameters set by the user during the encryption process as input conditions, and sends them to the identity feature editing module to calculate the restored facial identity features.

[0017] Optionally, in the protection stage, the identity feature encoder extracts the original identity feature, and uses the original identity feature and the password set by the user and the privacy protection degree parameter as input conditions, and sends them to the identity feature editing module to calculate and identify the face identity feature, including:

[0018] The identity feature encoder is used to extract the identity feature z of the input original face image. id and attribute feature z attr ;

[0019] According to the password p set by the user, the corresponding reference vector z is extracted from the pre-defined reference vector library r =f(p), where the reference vector library is obtained by randomly selecting k face images from the training data set to extract identity features, taking the average and normalizing them, and f is the one-to-one mapping relationship between the user-set password and the reference vector; from the reference vector z r Decompose the original identity feature z id The orthogonal part z 90 :

[0020] z 90 =z r -(z id ·f(p))·z id

[0021] where z 90 With z id A set of orthogonal bases is formed; the original identity feature modification angle θ=g(d) is calculated according to the privacy protection degree d set by the user, and the de-identified face identity feature is obtained by rotating the original identity feature in the orthogonal base:

[0022] z new =z id ·cosθ+z 90 ·sinθ.

[0023] Optionally, in the recovery stage, the identity feature encoder extracts the de-identified face identity features, and uses the de-identified face identity features and the password and privacy protection degree parameter set by the user during the encryption process as input conditions, and sends them to the identity feature editing module to calculate the restored face identity features, including:

[0024] The identity feature encoder is used to extract the identity features of the input to identify the face image. and attribute characteristics

[0025] According to the password p set by the user, the corresponding reference vector z is extracted from the pre-defined reference vector library r =f(p); according to the privacy protection degree d set by the user, the original identity feature modification angle θ=g(d) is calculated, and the restored face identity feature is calculated as:

[0026]

[0027] in:

[0028]

[0029] Optionally, synthesizing a corresponding de-identified face image or restored face image according to the calculated de-identified face identity features or restored face identity features and the attribute features extracted by the encoder from the input face image includes:

[0030] For the identity features and attribute features of the corresponding results generated by the input, an adaptive attention denormalization layer is used to improve the multi-level feature integration effect to control the synthesis of different parts;

[0031] The multi-level attribute features are input into different layers of the generator, and convolution dimension reduction and Sigmoid normalization operations are performed on the input of the previous layer to obtain a new feature map, which is then used to fuse the identity features and attribute features of this layer, and finally generate the corresponding face image.

[0032] Optionally, the facial image de-identification and restoration method further includes:

[0033] By scoring the authenticity of the synthesized de-identified face image or the restored face image, the adversarial loss is calculated, thereby optimizing the quality of the synthesized de-identified face image or the restored face image to make it closer to a real face image.

[0034] A second aspect of the present invention provides a facial image de-identification and restoration system for implementing the facial image de-identification and restoration method, comprising:

[0035] The encoder extracts features from an input face image to obtain relatively independent identity features and attribute features of the input face image; the input face image is an original face image or a de-identified face image;

[0036] An identity feature editing module is used to edit the identity features obtained by the encoder according to the password and privacy protection level set by the user, which includes a protection stage and a recovery stage. In the protection stage, the identity features extracted from the original face image are input to calculate the identity features of the de-identified face. In the recovery stage, the identity features extracted from the de-identified face image are input to calculate the identity features of the original image face.

[0037] A synthesis network generator synthesizes a corresponding de-identified face image or a restored face image according to the de-identified face identity features or the restored face identity features calculated by the identity feature editing module and the attribute features extracted by the encoder from the input face image, wherein:

[0038] In the protection stage, the de-identified face identity features and the attribute features extracted from the input original face image are input into the synthesis network generator to synthesize the de-identified face image;

[0039] In the restoration stage, the restored facial identity features and the attribute features extracted from the input de-identified facial image are input into the synthesis network generator to synthesize the restored facial image.

[0040] Optionally, the encoder includes: an identity feature encoder and an attribute feature encoder; wherein:

[0041] The identity feature encoder extracts identity features from the input face image, and the identity feature encoder is specifically implemented as a face recognition model, extracting feature vectors from before the last fully connected layer as identity features; the input face image is an original face image or a de-identified face image;

[0042] The attribute feature encoder extracts attribute features from the input face image. Specifically, the U-Net network structure is used to extract feature vectors from the kth layer of the U-Net network decoder. represents the total number of decoder layers, forming a multi-level feature map to represent attribute features; the input face image is an original face image or a de-identified face image;

[0043] The identity feature extracted by the identity feature encoder and the attribute feature extracted by the attribute feature encoder are de-entangled and independent of each other.

[0044] Optionally, the facial image de-identification and restoration system further includes:

[0045] The synthetic network discriminator calculates the adversarial loss by scoring the true or false of the de-identified face image or the restored face image synthesized by the synthetic network generator, thereby optimizing the quality of the de-identified face image or the restored face image synthesized by the synthetic network generator to make it closer to the real face image.

[0046] Optionally, the synthetic network discriminator uses m multi-scale discriminators to discriminate different resolution versions X of the de-identified face image or the restored face image. m The hinge loss function is used to calculate the adversarial loss, m represents the number of downsampling times, and the final sum is taken as the authenticity.

[0047] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0048] The facial image de-identification and restoration system and method provided by the present invention maintain targeted manipulation of image identity information by extracting relatively independent identity features and attribute features; identity feature editing only modifies the identity features, keeping the generated image consistent with the attribute features of the input image, so that the generated de-identification result effectively retains the visual similarity and image usability with the original image.

[0049] The face image de-identification and restoration system and method provided by the present invention use a multi-scale attribute feature encoder to extract multi-level feature maps in a multi-level decoder of a U-Net structure to form attribute features, which is conducive to better learning face attribute features and retaining details such as expression, posture, and lighting; further, the decoded image features (generator) are combined with attribute features from multiple scales to generate a target face image with authenticity and good visual effects.

[0050] The face image de-identification and restoration system and method provided by the present invention adjust the areas of focus of identity features and attribute features when generating face images by introducing an adaptive attention denormalization module in the generator module, input multi-level attribute features into different layers of the generator network, perform convolution dimension reduction and normalization operations on the previous layer input to obtain a new feature map, and use the feature map to fuse the identity features and attribute features of the layer, and finally generate the corresponding face image, thereby avoiding problems such as blurring of the generated image caused by directly cascading two features;

[0051] The face image de-identification and restoration system and method provided by the present invention distinguish different resolution versions of the image, and finally obtain the sum as the authenticity, and combine the idea of ​​multi-level processing into the synthetic network discriminator, which not only improves the perception range of the synthetic network discriminator, but also produces very effective discriminant guidance for local information such as texture and details of the image;

[0052] The face image de-identification and restoration system and method provided by the present invention use gradient penalty on the synthetic network discriminator to achieve the training goal of stabilizing the synthetic network. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0054] Figure 1 A flowchart of a facial image de-identification and restoration method according to an embodiment of the present invention;

[0055] Figure 2 A module diagram of a facial image de-identification and restoration system according to an embodiment of the present invention;

[0056] Figure 3 A flowchart of a facial image de-identification and restoration method according to a preferred embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a facial image de-identification and restoration system training process according to an embodiment of the present invention;

[0058] Figure 5A schematic diagram of the protection and recovery phases of a facial image de-identification and restoration system according to an embodiment of the present invention;

[0059] Figure 6 A diagram showing the effect of setting passwords for different users according to an embodiment of the present invention;

[0060] Figure 7 A rendering of the results of generating different privacy protection levels in an embodiment of the present invention;

[0061] Figure 8 It is a diagram showing the effect of different passwords and correct recovery and incorrect recovery generation results in one embodiment of the present invention;

[0062] Fig. 9 It is a comparison diagram of the generation results in one embodiment of the present invention and the generation results of different existing methods. DETAILED DESCRIPTION

[0063] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0064] Figure 1 Flow chart of a facial image de-identification and restoration method according to an embodiment of the present invention.

[0065] Reference Figure 1 As shown, in this embodiment, the face image de-identification and restoration method includes the following steps:

[0066] S100, extracting features from the input original face image or the de-identified face image to obtain relatively independent identity features and attribute features of the original face image or the de-identified face image;

[0067] S200, editing the obtained identity features according to the password and privacy protection level set by the user, and calculating the de-identified face identity features or the restored face identity features, wherein the identity features extracted from the original face image are input in the protection stage to calculate the de-identified face identity features, and the identity features extracted from the de-identified face image are input in the restoration stage to calculate the restored face identity features;

[0068] S300, synthesize the corresponding de-identified face image or restored face image based on the de-identified face identity features or restored face identity features calculated by S200 and the attribute features of the original face image or the de-identified face image obtained by S100, wherein, in the protection stage, the de-identified face identity features and the attribute features extracted from the input original face image are synthesized into the de-identified face image; in the restoration stage, the restored face identity features and the attribute features extracted from the input de-identified face image are synthesized into the restored face image.

[0069] Through the above steps, this embodiment can achieve facial image de-identification and corresponding restoration, and extract relatively independent identity features and attribute features to maintain targeted manipulation of image identity information; further, the identity features are modified through identity feature editing to keep the generated image consistent with the attribute features of the input image, so that the generated de-identification result effectively retains the visual similarity and image usability with the original image.

[0070] In the above embodiment, when executing S100, feature extraction is performed on the input face image to obtain relatively independent identity features and attribute features of the input face image, wherein the identity features and attribute features of the input face image are extracted using the identity feature encoder and the attribute feature encoder. Specifically: the identity feature encoder is implemented using a face recognition model. For the input face image, the face recognition model extracts a feature vector from before the last fully connected layer as the identity feature; the attribute feature encoder uses a U-Net network structure. For the input face image, the feature vector is extracted from the kth layer of the U-Net network decoder part. Represents the total number of decoder layers, forming a multi-level feature map to represent attribute features; the identity features extracted by the identity feature encoder and the attribute features extracted by the attribute feature encoder are mutually disentangled and independent of each other. In this preferred embodiment, through a multi-scale attribute feature encoder, a multi-level feature map is extracted in a multi-level decoder of a U-Net structure to form attribute features, which is conducive to better learning facial attribute features and retaining details such as expression, posture, and lighting. Furthermore, the subsequent decoding of image features combines the attribute features from multiple scales to generate a target face image with authenticity and good visual effects.

[0071] Figure 2 2 is a module diagram of a facial image de-identification and restoration system according to an embodiment of the present invention. Figure 4 A schematic diagram of a facial image de-identification and restoration system training process according to an embodiment of the present invention; Figure 5 Schematic diagram of the protection and recovery phases of a facial image de-identification and restoration system according to an embodiment of the present invention.

[0072] Reference Figure 2-5 As shown, the face image de-identification and restoration system in this embodiment is used to implement Figure 1 The facial image de-identification and restoration method shown specifically includes: an encoder, an identity feature editing module, and a synthetic network generator. Among them: the encoder extracts features from the input face image to obtain relatively independent identity features and attribute features of the input face image, and the input face image is an original face image or a de-identified face image; the identity feature editing module edits the identity features obtained by the encoder according to the password and privacy protection level set by the user, which includes a protection stage and a recovery stage. In the protection stage, the identity features extracted from the input original face image are used to calculate the de-identified face identity features, and in the recovery stage, the identity features extracted from the input de-identified face image are used to calculate the original image face identity features; the synthesis network generator synthesizes the corresponding de-identified face image or restored face image according to the de-identified face identity features or restored face identity features calculated by the identity feature editing module and the attribute features extracted from the input face image by the encoder, wherein in the protection stage, the de-identified face identity features and the attribute features extracted from the input original face image are input into the synthesis network generator to synthesize the de-identified face image; in the recovery stage, the restored face identity features and the attribute features extracted from the input de-identified face image are input into the synthesis network generator to synthesize the restored face image.

[0073] The system of this embodiment integrates the encoder, identity feature editing module and synthetic network generator to achieve diversified and restorable face image privacy protection. Figure 4 Schematic diagram of the training process of the face image de-identification and restoration system shown.

[0074] In order to obtain a result closer to a real face image, in a preferred embodiment, the face image de-identification and restoration system adopts a generative adversarial network, including: an encoder, an identity feature editing module, a synthetic network generator and a synthetic network discriminator; wherein: the encoder extracts features from the input face image to obtain relatively independent identity features and attribute features of the input face image; the identity feature editing module edits the identity features extracted from the input face image according to the password and privacy protection level set by the user, which includes a protection stage and a restoration stage. In the protection stage, the identity features extracted from the original face image are input to calculate the de-identified face identity features, and in the restoration stage, the identity features extracted from the de-identified face image are input to calculate the original image face identity features; the synthetic network generator synthesizes the corresponding de-identification or restoration face image results based on the identity features calculated by the identity feature editing module and the attribute features extracted by the encoder for the input face image; the synthetic network discriminator calculates the adversarial loss by scoring the generated face images for authenticity, thereby optimizing the quality of the generated face images synthesized by the synthetic network generator to be closer to real face images. Figure 5Schematic diagram of the protection and recovery stages of the facial image de-identification and restoration system shown.

[0075] like Figure 2 , 4 As shown in , 5, the face image de-identification and restoration system of this embodiment extracts relatively independent identity features and attribute features based on the input face; the second level uses the identity feature editing module to encrypt or restore the identity features in the latent space, and calculates the corresponding de-identified face identity features or restored face identity features; the synthetic network generator fuses the identity features and attribute features through the adaptive attention denormalization module, decodes and generates the corresponding face image, and better maintains the quality and authenticity of the generated image.

[0076] Compared with directly editing the image features, there are many advantages to extracting disentangled identity features and attribute features from the image and then only editing the identity features. The key to the de-identification task is to protect the image identity information. In order to retain the image availability, it is hoped that other irrelevant attributes remain unchanged as much as possible. For directly extracting features from the image for editing, it is possible to modify other irrelevant facial attributes while changing the image identity information. At the same time, a greater degree of modification may be required to achieve the purpose of changing the identity, reducing the effectiveness of identity protection under the same editing degree. For the de-identification task, it is not conducive to achieving a good trade-off between the effectiveness of privacy protection and the retention of image availability. If the original facial features are decoupled into identity features and attribute features, in the subsequent editing, only the attribute features can be retained consistent with the identity features, so as to more specifically change the identity of the facial image while retaining the visual similarity with the original image as much as possible. Therefore, in an embodiment of the present invention, relatively independent identity features and attribute features are extracted.

[0077] In some embodiments, the encoder extracts relatively independent identity features and attribute features based on the input face, which can be specifically implemented by using an identity feature encoder and an attribute feature encoder, wherein: the identity feature encoder can use a face recognition model to extract identity features from the input face image, and extract feature vectors from before the last fully connected layer of the face recognition model as identity features; the attribute feature encoder uses a multi-level feature map to represent attribute features in order to better retain details such as expression, posture, and lighting, and extracts feature vectors from the kth layer of the decoder. represents the total number of decoder layers; the identity feature extracted by the identity feature encoder and the attribute feature extracted by the attribute feature encoder are de-entangled and independent of each other.

[0078] In the above embodiment, in order to improve the final effect, for the identity features and attribute features of the corresponding results generated by the input in the synthesis network generator, an adaptive attention denormalization layer can be preferably used to improve the multi-level feature integration effect to control the synthesis of different parts; the multi-level attribute features are input into different layers of the generator, and convolution dimension reduction and Sigmoid normalization operations are performed on the input of the previous layer to obtain a new feature map, and the feature map is used to fuse the identity features and attribute features of this layer, and finally the corresponding face image is generated.

[0079] In order to obtain a better image synthesis effect, in some embodiments, the synthesis network generator may use identity consistency loss, that is, the cosine similarity between the identity features extracted from the generated image and the identity features input when the generated network reconstructs the image, to ensure that the identity features extracted from the generated image do not change from the identity features used during generation. Furthermore, the synthesis network generator may also choose to use attribute consistency loss, that is, the Euclidean distance between each level of the attribute features extracted from the generated image and the attribute features input when the generated network reconstructs the image and sum them, to ensure that the attribute features extracted from the generated image do not change from the attribute features used during generation. In addition, the synthesis network generator may also choose to use self-reconstruction loss to ensure that when the target identity features and attribute features are consistent with the input image, the network can accurately reconstruct the original image.

[0080] In the above embodiment, the synthetic network discriminator discriminates different resolution versions of the generated image, and finally takes the sum as the authenticity. Further, the synthetic network discriminator uses the hinge loss function to calculate the adversarial loss. Specifically, the synthetic network discriminator uses m multi-scale discriminators to discriminate different resolution versions X of the face image to be identified or restored. m The hinge loss function is used to calculate the adversarial loss, m represents the number of downsampling times, and the final sum is taken as the authenticity. Incorporating the idea of ​​multi-level processing into the synthetic network discriminator not only improves the perception range of the synthetic network discriminator, but also produces very effective discriminative guidance for local information such as image texture and details.

[0081] Based on the above embodiments, some embodiments further include: using a stable training strategy of gradient penalty on the synthetic network discriminator to stabilize the training process of the synthetic network discriminator. By performing gradient penalty on the synthetic network discriminator, the training goal of stabilizing the synthetic network is achieved.

[0082] In the above embodiment of the present invention, the identity feature encoder, the attribute feature encoder, the synthesis network generator and the synthesis network discriminator constitute a generative adversarial network: according to the identity feature calculated by the identity feature editing module, the input face image is end-to-end converted to complete face de-identification or restoration. The synthesis network generator and the synthesis network discriminator are used to handle training and reasoning problems: the target image is sent to the synthesis network discriminator to obtain the authenticity score of the face image; the encoder is used to extract features for the generated face, and the distance between the corresponding features of the input face image is calculated as the guidance and control conditions for the generated image to synthesize the real target face image. Furthermore, the synthesis network generator uses an adaptive attention denormalization layer to improve the multi-level feature integration effect, inputs the multi-level attribute features into different layers of the generator, performs convolution dimensionality reduction and normalization operations on the previous layer input, and obtains a new feature map to guide the fusion of the identity features and attribute features of this layer, so that the two parts of the features guide different generation areas, and better maintain the quality and realism of the generated image.

[0083] Figure 3 Flow chart of a facial image de-identification and restoration method according to a preferred embodiment of the present invention.

[0084] Reference Figure 3 As shown, this preferred embodiment provides a method for de-identifying and restoring a face image, which adopts Figure 2 The system shown in the figure is used to integrate the disentangled multi-level attribute features and the identity feature editing module to achieve high-quality face generation.

[0085] Specifically, the face image de-identification and restoration method provided in this embodiment includes the following steps:

[0086] S11, extracting identity features and attribute features of the input face image using an identity feature encoder and an attribute feature encoder;

[0087] S12, in the protection stage, the input is the original face image, the identity feature encoder extracts the original identity feature, takes the original identity feature and the password set by the user and the privacy protection degree parameter as input conditions, and sends them to the identity feature editing module to calculate the de-identified face identity feature; in the recovery stage, the input is the de-identified face image, the identity feature encoder extracts the de-identified face identity feature, takes the de-identified face identity feature and the password set by the user in the encryption process and the privacy protection degree parameter as input conditions, and sends them to the identity feature editing module to calculate the restored face identity feature;

[0088] In the protection stage, the de-identified face identity features and the attribute features extracted from the original image are input into the synthesis network generator to synthesize the de-identified face image; in the restoration stage, the restored face identity features and the attribute features extracted from the original image are input into the synthesis network generator to synthesize the restored face image.

[0089] S13, using the synthetic network discriminator, by scoring the generated face images for authenticity, calculate the adversarial loss, and thus optimize the quality of the generated face images synthesized by the synthetic network generator to make them closer to real face images.

[0090] In this embodiment, when S11 is executed, the method of extracting the original identity features and attribute features of the input face image by using the identity feature encoder and the attribute feature encoder includes:

[0091] S111, use the pre-trained face recognition model as the identity feature encoder E id , the identity features of a given image X are extracted from the last fully connected layer before the last feature vector;

[0092] S112, using attribute feature encoder E similar to U-Net structure attr Extract a multi-level feature map of a given image to represent attribute features:

[0093]

[0094] in, represents the feature vector extracted from the kth layer of the decoder, and n represents the total number of layers of the decoder. In the embodiment, the attribute feature encoder n layers refers to eight layers. It should be noted that the number of layers of the multi-level feature map in the attribute feature encoder is not limited to eight layers.

[0095] Since most of the most advanced face recognition models or face verification models map identity features into a hyperspherical space and use angle-based cosine similarity for judgment. Therefore, compared with vector operations such as translation, rotating identity features is more conducive to changing identity information. At the same time, considering the feasibility of recovery, a deterministic process should be selected in editing rather than simply introducing randomness. Using the user encryption password as a parameter of the identity feature editing module rather than as a condition of the synthetic network generator can make the user encryption process relatively independent of the generation network, and can define the password form more flexibly, while reducing the complexity of calculation and training. In addition, the range of identity changes during the de-identification process can also be expanded. Based on this, in an embodiment of the present invention, the identity feature editing module performs end-to-end identity conversion on the input face image according to the identity features calculated by the identity feature editing module to complete face de-identification or restoration.

[0096] Specifically, when executing S12, in the protection stage, the original identity features, the password set by the user, and the privacy protection degree parameter are used as input conditions and sent to the identity feature editing module to calculate and identify the face identity features; including:

[0097] S121, using the encoder to extract the identity feature z of the input original face image id and attribute feature z attr ;

[0098] S122, extracting the corresponding reference vector z from the predefined reference vector library according to the password p set by the user r =f(p), where the reference vector library is to extract identity features by randomly selecting k face images from the training data set, taking the average and normalizing them to avoid the leakage of other real identities, and f is the one-to-one mapping relationship between the user-set password and the reference vector; from the reference vector z r Decompose the original identity feature z id The orthogonal part z 90 :

[0099] z 90 =z r -(z id ·f(p))·z id

[0100] where z 90 With z id A set of orthogonal bases is formed. According to the privacy protection degree d set by the user, the modification angle of the original identity feature θ = g(d) is calculated, and the de-identified face identity feature is obtained by rotating the original identity feature in the orthogonal base:

[0101] z new =z id ·cosθ+z 90 sinθ

[0102] Specifically, when executing S12, in the restoration stage, the de-identified face identity features and the password and privacy protection degree parameters set by the user are used as input conditions and sent to the identity feature editing module to calculate the restored face identity features; including:

[0103] S123, using the encoder to extract input to identify the identity features of the face image and attribute characteristics

[0104] S124, extracting the corresponding reference vector z from the predefined reference vector library according to the password p set by the user r =f(p); the original identity feature modification angle θ=g(d) is calculated according to the privacy protection degree d set by the user, and the calculated restored face identity feature is:

[0105]

[0106] in:

[0107]

[0108] It should be understood that the above S121 to S124 are merely labels for step operations and do not necessarily have to be performed in sequence.

[0109] In this embodiment, when S12 is executed, the identity features calculated by the identity feature editing module and the attribute features extracted from the image are input into the synthesis network generator to synthesize the corresponding de-identified or restored face image, and the following operations are preferably performed: for the identity features and attribute features of the input corresponding results, an adaptive attention denormalization layer is used to improve the multi-level feature integration effect to control the synthesis of different parts; the multi-level attribute features are input into different layers of the generator, and convolution dimension reduction and Sigmoid normalization operations are performed on the previous layer input to obtain a new feature map, and the feature map is used to fuse the identity features and attribute features of this layer, and finally generate the corresponding face image.

[0110] Due to the instability of the generative adversarial network, the method of this embodiment randomly samples two face images from the data set during the training process to extract identity features and attribute features respectively. If the identities of the two face images are the same, the reconstruction loss is calculated between the face image generated by the synthetic network generator and the original image; if the identities of the two face images are different, the reconstruction loss is not calculated between the face image generated by the synthetic network generator and the original image, and it is recorded as 0.

[0111] The synthetic network generator and the synthetic network discriminator constitute a generative adversarial network, which is used to deal with training and reasoning problems: the target image is sent to the synthetic network discriminator to obtain a facial image authenticity score; the encoder is used to extract features of the generated face, and the distance to the input face image is calculated as the guidance and control condition for the generated image to synthesize the real target face image.

[0112] In the above embodiments, the implementation features of the facial image de-identification and restoration method and the facial image de-identification and restoration system correspond to each other, and the parts not described in detail can be referenced to each other and will not be repeated again.

[0113] In order to better illustrate the effect of the embodiment of the present invention, in a specific example, the code implementation is completed by Pytorch. In the training stage, the learning rates of the generator and the discriminator are set to 0.0004, and the Adam optimizer β1=0, β2=0.999 is used. The results of the face image de-identification and restoration method of the above embodiment are evaluated, and CelebA-HQ is selected as the training and test data sets. DeepPrivacy and the algorithm proposed by Gu et al. are used as the most advanced method at present, and quantitative and qualitative comparisons are made with the above embodiment of the present invention.

[0114] The quantitative evaluation of the algorithm adopts a combination of privacy protection effectiveness and de-identified image availability. The objective evaluation indicators include:

[0115] (1) Effectiveness of privacy protection: Since most face verification models determine whether two face images have the same identity by comparing the identity representation distance, the distance (Id-dis) between the identity feature of the de-identified result and the original image is calculated in the evaluation and further compared with the reference threshold of the face recognition model used. When the identity distance is greater than the model reference threshold, it is considered that the identities are different, that is, the de-identification is successful, and the protection success rate (SR) is calculated. The higher the identity feature distance (Id-dis) and the higher the protection success rate (SR), the more effective the identity protection of the face image is. For each model, the Face Recognition library, the FaceNet model trained on the CASIA-WebFace dataset, and the FaceNet model trained on the VGGFace2 dataset were used for testing. The password and privacy protection level used for de-identification by the method of this embodiment were randomly selected in the test. The comparison results are shown in Table 1. The method of this embodiment can achieve a larger identity distance and a higher success rate, which is more effective for identity protection.

[0116] Table 1 Comparison of identity distance and de-identification success rate between the embodiment of the present invention and the existing method

[0117]

[0118] (2) Availability of de-identified images: The de-identification results should also be applicable to other computer vision tasks. The proportion of recognizable faces in the de-identified image is defined as face detectability (DR). The more detectable faces, the higher the availability. Detect the face area and key points of the de-identified image, and calculate the pixel distance (pixel-dis) between the de-identified image and the original image. The smaller the pixel distance between the de-identified image and the original image, the higher the availability. In the test, OpenCV is used to detect the face area, and Dlib is used to detect the face key points. The password and privacy protection level used in the de-identification process of the method of this embodiment are randomly selected in the test. The comparison results of the three methods are shown in Table 2. According to the experimental results, it can be seen that the method of this embodiment can better preserve the consistency between the face area and key points, and most of the generated faces can be detected, which shows that the method of this embodiment can better ensure the availability of de-identified images for identity-independent computer vision tasks.

[0119] Table 2 Comparison of face detectability, face area and key point pixel distance of the embodiment of the present invention and the existing method

[0120]

[0121] In order to demonstrate the diversity of de-identification results generated by this method, this embodiment tests the de-identification effects obtained by using different encryption passwords. The results are as follows: Figure 6 As shown, the first column is the original image, and the second to sixth columns represent the de-identification results generated by different passwords, where each column of images shares the same password. It can be seen that this embodiment can convert the original identity into different identities in a large range, and the identity change direction of different original images can be determined by the password.

[0122] In order to demonstrate the controllability of the de-identification results generated by this method, this embodiment tests the de-identification results obtained at different privacy levels. The results are as follows: Figure 7 As shown in the figure, the first column is the original image, and the privacy protection levels in the second to eleventh columns increase successively. It can be seen that increasing the privacy protection level will increase the identity distance between the de-identified result and the original image, and obtain a higher degree of privacy protection; at the same time, the de-identified result can usually retain a similar appearance to the original image.

[0123] In order to demonstrate the recoverability of the de-identified results generated by this method, this embodiment uses correct and incorrect passwords to recover the protection results. The results are as follows: Figure 8As shown, the first column is the original image, the second and third columns represent the protection results generated by different passwords, the fourth column represents the restoration of the third column image using the second column password, the fifth column represents the restoration of the second column image using the third column password, and the sixth and seventh columns represent the restoration results of the correct password, where the sixth column represents the restoration of the second column image using the second column password, and the seventh column represents the restoration of the third column image using the third column password. It can be seen that using the correct password can get the correct restoration result, while using the wrong password can still get a high-quality face image, which can confuse attackers to a certain extent and enhance the security of protection, but the original identity information cannot be obtained.

[0124] like Fig. 9 As shown in the figure, the first column represents the original image, the second to fourth columns represent traditional privacy protection methods (blurring, pixelation, Gaussian noise), the fifth column represents the results of the DeepPrivacy method, the sixth column represents the results of the method proposed by Gu et al., and the seventh column represents the de-identification results of this method. It can be seen that the de-identification algorithm based on deep learning can generate more natural results with better visual quality, and this method further considers the similarity with the original image, which is closer to the original image semantics while effectively realizing identity protection, and retains more visual similarities.

[0125] The facial image de-identification and restoration method provided by the embodiment of the present invention integrates: an identity feature encoder module, an attribute feature encoder module, an identity feature editing module, and a generator module; relatively independent identity features and attribute features are extracted through the identity feature encoder and the attribute feature encoder, so as to maintain targeted manipulation of the image identity information; the identity feature editing module only modifies the identity features, keeps the generated image consistent with the attribute features of the input image, so that the generated de-identification result effectively retains the visual similarity and image usability with the original image; the generator module decodes the image features, and combines the attribute features from multiple scales to generate a target facial image with authenticity.

[0126] The face image de-identification and restoration method provided by the embodiment of the present invention extracts multi-level feature maps to form attribute features in a multi-level decoder of a U-Net structure through a multi-scale attribute feature encoder, which is conducive to better learning face attribute features and retaining details such as expression, posture, and lighting;

[0127] The face image de-identification and restoration system and method provided by the embodiments of the present invention adjust the areas that identity features and attribute features pay attention to when generating face images by introducing an adaptive attention denormalization module in the generator module, input multi-level attribute features into different layers of the generator network, perform convolution dimension reduction and normalization operations on the previous layer input to obtain a new feature map, and use the feature map to fuse the identity features and attribute features of the layer, and finally generate the corresponding face image, avoiding problems such as blurred generated images caused by directly cascading two features.

[0128] The facial image de-identification and restoration system and method provided by the above-mentioned embodiments of the present invention integrate an identity feature encoder module, an attribute feature encoder module, an identity feature editing module, and a generator module to achieve high-quality face synthesis.

[0129] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, that is, the embodiments in the system can be understood as preferred examples of implementing the method, which will not be elaborated here.

[0130] Those skilled in the art know that, in addition to implementing the system and its various devices provided by the present invention in a purely computer-readable program code, the system and its various devices provided by the present invention can be made to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0131] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A facial image de-identification and restoration method, characterized in that: include: Extract features from the input face image to obtain relatively independent identity features and attribute features of the input face image; The input face image is an original face image or a de-identified face image; The obtained identity features are edited according to the password and privacy protection level set by the user, including a protection stage and a recovery stage. In the protection stage, the identity features extracted from the original face image are input to calculate the identity features of the de-identified face. In the recovery stage, the identity features extracted from the de-identified face image are input to calculate the identity features of the restored face. According to the calculated de-identified face identity features or restored face identity features and the attribute features extracted from the input face image, a corresponding de-identified face image or restored face image is synthesized, wherein: In the protection stage, the de-identified face identity features and the attribute features extracted from the input original face image are synthesized to form a de-identified face image; In the restoration stage, the restored face identity features and the attribute features extracted from the input de-identified face image are synthesized to restore the face image; The feature extraction of the input face image is performed to obtain relatively independent identity features and attribute features of the input face image, wherein the identity features and attribute features of the input face image are extracted using an identity feature encoder and an attribute feature encoder; specifically including: The identity feature encoder is implemented by a face recognition model. For an input face image, the face recognition model extracts a feature vector before the last fully connected layer as an identity feature; The attribute feature encoder adopts the U-Net network structure, and extracts the feature vector from the kth layer of the U-Net network decoder part for the input face image. k∈n, n represents the total number of decoder layers, forming a multi-level feature map to represent attribute features; The identity feature extracted by the identity feature encoder and the attribute feature extracted by the attribute feature encoder are mutually de-entangled and independent of each other; In the protection stage, the input face image is the original face image, the identity feature encoder extracts the original identity feature, and the original identity feature and the password and privacy protection degree parameter set by the user are used as input conditions and sent to the identity feature editing module to calculate and identify the face identity feature; In the recovery phase, the input face image is a de-identified face image, the identity feature encoder extracts the de-identified face identity features, and the de-identified face identity features and the password and privacy protection degree parameters set by the user during the encryption process are used as input conditions and sent to the identity feature editing module to calculate the restored face identity features; In the protection stage, the identity feature encoder extracts the original identity feature, and uses the original identity feature and the password and privacy protection degree parameter set by the user as input conditions, and sends them to the identity feature editing module to calculate and identify the face identity feature, including: The identity feature encoder is used to extract the identity feature z of the input original face image. i d and attribute feature z attr ; According to the password p set by the user, the corresponding reference vector z is extracted from the pre-defined reference vector library r =f(p), where the reference vector library is obtained by randomly selecting k face images from the training data set to extract identity features, taking the average and normalizing them, and f is the one-to-one mapping relationship between the user-set password and the reference vector; from the reference vector z r Decompose the original identity feature z id The orthogonal part z 90 : With 90 =from r -(With id ·f(p))·z id where z 90 With z id A set of orthogonal bases is formed; the original identity feature modification angle θ=g(d) is calculated according to the privacy protection degree d set by the user, and the de-identified face identity feature is obtained by rotating the original identity feature in the orthogonal base: With new =from id cosθ+z 90 ·sinθ。 2. The facial image de-identification and restoration method according to claim 1, characterized in that: In the recovery phase, the identity feature encoder extracts the de-identified face identity features, and uses the de-identified face identity features and the password and privacy protection degree parameters set by the user during the encryption process as input conditions, and sends them to the identity feature editing module to calculate the restored face identity features, including: The identity feature encoder is used to extract the identity features of the input to identify the face image. and attribute characteristics According to the password p set by the user, the corresponding reference vector z is extracted from the pre-defined reference vector library r =f(p); according to the privacy protection degree d set by the user, the original identity feature modification angle θ=g(d) is calculated, and the restored face identity feature is calculated as: in:

3. The facial image de-identification and restoration method according to claim 1, characterized in that: According to the calculated de-identified face identity features or restored face identity features and the attribute features extracted by the encoder from the input face image, a corresponding de-identified face image or restored face image is synthesized, including: For the identity features and attribute features of the corresponding results generated by the input, an adaptive attention denormalization layer is used to improve the multi-level feature integration effect to control the synthesis of different parts; The multi-level attribute features are input into different layers of the generator, and convolution dimension reduction and Sigmoid normalization operations are performed on the input of the previous layer to obtain a new feature map, which is then used to fuse the identity features and attribute features of this layer, and finally generate the corresponding face image.

4. The facial image de-identification and restoration method according to claim 2, characterized in that: Also includes: By scoring the authenticity of the synthesized de-identified face image or the restored face image, the adversarial loss is calculated, thereby optimizing the quality of the synthesized de-identified face image or the restored face image to make it closer to a real face image.

5. A facial image de-identification and restoration system, used to implement the facial image de-identification and restoration method according to any one of claims 1 to 4, characterized in that: include: The encoder extracts features from the input face image to obtain relatively independent identity features and attribute features of the input face image; The input face image is an original face image or a de-identified face image; An identity feature editing module is used to edit the identity features obtained by the encoder according to the password and privacy protection level set by the user, which includes a protection stage and a recovery stage. In the protection stage, the identity features extracted from the original face image are input to calculate the identity features of the de-identified face. In the recovery stage, the identity features extracted from the de-identified face image are input to calculate the identity features of the original image face. A synthesis network generator synthesizes a corresponding de-identified face image or a restored face image according to the de-identified face identity features or the restored face identity features calculated by the identity feature editing module and the attribute features extracted by the encoder from the input face image, wherein: In the protection stage, the de-identified face identity features and the attribute features extracted from the input original face image are input into the synthesis network generator to synthesize the de-identified face image; In the restoration stage, the restored facial identity features and the attribute features extracted from the input de-identified facial image are input into the synthesis network generator to synthesize the restored facial image.

6. The facial image de-identification and restoration system according to claim 5, characterized in that: Also includes: The synthetic network discriminator calculates the adversarial loss by scoring the true or false of the de-identified face image or the restored face image synthesized by the synthetic network generator, thereby optimizing the quality of the de-identified face image or the restored face image synthesized by the synthetic network generator to make it closer to the real face image.

7. The facial image de-identification and restoration system according to claim 6, characterized in that: The synthetic network discriminator uses m multi-scale discriminators to identify the face image or restore the face image with different resolution versions X m The hinge loss function is used to calculate the adversarial loss, m represents the number of downsampling times, and the final sum is taken as the authenticity.

Images