Transparent and restorable face anonymization processing method and system

Abstract

The invention relates to the technical field of image processing, and discloses a transparent and restorable face anonymization processing method and system, and the method comprises the steps: obtaining to-be-anonymized original face images, extracting initial identity features from the original face images through a face feature extractor, and generating a group of keys for each original face image; constructing a reversible neural network, inputting the identity features and the secret keys belonging to the same original face image as a group of input data into the reversible neural network, performing affine transformation through the reversible neural network, and finally generating anonymized identity features controlled by the secret keys; and inputting the original face image and the anonymized identity features into a pre-trained generator so as to generate an anonymized face image. According to the method, the application requirements are fully met, and the dual goals of data security and privacy protection are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a transparent and restorable face anonymization processing method and system. Background Art

[0002] There are relatively few known reversible anonymization methods, and most methods do not fully consider the issue of explainability. Existing methods usually train directly through generative adversarial networks (GANs), that is, the original image is input and the anonymized image is obtained after network processing. However, this method has limitations in the restoration process: the restored input is still the anonymized image, rather than the original image being reversely deduced from the output, resulting in a lack of transparency in the restoration process. In addition, due to the unexplainability of GAN itself, users cannot clearly understand how the various steps in the anonymization process affect the final result. This makes debugging and optimization in applications more difficult, especially in scenarios involving privacy protection and data security.

[0003] The current mainstream face privacy protection methods only focus on the protection process, and there is little research on restoring anonymized faces. In addition, the existing restorable anonymization methods are learned through deep networks. Due to the black box characteristics of the network, the restoration process is not transparent.

[0004] Therefore, in order to make the anonymization process explainable, it is necessary to design a transparent and restorable face anonymization method to make the restoration process transparent. Summary of the invention

[0005] In order to make the anonymization process explainable, the present invention provides a transparent and restorable face anonymization processing method and system, which is composed of multiple reversible modules. Each reversible module has a set of dual-objective mapping functions that perform reversible transformation between original features and anonymized features and use a pre-trained generator to generate anonymized face images. The unique structure makes it naturally reversible and explainable.

[0006] The present invention is achieved through the following technical solutions:

[0007] A transparent and restorable face anonymization processing method, comprising:

[0008] Obtaining an original face image to be anonymized, extracting initial identity features from the original face image by a face feature extractor, and generating a set of keys for each of the original face images;

[0009] Constructing a reversible neural network, and inputting the identity features and the key belonging to the same original face image into the reversible neural network as a set of input data, performing affine transformation through the reversible neural network, and finally generating anonymized identity features controlled by the key;

[0010] The original face image and the anonymized identity features are input into a pre-trained generator to generate an anonymized face image.

[0011] As an optimization, the reversible neural network includes a plurality of identical reversible modules, each of which includes a first four arithmetic operation function group, a second four arithmetic operation function group and a plurality of encryption networks, and the initial identity feature is split into a first intermediate feature and a second intermediate feature, wherein,

[0012] The first intermediate feature is input into the first four arithmetic operation function group, and at the same time, the second intermediate feature is encrypted by the encryption network and then input into the first four arithmetic operation function group to obtain a first intermediate result;

[0013] The second intermediate feature is input into the second four arithmetic operation function group, and at the same time, the first intermediate result is encrypted by the encryption network and then input into the second four arithmetic operation function group to obtain a second intermediate result;

[0014] The first intermediate result and the second intermediate result are combined as the output of the reversible module, and the encryption result of the data by the encryption network is consistent with the encryption result of the data by the key.

[0015] As an optimization, the first four arithmetic operation function group includes at least two different four arithmetic operation functions, the second four arithmetic operation function group includes at least two different four arithmetic operation functions, and the first four arithmetic operation function group and the second four arithmetic operation function group are the same.

[0016] As an optimization, when training the encrypted network, the total loss function is used to update the parameters of the encrypted network. The total loss function is specifically:

[0017]

[0018] in, Represents the total loss function during training; To anonymize the loss of identity information between face images; To anonymize the image information loss between face images; To reduce the image information loss between anonymized face images and restored face images, is the background loss between the anonymized face image and the original face image, λ1, λ2, λ3, and λ4 are and The weight of .

[0019] As an optimization, the identity information loss between the anonymized face images The specific expression is:

[0020]

[0021] in, represents the feature difference loss function, which is used to measure the feature difference between T and P, where T and P are The input of the function, I represents the original face image, p1 and p2 represent different keys, p1′ represents the inverse of key p1, which is used to correctly restore the anonymized face image generated by key p1, p2′ represents the inverse of key p2, which is used to correctly restore the anonymized face image generated by key p2, T p1 I represents the anonymized face image obtained by using the key p1 for the original face image I, T p2 I represents the anonymized face image obtained by using the key p2 from the original face image I, T p I represents the anonymized face image generated by the original face image I and the key p through the reversible neural network, T p1′ T p I represents the face image restored by the wrong key corresponding to the key p1, T p2′ T p I represents the face image restored from the wrong key corresponding to the key p2.

[0022] As an optimization, the feature difference loss function is used to measure the identity information loss between anonymized face images generated by different keys. The specific expression is:

[0023]

[0024] Wherein, F is the pre-trained facial feature extractor, and M1 and M2 are anonymized facial images generated with different keys.

[0025] As an optimization, the image information loss between the anonymized face images The specific expression is:

[0026]

[0027] Where I represents the original face image, F is the face feature extractor, and T p I represents the anonymized face image generated by the original face image I and the key p through the reversible neural network, T p′ T p I represents the restored face image after the wrong key is restored, is the cross entropy loss, a loss function used to measure the difference between the classification model prediction and the true label, y I Represents the true label of the original face image.

[0028] As an optimization, the loss function of anonymizing face images and restoring face images The specific expression is:

[0029]

[0030] Where I represents the original face image, T p and T -p denote the anonymized key and the correctly recovered key respectively, T - p T p I is the image that is expected to be correctly restored, and ‖·,·‖1 represents the L1 distance between the pixels of the two images.

[0031] As an optimization, the background loss between the anonymized face image and the original face image The specific expression is:

[0032]

[0033] Where I represents the original face image, T p and T p′ are different keys for anonymization, and ‖·,·‖1 represents two images.

[0034] A transparent and restorable face anonymization processing system, characterized by comprising:

[0035] An identity extraction module is used to obtain the original face image to be anonymized, extract the initial identity features from the original face image through the face feature extractor, and generate a set of keys for each of the original face images;

[0036] an anonymization module, used to construct a reversible neural network, and input the identity features and the key belonging to the same original face image into the reversible neural network as a set of input data, and finally generate an anonymized identity feature controlled by the key after the reversible neural network performs affine transformation;

[0037] The generation module is used to input the original face image and the anonymized identity feature into a pre-trained generator to generate an anonymized face image.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] The present invention achieves efficient anonymization and restoration of facial images by designing a reversible neural network architecture. The architecture consists of multiple reversible modules, each of which has a dual-objective mapping function, so that the transformation between input and output is reversible and interpretable. This method can effectively protect personal privacy and ensure the ability to restore in specific situations, fully meet application requirements, and ensure the dual goals of data security and privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0041] Figure 1 A flow chart of a transparent and restorable face anonymization processing method according to the present invention;

[0042] Figure 2 A module connection diagram for executing a transparent and restorable face anonymization processing method according to the present invention;

[0043] Figure 3 It is a structural schematic diagram of one of the reversible modules in the embodiment. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0045] This embodiment 1 provides a transparent and restorable face anonymization processing method, such as Figure 1 As shown, including:

[0046] S1. Obtain an original face image to be anonymized, extract initial identity features from the original face image through a face feature extractor, and generate a set of keys for each of the original face images.

[0047] For an original face image to be processed, identity features are first extracted from the original face image through a pre-trained identity feature extractor. The identity feature extractor can use a pre-trained deep model and generate a set of specific keys for anonymization and restoration. Specifically, a set of keys is generated for the original face image to control different anonymization and restoration processes.

[0048] In some embodiments, different original face images correspond to different keys, or different original face images correspond to the same keys. For the confidentiality of image restoration, in this embodiment, different original face images have different keys.

[0049] At the same time, if Figure 1 As shown in the figure, a set of keys for an original face image corresponds to 4 keys, namely:

[0050] Key 1: the key used for anonymization, i.e. the anonymization key;

[0051] Key 2: Random anonymized key;

[0052] Key 3: Correctly recovered key;

[0053] Key 4: Key for error recovery.

[0054] Among them, key one and key three are one-to-one corresponding and are used for anonymization and restoration. The image anonymized by key two cannot be restored by key three. If key three is not used in the restoration process, it cannot be restored correctly.

[0055] Essentially, all four keys are used for anonymization. The only difference is that if you use key 1 to anonymize, you can use key 3 to recover it. If you use key 2 to anonymize, the recovered key is unknown. You can use any key to anonymize an image, but the recovered key is unique to it.

[0056] Figure 3 The hexagon (encrypted network) in is an encryption process or recovery process that only applies to one key.

[0057] S2. Construct a reversible neural network, and input the identity features and the key belonging to the same original face image into the reversible neural network as a set of input data, perform affine transformation through the reversible neural network, and finally generate anonymized identity features controlled by the key.

[0058] In some embodiments, the reversible neural network includes a plurality of identical reversible modules, a key is embedded in each reversible module, and a reversible transformation is performed between the original features and the anonymized features through a set of dual-objective mapping functions.

[0059] The identity features of the original image are reversibly transformed through a reversible network architecture. The reversible neural network includes several identical reversible modules, each of which is composed of multiple arbitrary network couplings. A set of dual-objective mapping functions is used in combination with a key to perform reversible transformations between the original features and the anonymized features to achieve controllability and reversibility of anonymization. The natural reversibility of the reversible neural network can make the entire anonymization process transparent.

[0060] More specifically, each of the reversible modules includes a first four arithmetic operation function group, a second four arithmetic operation function group and a plurality of encryption networks, and splits the initial identity feature into a first intermediate feature and a second intermediate feature, wherein:

[0061] The first intermediate feature is input into the first four arithmetic operation function group, and at the same time, the second intermediate feature is encrypted by the encryption network and then input into the first four arithmetic operation function group to obtain a first intermediate result;

[0062] The second intermediate feature is input into the second four arithmetic operation function group, and at the same time, the first intermediate result is encrypted by the encryption network and then input into the second four arithmetic operation function group to obtain a second intermediate result;

[0063] The first intermediate result and the second intermediate result are combined as the output of the reversible module, and the encryption result of the data by the encryption network is consistent with the encryption result of the data by the key.

[0064] The first four arithmetic operation function group includes at least two different four arithmetic operation functions, the second four arithmetic operation function group includes at least two different four arithmetic operation functions, and the first four arithmetic operation function group and the second four arithmetic operation function group are the same.

[0065] For example, Figure 3 As shown, the structure of different data streams corresponding to a reversible module.

[0066] For example, in the data flow direction from the initial face image to the anonymized face image, the first four arithmetic operation function group and the second four arithmetic operation function group are respectively multiplication functions and addition functions. Then, the first intermediate feature is first multiplied with the encrypted second intermediate feature to obtain the first data, and then the first data is added with the encrypted second intermediate feature to obtain the first intermediate result.

[0067] The first intermediate result is encrypted and multiplied with the second intermediate feature to obtain second data, and then the second data is added to the encrypted first intermediate feature to obtain the second intermediate result.

[0068] When the image needs to be restored, the addition operation becomes a subtraction operation, and the multiplication operation becomes a division operation.

[0069] In this embodiment, the construction process of the reversible neural network architecture used for identity feature transformation includes:

[0070] Step 1) First, a reversible module is constructed. The module consists of four identical arbitrary networks (encrypted networks) coupled together. The input of the module is the initial identity feature, and the output is the anonymized identity feature. The anonymization and restoration process is constrained by embedding a key in each reversible block. The reversible module is used to construct an image fusion network.

[0071] Step 2) The final reversible neural network is constructed by serially connecting multiple reversible blocks. The output of each reversible module is the input of the next reversible module. The network is used to anonymize and restore identity features. Each reversible module in the reversible neural network is exactly the same, and the same network is used in the anonymization and restoration process. In each reversible module, a reversible transformation is performed between the original image and the anonymized image through a set of dual-objective mapping functions.

[0072] The encrypted network can be any network, such as convolutional neural network, deep learning network, etc.

[0073] The reversible neural network architecture for identity feature transformation adopted in this embodiment includes multiple identical reversible modules. In the forward anonymization process, the key plays a constraining role. The anonymized image finally generated by the pre-trained model is unrecognizable to the human eye and the machine. The restoration process can only be correctly restored with the correct key, and the restored image is visually similar to the original image.

[0074] In this embodiment, several identical reversible modules are selected to construct a reversible neural network architecture, and the reversible module contains four identical deep networks. Reversible modules of other structures can also be selected in this field. The present invention does not require the specific coupling structure of the reversible module. In addition, the present invention does not limit the number of reversible module connections.

[0075] S3. Inputting the original face image and the anonymized identity features into a pre-trained generator to generate an anonymized face image.

[0076] The pre-trained generator is used to generate an anonymized face image from the anonymized features and the original face image. The anonymized image can be used for machine recognition, and the recognized identity is not the original face. The original face image can be finally restored by the correct key when needed. That is, the pre-trained generator is used to generate an anonymized face image from the original face and the anonymized features. The image cannot be accurately identified by the human eye as the original identity, and the pre-trained machine recognition model cannot recognize the original identity.

[0077] In order to make the anonymized image as dissimilar to the original image as possible, so that neither human nor machine can recognize it, and the restored image can be recognized by both human and machine, a loss function is used to update the parameters of the reversible network. The loss function used includes at least the identity information loss and image information loss between anonymous images generated by different keys, and the identity information loss and image information loss between anonymous images and restored images. That is, the entire anonymization and restoration process is constrained by four loss functions, namely, the identity information loss between anonymous images, the image information loss between anonymous images, the restored identity information loss and the image information loss. The identity loss function is used to ensure that the generated image is similar to the original image in terms of identity feature representation, and the image loss function is used to ensure that the generated image is visually dissimilar to the anonymized image.

[0078] More specifically, the encrypted network is trained. When the encrypted network is trained, the total loss function is used to update the parameters of the encrypted network. The total loss function is specifically:

[0079]

[0080] in, Represents the total loss function during training; To anonymize the loss of identity information between face images; To anonymize the image information loss between face images; To reduce the image information loss between anonymized face images and restored face images, is the background loss between the anonymized face image and the original face image, λ1, λ2, λ3, and λ4 are and The weight of .

[0081] The identity information loss between the anonymized face images The specific expression is:

[0082]

[0083] in, represents the feature difference loss function, which is used to measure the feature difference between T and P, where T and P are The input of the function, I represents the original face image, p1 and p2 represent different keys, p1′ represents the inverse of key p1, which is used to correctly restore the anonymized face image generated by key p1, p2′ represents the inverse of key p2, which is used to correctly restore the anonymized face image generated by key p2, T p1 I represents the original face image, the anonymized face image obtained by key p1, T p2 I represents the original face image, the anonymized face image obtained by key p2, Tp I represents the original face image, and the anonymized face image generated by the key p through the reversible neural network, T p1′ T p I represents the face image restored by the wrong key corresponding to the key p1, T p′ T p I represents the face image restored by the key p; T p2′ T p I represents the face image restored from the wrong key corresponding to the key p2.

[0084] It should be noted that the key of the anonymization process is p, and the restored process keys p1 and p2 are both incorrect keys.

[0085] This loss function is used to measure the identity information loss between anonymous images generated by different keys. Through this loss function, we can effectively optimize the model so that anonymous images generated using different keys are as dissimilar as possible in features, thereby ensuring that identity privacy is maintained to a certain extent while providing recovery capabilities.

[0086] The feature difference loss function is used to measure the identity information loss between anonymized face images generated by different keys. The specific expression is:

[0087]

[0088] Among them, F is the pre-trained face feature extractor, which is a pre-trained deep model for extracting the identity features of the image; M1 and M2 are anonymized face images generated by different keys, that is, anonymized face images generated by different keys via a reversible neural network and a pre-trained generator; the loss function constrains the reversible neural network by calculating the similarity between the anonymized faces, ensuring that the feature differences between the anonymous images generated by different keys are as large as possible.

[0089] The image information loss between the anonymized face images The specific expression is:

[0090]

[0091] Where I represents the original face image, F is the face classification model (i.e., the face feature extractor, which is pre-trained and has two outputs, one is the identity label and the other is the identity feature. Here, F is used to predict the identity label and then calculate the cross entropy loss with the real label yi), T p I represents the anonymized face image generated by the original face image I and the key p through the reversible neural network, T p′ T p I represents the face image recovered from the wrong key, It is the cross entropy loss, a loss function used to measure the difference between the classification model prediction and the true label. It calculates the distance between the predicted probability distribution and the true distribution, y I Represents the true label of the original face image, which is used to calculate the loss with the prediction of the face recognition model.

[0092] Loss function for anonymizing and restoring face images The specific expression is:

[0093]

[0094] Where I represents the original face image, T p and T -p denote the anonymized key and the correctly recovered key respectively, T - p T p I is the image that is expected to be correctly restored, and ||·,·||1 represents the L1 distance between the pixels of the two images.

[0095] The background loss between the anonymized face image and the original face image The specific expression is:

[0096]

[0097] Where I represents the original face image, T p and T p′ are different keys for anonymization, ||·,·||1 represent two images, and this loss function can ensure that the background difference between the generated anonymized image and the original image is as small as possible.

[0098] Identity features are extracted from the original face image. The feature extractor can use a pre-trained deep model to generate a set of keys for anonymization and restoration, and the original identity features are sent to the reversible neural network. In each reversible module, a set of dual-objective mapping functions are used to achieve reversible transformation between the original features and the anonymized features. The controllability and security of anonymization and restoration are enhanced by introducing keys. Finally, an anonymized image is generated from the anonymized features and the original face through a pre-trained generator. The image is unrecognizable by both machines and human eyes. When needed, the anonymized image can be restored with the correct key, and the restored image is visually similar to the original image.

[0099] Embodiment 2 also proposes a transparent and restorable face anonymization processing system, including an image feature extraction module, a feature conversion module and an anonymous face image generation module. The image feature extraction module extracts identity features from the input original image, and then generates a set of keys for constraining anonymization and restoration; the feature conversion module uses a reversible neural network to anonymize the original identity features, and controls the anonymization and restoration process by introducing keys. The anonymous face generation module is used to generate an anonymous image from the anonymized features and the original image; wherein the reversible neural network includes a plurality of reversible modules, each of which is composed of a plurality of arbitrary network couplings, and a reversible transformation is performed between the original features and the anonymized features through a set of dual-objective mapping functions, each reversible module is exactly the same, and the output of the previous reversible module is the input of the next reversible module, and the process control of anonymization and restoration is achieved by introducing keys, and when necessary, the original image can be finally restored and generated by the correct key.

[0100] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A transparent and restorable face anonymization method, characterized in that: include: Obtaining an original face image to be anonymized, extracting initial identity features from the original face image using a face feature extractor, and generating a set of keys for each of the original face images; Constructing a reversible neural network, and inputting the identity features and the key belonging to the same original face image into the reversible neural network as a set of input data, performing affine transformation through the reversible neural network, and finally generating anonymized identity features controlled by the key; The original face image and the anonymized identity features are input into a pre-trained generator to generate an anonymized face image.

2. A transparent and restorable face anonymization processing method according to claim 1, characterized in that: The reversible neural network includes a plurality of identical reversible modules, each of which includes a first four arithmetic operation function group, a second four arithmetic operation function group and a plurality of encryption networks, and splits the initial identity feature into a first intermediate feature and a second intermediate feature, wherein: The first intermediate feature is input into the first four arithmetic operation function group, and at the same time, the second intermediate feature is encrypted by the encryption network and then input into the first four arithmetic operation function group to obtain a first intermediate result; The second intermediate feature is input into the second four arithmetic operation function group, and at the same time, the first intermediate result is encrypted by the encryption network and then input into the second four arithmetic operation function group to obtain a second intermediate result; The first intermediate result and the second intermediate result are combined as the output of the reversible module, and the encryption result of the data by the encryption network is consistent with the encryption result of the data by the key.

3. A transparent and restorable face anonymization processing method according to claim 2, characterized in that: The first four arithmetic operation function group includes at least two different four arithmetic operation functions, the second four arithmetic operation function group includes at least two different four arithmetic operation functions, and the first four arithmetic operation function group and the second four arithmetic operation function group are the same.

4. A transparent and restorable face anonymization processing method according to claim 2, characterized in that: When training the encrypted network, the total loss function is used to update the parameters of the encrypted network. The total loss function is specifically: in, Represents the total loss function during training; To anonymize the loss of identity information between face images; To anonymize the image information loss between face images; To reduce the image information loss between anonymized face images and restored face images, is the background loss between the anonymized face image and the original face image, λ1, λ2, λ3, and λ4 are and The weight of .

5. A transparent and restorable face anonymization processing method according to claim 4, characterized in that: The identity information loss between the anonymized face images The specific expression is: in, represents the feature difference loss function, which is used to measure the feature difference between T and P, where T and P are The input of the function, I represents the original face image, p1 and p2 represent different keys, p1′ represents the inverse of key p1, which is used to correctly restore the anonymized face image generated by key p1, p2′ represents the inverse of key p2, which is used to correctly restore the anonymized face image generated by key p2, T p1 I represents the anonymized face image obtained by using the key p1 for the original face image I, T p2 I represents the anonymized face image obtained by using the key p2 from the original face image I, T p I represents the anonymized face image generated by the original face image I and the key p through the reversible neural network, T p1′ T pI represents the face image of the key p1 corresponding to the wrong key recovery, T p′ T p I represents the face image restored by the key p; T p2′ T p I represents the face image restored from the wrong key corresponding to the key p2.

6. A transparent and restorable face anonymization processing method according to claim 5, characterized in that: The feature difference loss function is used to measure the identity information loss between anonymized face images generated by different keys. The specific expression is: Wherein, F is the pre-trained facial feature extractor, and M1 and M2 are anonymized facial images generated by different keys.

7. A transparent and restorable face anonymization processing method according to claim 4, characterized in that: The image information loss between the anonymized face images The specific expression is: Where I represents the original face image, F is the face feature extractor, T p I represents the anonymized face image generated by the original face image I and the key p through the reversible neural network, T p′ T p I represents the restored face image after the wrong key is restored, is the cross entropy loss, a loss function used to measure the difference between the classification model prediction and the true label, y I Represents the true label of the original face image.

8. A transparent and restorable face anonymization processing method according to claim 4, characterized in that: Loss function for anonymizing and restoring face images The specific expression is: Where I represents the original face image, T p and T _p denote the anonymized key and the correctly recovered key respectively, T _p T p I is the image that is expected to be correctly restored, and ‖·,·‖1 represents the L1 distance between the pixels of the two images.

9. A transparent and restorable face anonymization processing method according to claim 4, characterized in that: The background loss between the anonymized face image and the original face image The specific expression is: Where I represents the original face image, T p and T p′ are different keys for anonymization, and ‖·,·‖1 represents two images.

10. A transparent and restorable face anonymization processing system, characterized in that: include: An identity extraction module is used to obtain the original face image to be anonymized, extract the initial identity features from the original face image through the face feature extractor, and generate a set of keys for each of the original face images; an anonymization module, used to construct a reversible neural network, and input the identity features and the key belonging to the same original face image into the reversible neural network as a set of input data, and finally generate an anonymized identity feature controlled by the key after the reversible neural network performs affine transformation; The generation module is used to input the original face image and the anonymized identity feature into a pre-trained generator to generate an anonymized face image.

Images