Personalized facial information protection method based on feature reconstruction

Abstract

This invention discloses a personalized face information protection method based on feature recombination, comprising the following steps: S1, using a pre-trained GAN Inversion network to preprocess the feature of the image to be anonymized to obtain latent space features, and obtaining the user's anonymized attribute keyword list K1 and preserved attribute keyword list K2; S2, constructing a cross-modal driven attribute decoupling-fusion network; S3, anonymizing the user-provided anonymized attribute keywords; S4, restoring the user-provided anonymized attribute keywords; S5, training using a public dataset and performing multi-dataset generalization testing, outputting the final result. This method can support fine-grained anonymization and attribute preservation according to actual needs, allowing users to flexibly decide which attributes to retain or anonymize in different scenarios.

Description

Technical Field

[0001] This invention relates to the field of facial recognition and information protection technology, specifically to a personalized facial information protection method based on feature recombination. Background Technology

[0002] The large-scale collection and application of facial images poses a serious threat to personal privacy, such as the leakage of sensitive information. This has led to the development of stricter laws and regulations (such as GDPR, CCPA, PDPA, and PIPA) for data management. A direct impact is that people need to comply with complex legal or ethical constraints to alleviate apprehension before accessing, using, or disseminating facial data, which can hinder many important scientific research or intelligent applications. One viable approach is to develop effective anonymization techniques to remove sensitive information while retaining necessary non-sensitive information (such as facial expressions), as people also expect the protected data to remain useful. Such techniques can be used to support various computer vision tasks (such as street view mapping and medical diagnosis) to remove restrictions on data management and unlock its potential.

[0003] In recent years, privacy-preserving methods based on Generative Adversarial Networks (GANs) have received increasing attention. Compared to commonly used blurring, pixelation, and blackening methods, these GAN-based methods have the potential to generate more realistic images and preserve the usability of data, such as facial expressions, age, and gender. Various promising strategies have been proposed to alter or hide identity information, such as adding Laplacian noise and performing feature operations. Furthermore, some privacy-preserving strategies, such as k-same and differential privacy, have been applied to provide formal privacy guarantees for identity security. While previous work has achieved good results in privacy preservation, it lacks flexibility in controlling which information can be removed or retained under different conditions and requirements. Existing work primarily treats identity as sensitive information and retains other information. However, this is not always the case. For example, sometimes certain attributes (such as gender and age) may also be sensitive because they may cause legal or regulatory problems for individuals or organizations.

[0004] In conclusion, the protection of facial image identity privacy is a topic worthy of in-depth research. This patent aims to explore this topic from several key aspects of the field and address the difficulties and key points of current methods.

[0005] The main challenge in protecting facial image privacy lies in balancing anonymity with data reusability. Data reusability can be categorized into three aspects: detection rate, image quality, and non-identifiable facial attributes. Most current methods only consider anonymity, detection rate, and image quality, neglecting non-identifiable facial attributes. Specifically, there are two main difficulties:

[0006] 1. How to balance anonymity and data reusability. Data reusability mainly refers to detection rate, image quality, and non-identification attributes of faces. The main challenge of this task is to reduce the face re-identification rate while ensuring that the generated images can still detect faces with the same attributes and have good image quality.

[0007] 2. Most current methods achieve anonymity by face swapping or merging other people's IDs with their own data. They lack effective control over non-identity attributes and cannot change the anonymity result according to the anonymity requirements. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing a personalized facial information protection method based on feature recombination. This method supports fine-grained anonymization and attribute preservation according to actual needs. It effectively ensures data reusability and demonstrates good anonymization capabilities.

[0009] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0010] A personalized facial information protection method based on feature recombination includes the following steps:

[0011] S1. Use a pre-trained GAN Inversion network to preprocess the feature of the image to be anonymized to obtain latent space features, and obtain the user's anonymized attribute keyword list K1 and preserved attribute keyword list K2.

[0012] S2. Construct a cross-modal driven attribute decoupling-fusion network;

[0013] S3. Anonymize the anonymous attribute keywords provided by the user;

[0014] S4. Restore the anonymous attribute keywords provided by the user.

[0015] S5. Use the public dataset for training and multi-dataset generalization testing, and output the final results.

[0016] Preferably, step S1 uses a pre-trained GAN Inversion network to preprocess the features of the image to be anonymized, and the specific steps are as follows:

[0017] S1-1. Image Preprocessing. Using existing methods, the coordinates of the main facial features are obtained, the face is cropped from the image, and the face is kept parallel to the bottom edge of the image as input for facial feature mapping.

[0018] S1-2. Facial Feature Mapping. Facial images are mapped to the StyleGAN latent space using GAN Inversion technology, and the latent space features are used as input for training and testing.

[0019] As a preferred option, step S2 is as follows:

[0020] S2-1. A reversible neural network structure is used as the decoupled fusion network, which can accept bidirectional input. The network is divided into a forward process and a backward process according to the data flow. The forward and backward processes share the same network structure, as well as their building blocks and parameters, but differ only in the basic arithmetic operations within the building blocks because the data flow is reversed.

[0021] S2-2. Using CLIP's text encoder as a conditional encoding network, the input keyword k i Encoding into conditional features

[0022] S2-3. Add a conditional building block consisting of three fully connected (FC) layers and an activation function Tanh, which incorporates the input conditional features. Combined with the original input of the module, it realizes the process of decoupling the positive condition and fusing the negative condition.

[0023] As a preferred option, step S3 is as follows:

[0024] S3-1. Take a keyword from the anonymous attribute keyword list K1. by Conditional use of attribute decoupling-fusion network input latent space features Decouple the corresponding attribute features and residual characteristics

[0025]

[0026] S3-2. Calculate current attribute features through attribute decoupling-fusion network. Compared with all other candidate features The distance between features is calculated by defining the feature distance l2 between features, which is expressed as:

[0027]

[0028] S3-3. Calculate current attribute features through attribute decoupling-fusion network. For each candidate feature The feature distance is based on the global minimum-maximum normalization result, and is used as the feature distance for each structure with respect to the current attribute features. The utility fraction u is expressed as:

[0029]

[0030] S3-4. For each candidate feature Differential privacy through an exponential mechanism is used to compute each candidate feature. The selection probability P is used to randomly select a surrogate feature based on the probability. The probability calculation formula is:

[0031]

[0032] S3-5. Using attribute decoupling-fusion network in reverse fusion process to fuse residual features and proxy attribute features Obtain the final latent space features

[0033] S3-6, Repeat 3-1 to 3-5 until all attributes corresponding to the keywords in K1 in the image are anonymized;

[0034] S3-7, Using the StyleGAN generator part as the image generation network, the final latent space features Input yields an anonymous image;

[0035] As a preferred option, step S4 is as follows:

[0036] S4-1. Take a keyword from the attribute keyword list K2. use From the original input latent space features Decouple the corresponding attribute features Repeat until all the corresponding attributes that need to be preserved have been extracted;

[0037] S4-2. Take a keyword from the user-provided list of keyword keywords for retaining attributes, K2. use From input latent space features Decouple the corresponding attribute features and residual characteristics Replace the attribute features with the original features extracted in step 3-1, and then... The corresponding attribute portion in the feature is recovered by fusion. The above steps are repeated until all of them are recovered in K2.

[0038] This invention has the following characteristics and beneficial effects:

[0039] This method supports fine-grained anonymization and attribute preservation based on actual needs. Most existing work typically treats anonymization as a binary problem of hiding the original identity information while striving to preserve other information. In contrast, this method generalizes it to a fine-grained problem by unifying anonymization and utility preservation into a cyclical process, allowing users to flexibly decide which attributes to retain or anonymize in different scenarios.

[0040] The specific features are as follows: (1) This method is preprocessed and trained in the latent space of StyleGAN, which can generate virtual faces; (2) By repeatedly adding or deleting certain attributes, flexible and controllable attribute preservation can be achieved; (3) Real human face images can be output. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the network architecture according to an embodiment of the present invention.

[0043] Figure 2 This is a flowchart of the anonymous recovery method according to an embodiment of the present invention.

[0044] Figure 3 The diagrams show a comparison of the effects of the embodiments of the present invention with those of the prior art: (a) original image, (b) GAN Inversion diagram, (c) Ours, (d) IdentityDP, (e) CIAGAN, (f) DeepPrivacy, (g) Blurring, and (h) Pixelation.

[0045] Figure 4 This is a screenshot showing the personalized anonymization effect of this method.

[0046] Figure 5 This is an experimental result comparing the anonymity rate and image quality preservation ability of this method with other methods. Detailed Implementation

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0048] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0049] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0050] This invention provides a personalized facial information protection method based on feature recombination, such as... Figure 1 As shown, it includes the following steps:

[0051] S1. Use a pre-trained GAN Inversion network to preprocess the feature of the image to be anonymized to obtain latent space features, and obtain the user's anonymized attribute keyword list K1 and preserved attribute keyword list K2.

[0052] Among them, the anonymous attribute keyword list K1 and the preserved attribute keyword list K2 are trained using a complete set of keywords (e.g., ["gender", "age", "identity"...]). K1 and K2 are subsets of this set, and in this embodiment, they are selected by the user.

[0053] Specifically, in this embodiment, step S1 includes the following sub-steps:

[0054] S1-1. Image Preprocessing. Using existing methods, the coordinates of the main facial features are obtained, the face is cropped from the image, and the face is kept parallel to the bottom edge of the image as input for facial feature mapping.

[0055] S1-2. Face Feature Mapping. Face images are mapped to the StyleGAN latent space using GAN Inversion technology. The latent space features are used as inputs for training and testing, and the shape of the features is (1, 4928).

[0056] S2. Construct a cross-modal driven attribute decoupling-fusion network;

[0057] Specifically, S2-1 uses a reversible neural network structure as a decoupled fusion network that can accept bidirectional input. The network is divided into a forward process and a reverse process based on the data flow. The forward and reverse processes share the same network structure, as well as their building blocks and parameters, but differ only in the basic arithmetic operations within the building blocks because the data flow is reversed.

[0058] S2-2. Using CLIP's text encoder as a conditional encoding network, the input keyword k i Encoding into conditional features

[0059] S2-3. Add a conditional building block consisting of three fully connected (FC) layers and an activation function Tanh, which incorporates the input conditional features. Combined with the original input of the module, it realizes the process of decoupling the positive condition and fusing the negative condition.

[0060] S3. Anonymize the anonymous attribute keywords provided by the user;

[0061] Specifically, such as Figure 2 As shown, S3-1, take a keyword from the anonymous attribute keyword list K1. by Conditional use of attribute decoupling-fusion network input latent space features Decouple the corresponding attribute features and residual characteristics

[0062] S3-2. Calculate current attribute features through attribute decoupling-fusion network. Compared with all other candidate features The distance between features is calculated by defining the feature distance l2 between features, which is expressed as:

[0063] It should be noted that in this embodiment, the candidate features are features obtained in advance from 10,000 virtual images generated by StyleGAN using an attribute decoupling-fusion network.

[0064] S3-3. Calculate current attribute features through attribute decoupling-fusion network. For each candidate feature The feature distance is based on the global minimum-maximum normalization result, and is used as the feature distance for each structure with respect to the current attribute features. The utility fraction u is expressed as:

[0065]

[0066] S3-4. For each candidate feature Differential privacy through an exponential mechanism is used to compute each candidate feature. The selection probability P is used to randomly select a surrogate feature based on the probability. The probability calculation formula is:

[0067]

[0068] S3-5. Using attribute decoupling-fusion network in reverse fusion process to fuse residual features and proxy attribute features Obtain the final latent space features

[0069] S3-6, Repeat 3-1 to 3-5 until all attributes corresponding to the keywords in K1 in the image are anonymized;

[0070] S3-7, Using the StyleGAN generator part as the image generation network, the final latent space features Input yields an anonymous image;

[0071] S4. Restore the anonymous attribute keywords provided by the user, specifically as follows: Figure 2 As shown,

[0072] S4-1. Take a keyword from the attribute keyword list K2. use From the original input latent space features Decouple the corresponding attribute features Repeat until all the corresponding attributes that need to be preserved have been extracted;

[0073] S4-2. Take a keyword from the user-provided list of keyword keywords for retaining attributes, K2. use From input latent space features Decouple the corresponding attribute features and residual characteristics Replace the attribute features with the original features extracted in step 3-1, and then... The corresponding attribute portion in the feature is recovered by fusion. The above steps are repeated until all of them are recovered in K2.

[0074] It should be noted that in this embodiment, k iIt is a streaming process. First, anonymize K1 and then restore K2. Unpacking these two sets gives K1 = [k1, k2, k3] and K2 = [k4, k5].

[0075] In this embodiment, K1 and K2 can be the same. When there is a scenario that requires anonymous restoration, the same keyword is used in K1 and K2 (for example, if I want to anonymize gender, that is, K1 = ["gender"], the intermediate result can generate an image with gender anonymized, and then I use K2 = ["gender"], the latent space of the anonymized gender in the middle as input, and I can get an image with restored gender).

[0076] S5. Use the public dataset for training and multi-dataset generalization testing, and output the final results.

[0077] Furthermore, this embodiment provides a specific testing method as follows:

[0078] Prepare the dataset, for example, using the CelebA-HQ and FFHQ datasets, and preprocess them as described in step 1 to obtain the required input features. CelebA-HQ contains 30,000 face images from 6,216 identities, with 5,000 images used as the test set and the remainder used for training. FFHQ contains 70,000 face images at a resolution of 1024×1024. This dataset exhibits significant variation in age, image background, and facial attributes.

[0079] The training data is fed into the network for training, and the model is tested using test data and the trained network. We calculate the Rank@1 face re-recognition rate (lower is better) using a pre-trained FaceNet and perform face detection using FaceAlignment (higher is better). We evaluate attribute preservation performance based on a pre-trained ResNet classifier (higher is better). We use the Frechet Inception Distance (FID) as a reference metric to evaluate image quality (or similarity) compared to the original dataset (lower is better).

[0080] To verify the efficiency of the proposed method, it is compared with current best practices (such as DeepPrivacy and CIAGAN), and the anonymity effect is analyzed by calculating the anonymity rate. A face detection model is used to calculate the detection rate of the generated images, and FID is used to calculate image quality. Furthermore, non-identity attribute classifications, such as gender and age, are performed on the generated images to analyze the data reusability of the method. Specific comparison images are shown below. Figure 3 As shown.

[0081] To verify the generalization ability of the proposed method, test result images were generated using FFHQ test data and a network using CelebA-HQ as training data, and the metrics were evaluated.

[0082] Comparison of experimental results between this embodiment and existing technologies:

[0083] 1. The re-identification rate of our method was tested against that of blurring, mosaic, DeepPrivacy, CIAGAN, and IdentityDP methods. Detailed results are as follows: Figure 5 As shown.

[0084] 2. The attribute preservation rates of our method were compared with those of fuzzing, mosaic, DeepPrivacy, CIAGAN, and IdentityDP methods. Detailed data results are as follows... Figure 5 As shown.

[0085] 3. The image quality retention rates of our method and other methods such as blurring, mosaic, DeepPrivacy, CIAGAN, and IdentityDP were tested. Detailed data results are as follows: Figure 5 As shown.

[0086] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A personalized facial information protection method based on feature recombination, characterized in that, Includes the following steps: S1. Use a pre-trained GAN Inversion network to preprocess the image to be anonymized to obtain latent space features, and obtain the user's list of anonymous attribute keywords. and maintain the list of attribute keywords ; S2. Construct the cross-modal driven attribute decoupling-fusion network CDFNet; Step S2 includes the following sub-steps: S2-1. Use a reversible neural network structure as the decoupling and fusion main network; S2-2. Using CLIP's text encoder as a conditional encoding network, input keywords... Encoding into conditional features ; S2-3. Add a conditional building block, consisting of three fully connected layers and the Tanh activation function, to process the input conditional features. Combined with the original input of the condition construction module, it realizes the process of decoupling forward conditions and fusing backward conditions; S3. Anonymize the anonymous attribute keywords provided by the user. S3-1, From the list of anonymous attribute keywords Take a keyword ,by Conditional use of attribute decoupling-fusion network input latent space features Decouple the corresponding attribute features and residual characteristics The expression is as follows; S3-2. Calculate current attribute features through attribute decoupling-fusion network. , and all other candidate features Calculate the feature distance between features. , which is represented as ; S3-3. Calculate current attribute features through attribute decoupling-fusion network. For each candidate feature The feature distance is based on the global minimum-maximum normalization result, and is used as the feature distance for each structure with respect to the current attribute features. utility score Its expression is: S3-4. For each candidate feature Differential privacy through an exponential mechanism is used to calculate each candidate feature. The selection probability P is used to randomly select a surrogate feature based on the probability. The probability calculation formula is: S3-5. Using attribute decoupling-fusion network in reverse fusion process to fuse residual features and proxy attribute features The final latent space features are obtained. ; S3-6, Repeat steps 3-1 to 3-5 until the image is displayed. All attributes corresponding to the keywords in the text are anonymous; S3-7, Using the StyleGAN generator part as the image generation network, the final latent space features Input yields an anonymous image; S4. Restore the anonymous attribute keywords provided by the user. S4-1, From the list of retained attribute keywords Take a keyword ,use From the original input latent space features Decouple the corresponding attribute features Repeat this process until all the corresponding attributes that need to be preserved have been extracted; S4-2, From the list of retainable attribute keywords provided by the user Take a keyword ,use From input latent space features Decouple the corresponding attribute features and residual characteristics Replace the attribute features with the original features extracted in S3-1, and then... To restore the corresponding attribute portion of the feature, fuse the data and repeat the above steps until... All of them have been restored; S5. Use the public dataset for training and multi-dataset generalization testing, and output the final results.

2. The personalized facial information protection method based on feature recombination according to claim 1, characterized in that, The method for image data preprocessing is as follows: S1-1. Image preprocessing: Obtain face coordinates, crop the face from the image, and ensure that the face is parallel to the bottom edge of the image, as input for face feature mapping. S1-2, Face Feature Mapping: The face image is mapped to the StyleGAN latent space using GAN Inversion technology, and the latent space features are used as input for training and testing.

3. The personalized facial information protection method based on feature recombination according to claim 1, characterized in that, The anonymous attribute keyword list and maintain the list of attribute keywords It includes information such as gender, age, and identity.

Images