Face recognition privacy protection method based on virtual identity
By using a virtual identity-based facial recognition method, protected facial images are generated and different virtual identities are used, solving the problems of privacy leakage and identity feature leakage in existing technologies, and achieving efficient privacy protection and accurate facial recognition.
Patent Information
- Application Number
- CN202310469246.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2043-04-27
AI Technical Summary
Existing technologies cannot effectively protect user privacy in facial recognition and pose a risk of sensitive information leakage, especially when matching data across multiple servers.
A face recognition method based on virtual identities is adopted. Protected face images are generated through an identity residual transformation network, and different virtual identities are used on different servers to avoid the leakage of real identity features.
It achieves stronger privacy protection while maintaining high facial recognition accuracy and avoids data matching between servers.
Smart Images
Figure CN116631025B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of privacy protection and face recognition, and particularly relates to a face recognition privacy protection method based on virtual identity. BACKGROUND
[0002] Face recognition has rapidly become the default technology for identity management in the Internet of Everything society. The high level of security and convenience has prompted face recognition to be widely used in various fields, such as intelligent security. In particular, various smartphone applications require face recognition for identity verification to enable electronic payment or access control.
[0003] However, the vast amount of face images collected by face recognition contains a large amount of sensitive personal information. Current computer vision technology based on deep learning already has the ability to automatically extract such information. Once these face images are illegally misused by unregulated service providers or hijackers, user privacy will face potential and increasingly serious threats. Therefore, it is necessary to consider the protection of face privacy while enjoying the convenience brought by face recognition.
[0004] Existing work on protecting face recognition privacy focuses on removing sensitive information while retaining original identity features for recognition. However, sensitive information (such as soft biometric attributes) still exists in the original identity features, leaving potential risks of privacy leakage. SUMMARY
[0005] The technical problem solved by the present application is to provide a face recognition privacy protection method based on virtual identity, which uses virtual identity features instead of real identity features to perform face recognition, ensuring recognizability while effectively protecting user privacy. At the same time, users have different virtual identities on different servers, avoiding data matching between servers.
[0006] The present application adopts the following technical solutions to solve the above technical problems:
[0007] A face recognition privacy protection method based on virtual identity, comprising the following steps:
[0008] Step 1), the service end performing face recognition selects a virtual identity for the user and transmits it to the client, and then the client solves the identity residual for the user according to the received virtual identity, wherein the specific steps of solving the identity residual are as follows:
[0009] Step 1.1), select n face images of the user, and extract the identity features of the user from each image;
[0010] Step 1.2), calculate the identity residual Δz according to the following formula id :
[0011]
[0012] In the formula, represents the identity feature extracted from the cth face image, z vid is the virtual identity feature received by the client;
[0013] Solving,
[0014] Step 2), with the assistance of the identity residual, the client uses the identity conversion network to convert the image provided by the user for face recognition into a protected image corresponding to the virtual identity; the identity conversion network includes an identity encoder, an attribute encoder, a generator, and a discriminator, wherein the identity encoder is used to extract the identity feature in the face, the attribute encoder is used to extract the attribute feature in the face, the generator is used to couple the identity feature and the attribute feature to generate a new face, and the discriminator is used to improve the visual quality of the generated new face. The specific conversion steps are as follows:
[0015] Step 2.1), the identity encoder and the attribute encoder in the identity conversion network are used to extract the identity feature and the attribute feature from the image provided by the user for face recognition, respectively;
[0016] Step 2.2), the extracted identity feature is added to the identity residual to obtain a virtual identity feature;
[0017] Step 2.3), the obtained virtual identity feature and the attribute feature are jointly input into the generator to generate a protected face;
[0018] Step 3), the client transmits the protected face to the server;
[0019] Step 4), the server performs face detection on the protected face, uses a face recognizer to extract a virtual identity feature, then performs matching in a virtual identity database and gives a matching result.
[0020] As a further optimization scheme of the virtual identity-based face recognition privacy protection method, the way for the server to select a virtual identity for the user in step 1) is that the server initializes a pseudo-random number generator through a random seed, and then selects a corresponding virtual identity in the virtual identity database according to the output result of the generator.
[0021] As a further optimization scheme of the virtual identity-based face recognition privacy protection method, a virtual identity database is generated by randomly selecting a plurality of real identities in the CelebA dataset for combination to avoid leakage of real identities.
[0022] As a further optimization scheme of the virtual identity-based face recognition privacy protection method, in the identity conversion network in step 2), the identity encoder adopts a pre-trained lightweight LightCNN; the attribute encoder adopts a U-Net structure; the generator adopts a stack of several deconvolution layers, and simultaneously adopts adaptive attention normalization to couple the identity and attribute features; and the discriminator adopts a stack of several convolution layers.
[0023] As a further optimization scheme of the virtual identity-based face recognition privacy protection method, the training loss function of the identity conversion network is:
[0024] In the formula, is an adversarial loss, X i X is a face provided by a user for face recognition, p is a protected face, m is the number of scales adopted by the discriminator, D j (j) represents the output result of the scale j discriminator, and ReLU(·) is a ReLU activation function;
[0025] is an identity conversion loss, λ1 is a hyperparameter for balancing the identity conversion loss, E id is an identity feature extracted by the identity encoder, and cos(·,·) represents a cosine similarity;
[0026] is an attribute-preserving identity loss, λ2 is a hyperparameter for balancing the attribute-preserving identity loss, represents an attribute feature of the kth layer of the attribute encoder, and K is the number of layers of the attribute encoder;
[0027] is a reconstruction loss, λ3 is a hyperparameter for balancing the reconstruction loss,
[0028] As a further optimization scheme of the virtual identity-based face recognition privacy protection method, the identity conversion network is trained by using an Adam optimizer, β1=0, β2=0.99, the training batch size is set to 8, the initial learning rate is 0.0002, and the hyperparameters are: λ1=10, λ2=20, and λ3=10.
[0029] Compared with the prior art, the above technical scheme has the following technical effects:
[0030] 1. Existing methods for protecting facial recognition privacy require retaining the original identity features, but these features still reveal some sensitive information. This invention uses a virtual identity to completely remove the user's sensitive information, resulting in stronger privacy protection.
[0031] 2. Existing methods for protecting facial recognition privacy inevitably result in the loss of some identity information, thus reducing the accuracy of facial recognition. This invention avoids this problem by using virtual identities, preserving higher facial recognition accuracy.
[0032] 3. Unlike existing methods for protecting facial recognition privacy by using the same identity on multiple servers, this invention assigns different virtual identities to users, avoiding data matching between servers. Attached Figure Description
[0033] Figure 1 This is a flowchart illustrating the present invention;
[0034] Figure 2 This is step 1 of the present invention, initialization;
[0035] Figure 3 This is step 2 of the present invention, which protects face generation;
[0036] Figure 4 This is step 3 of the present invention: face recognition. Detailed Implementation
[0037] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:
[0038] This invention can be implemented in many different forms and should not be considered limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully express the scope of the invention to those skilled in the art. In the drawings, components are enlarged for clarity.
[0039] like Figure 1 As shown, this invention discloses a face recognition privacy protection method based on virtual identity, specifically including the following steps:
[0040] Step 1: Initialization: The server performing face recognition selects a virtual identity for the user and sends it to the client. The client then calculates the identity residual for the user based on the given virtual identity.
[0041] Step 2: Protected Face Generation: With the assistance of identity residuals, the client uses an identity transformation network to convert the image provided by the user for face recognition into a protected image with a virtual identity;
[0042] Step 3: Face recognition: The client transmits the protected image to the server to perform face recognition.
[0043] As shown in Figure 2 Step 1, the server first initializes the pseudo-random number generator by a random seed, wherein the random seed is selected as the ID of the server or a key defined by the server.
[0044] The virtual identity database is generated by combining a plurality of real identities randomly selected from the CelebA dataset, thereby avoiding the leakage of real identities.
[0045] Subsequently, according to the output result of the pseudo-random number generator, the client selects a corresponding virtual identity for the user in the virtual identity database and transmits it to the client.
[0046] The client selects n face images from the user and extracts identity features represented by the images, and calculates the identity residual Δz according to the following formula id :
[0047]
[0048] In the formula, z c represents the identity feature extracted from the cth face image, z represents the virtual identity feature received by the client; and vid
[0049] Solving,
[0050] As shown in Figure 3 In step 2, when the user provides a face image for face recognition on the client side, the face image is first transmitted to the identity conversion network. The identity adversarial network includes an identity encoder, an attribute encoder, a generator, and a discriminator, wherein the identity encoder is used to extract the identity feature of the face, the attribute encoder is used to extract the attribute feature of the face, the generator is used to couple the identity feature and the attribute feature to generate a new face, and the discriminator is used to improve the visual quality of the generated new face.
[0051] The identity encoder and the attribute encoder in the identity conversion network extract the identity feature and the attribute feature from the image provided by the user for face recognition, respectively. Then, the identity feature is added with the identity residual to obtain the virtual identity feature. Finally, the virtual identity feature and the attribute feature are jointly input into the generator to generate a protected face.
[0052] The specific network structure of the identity conversion network is that the identity encoder adopts a pre-trained lightweight LightCNN, the attribute encoder adopts a U-Net structure, the generator adopts a plurality of deconvolution layers stacked and simultaneously adopts adaptive attention normalization to couple the identity and attribute features, and the discriminator adopts a plurality of convolution layers stacked.
[0053] The training loss function of the identity conversion network is:
[0054] In the formula, is an adversarial loss, X i is the original face, X p is the protected face, m is the number of scales used by the discriminator, D j (·) represents the output result of the scale j discriminator, and ReLU(·) is the ReLU activation function.
[0055] is the identity conversion loss, λ1 is a hyperparameter for balancing the identity conversion loss, E id (·) is the identity feature extracted by the identity encoder, and cos(·,·) represents the cosine similarity.
[0056] is the attribute-preserving identity loss, λ2 is a hyperparameter for balancing the attribute-preserving identity loss, represents the attribute feature of the kth layer of the attribute encoder, and K is the number of layers of the attribute encoder.
[0057] is the reconstruction loss, λ3 is a hyperparameter for balancing the reconstruction loss,
[0058] In the code implementation, the Pytorch deep learning framework is used to train on a 3090Ti server, and the Adam optimizer is used for parameter optimization, where β1=0, β2=0.99. The training batch size is set to 8, the initial learning rate is 0.0002, and the hyperparameters are λ1=10, λ2=20, and λ3=10.
[0059] As shown in Figure 4 , the client transmits the protected image to the server for face recognition.
[0060] The face recognition step is, first, performing face detection on the protected image, then using the face recognizer to extract the virtual identity feature, and finally matching in the virtual identity database and giving the matching result.
[0061] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the prior art, and unless defined as such, should not be interpreted in an idealized or overly formal sense.
[0062] The above detailed description of the specific embodiments of the present application is provided for the purpose of further explaining the objects, technical solutions and advantages of the present application, and it should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1.A method for privacy protection of face recognition based on virtual identity, characterized in that, The method comprises the following steps: Step 1), the server performing face recognition selects a virtual identity for the user and transmits it to the client, and then the client solves the identity residual for the user according to the received virtual identity, wherein the specific steps of solving the identity residual are as follows: Step 1.1), select n face images of the user, and extract the identity features of the user in each face image; Step 1.2), the identity residual Δz is calculated according to the following formula id : In the formula, represents the identity features extracted from the cth face image, z vid is the virtual identity features received by the client; Solving, Step 2), with the assistance of the identity residual, the client uses an identity conversion network to convert the image provided by the user for face recognition into a protected image corresponding to the virtual identity; the identity conversion network comprises an identity encoder, an attribute encoder, a generator and a discriminator, wherein the identity encoder is used to extract the identity features in the face, the attribute encoder is used to extract the attribute features in the face, the generator is used to couple the identity features and the attribute features to generate a new face, and the discriminator is used to improve the visual quality of the generated new face, and the specific conversion steps are as follows: Step 2.1), the identity encoder and the attribute encoder in the identity conversion network are used to extract the identity features and the attribute features from the image provided by the user for face recognition; Step 2.2), the extracted identity features are added to the identity residual to obtain virtual identity features; Step 2.3), the obtained virtual identity features and attribute features are input into the generator to generate a protected face; Step 3), the client transmits the protected face to the server; Step 4), the server performs face detection on the protected face, uses a face recognizer to extract virtual identity features, and then performs matching in a virtual identity database and gives a matching result. 2.The virtual identity based face recognition privacy protection method of claim 1, wherein, The way the server selects a virtual identity for the user in step 1) is that the server initializes a pseudo-random number generator through a random seed, and then selects a corresponding virtual identity in the virtual identity database according to the output result of the generator. 3.The virtual identity based face recognition privacy protection method of claim 2, wherein, The virtual identity database is generated by randomly selecting a number of real identities in the CelebA dataset to avoid leakage of real identities. 4.The virtual identity based face recognition privacy protection method of claim 1, wherein, In the identity conversion network in step 2), the identity encoder uses a pre-trained lightweight LightCNN; the attribute encoder uses a U-Net structure; the generator uses a number of deconvolution layers stacked together, and uses adaptive attention normalization to couple the identity and attribute features; the discriminator uses a number of convolution layers stacked together. 5.The virtual identity based face recognition privacy protection method of claim 1, wherein, The training loss function of the identity conversion network is: In the formula, To combat loss, X i The face provided by the user for face recognition, X p The protected face, m is the number of scales used by the discriminator, D j (·) represents the result of the scale j discriminator output, and ReLU(·) is the ReLU activation function. is the identity conversion loss, λ1is a hyper-parameter balancing the identity conversion loss, E id is the identity feature extracted by the identity encoder, and cos(·, ·) denotes the cosine similarity. λ2is a hyper-parameter balancing the attribute-preserving identity loss, represents the attribute feature of the k-th layer of the attribute encoder, K is the number of layers of the attribute encoder; For the reconstruction loss, λ3 is a hyperparameter balancing the reconstruction loss, 6. The virtual identity based face recognition privacy protection method of claim 5, wherein, The identity conversion network is trained using the Adam optimizer, β1=0, β2=0.99, the training batch size is set to 8, the initial learning rate is 0.0002, and the hyperparameters are: λ1=10, λ2=20, λ3=10.